Paperspace

Our Engineering team has confirmed that the issue preventing users from accessing or logging in to the Paperspace Console has been resolved, and the service has been fully restored. Thank you for your patience and understanding throughout this incident. If you continue to experience any issues, please reach out to our Support team, and we will be happy to assist you further.

Our Engineering team has resolved the issue that was preventing users from accessing or logging in to the Paperspace Console, and we are currently monitoring the results. We apologize for any inconvenience this may have caused and thank you for your patience and understanding.

We are currently investigating an issue that is preventing users from accessing or logging in to the Paperspace Console. We apologize for any inconvenience this may have caused and thank you for your patience and understanding.

This incident has been resolved now.

We've identified and implemented a fix for the issue and are monitoring the results. Some users might still experience issues as we continue working on this.

We are currently investigating an issue preventing users from interacting with Core Virtual Machines in the CA1 Region.

Our Engineering team has resolved the incident with intermittent connectivity issues affecting a few Core VMs in the NY2 data center. If you continue to experience problems, please open a ticket with our Support team. We apologize for any inconvenience this may have caused.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

As of 14:30 UTC, our Engineering team is investigating reports of network connectivity issues impacting a small subset of customer VMs in NY2 region. Users may experience intermittent connectivity issues. We apologize for the inconvenience and will share an update once we have more information.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

The issue affecting a subset of GPU instances in the NYC region has been resolved. Customers using P5000, P6000, and RTX5000 instances may have previously experienced difficulties when attempting to power on these resources. Importantly, this issue had no impact on running instances, only power-on actions were affected. Our engineering team has implemented a fix and is continuing to monitor the environment to ensure stability.

We are currently investigating an issue affecting a subset of GPU instances in the NYC region. Customers using P5000, P6000, and RTX5000 instances may experience difficulties when attempting to power on these resources. Our engineering team is actively working to identify the root cause and implement a resolution. In the meantime, impacted customers may encounter delays or failures when starting affected instances. We will continue to provide updates as more information becomes available. Thank you for your patience and understanding.

This incident has been resolved.

A fix has been implemented and the service has been fully restored. Our engineering team has resolved the issue, and we are actively monitoring the system to ensure continued stability.

Our Engineering team has identified the cause of the service disruption impacting Gradient notebooks and is actively working on a fix. At this time, users may continue to experience intermittent issues with their notebooks. Our team is implementing a solution to resolve the issue as soon as possible. We will provide another update once the fix is deployed and the service is restored.

Our engineering team is investigating reports of service disruption impacting Gradient notebooks. At this point, users may experience intermittent issues with their notebooks. We apologize for the inconvenience and appreciate your patience and cooperation as we work to resolve the issue. Our engineers are actively investigating the cause and working towards a solution. We will share an update once we have further details.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are currently investigating a network disruption that is preventing users from interacting with Gradient notebooks. Our engineers are working on resolution of the Gradient notebooks' unavailability.

A fix has been deployed. Users should be able to create and interact with their notebooks normally. Awaiting confirmation before closure.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We are currently investigating a network disruption that is preventing users from interacting with Gradient notebooks.

This incident has been resolved. The core VMs should be working as intended. Thanks for your cooperation.

We've identified an ongoing issue impacting Core Machines. Our engineering team is actively investigating the root cause and working diligently toward a resolution. We understand how disruptive this may be and sincerely apologize for the inconvenience. Please rest assured that resolving this remains a top priority, and we’ll keep you updated with progress as soon as more information becomes available.

This incident has been resolved. The core machines should be working as intended. Thank you for your patience and cooperation.

Our teams have applied a fix and connectivity in the CA1 region is returning to normal. We are monitoring the environment to confirm full recovery.

We have identified a service disruption affecting all resources in the CA1 region. Our team is working to restore services as quickly as possible. Users may be unable to access their Core VMs during this time. Further updates will be provided as recovery progresses.

This incident has been resolved. Thank you everyone.

A fix has been implemented and we're monitoring the results. Thanks for your patience.

The issue has been identified, a fix has been implemented and everything is operational now.

We are currently experiencing an outage affecting the Paperspace main website (paperspace.com). At this time, the Paperspace Console remains fully operational and can be accessed directly at: https://console.paperspace.com In addition, our ticketing system is encountering delays, which may impact our ability to respond as quickly as usual. Our Support team is actively monitoring the situation and will reply as soon as the system stabilizes. We will continue to provide updates as we work to restore full functionality. Thank you for your patience and understanding.

The incident in the AMS region has been fully resolved, and all services are now available. If you experience any further issues, please don't hesitate to contact our support team for assistance.

A fix has been implemented and we are monitoring the results.

Our engineers are continuing to work on restoring full functionality in the AMS region. While some services remain partially unavailable for certain customers, recovery efforts are actively ongoing. We will provide further updates as progress continues.

The AMS region is operating in a partially degraded state. While some services are accessible, others remain unavailable for certain customers. We have identified the issue and our engineers are actively working on recovery. We’ll provide further updates as progress continues.

The AMS region is now operational, but some users may experience slow performance as systems stabilize. Our team is monitoring the recovery and working to restore full functionality.

Update on Paperspace AMS Region Outage Our teams continue to work on recovering Paperspace services in the AMS region. Some customers may see improvements in bringing up their instances. We'll provide further updates as progress is made.

The issue has been identified and a fix is being implemented.

We have identified a service disruption affecting our AMS region due to a power issue. Our team is working diligently to resolve this matter as quickly as possible. In the meantime, users may experience issues with their Core Products.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are currently investigating a network issue that is preventing users from interacting with Gradient notebooks.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We’re currently investigating an issue affecting Core Machines. Our engineers are actively looking into the root cause to restore full functionality as quickly as possible. Thank you for your patience while we work on this, and we’ll share updates as soon as we have more details.

We've identified an ongoing issue impacting Core Machines in NY2. Our engineering team is actively investigating the root cause and working diligently toward a resolution. We understand how disruptive this may be and sincerely apologize for the inconvenience. Please rest assured that resolving this remains a top priority, and we’ll keep you updated with progress as soon as more information becomes available.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are currently investigating customers being unable to log in using Single Sign-On (SSO). Standard login methods remain available. Our engineering team is actively working to restore full functionality, and we will share updates as they become available.

The issue was identified as affecting only a specific cohort of users. We have verified that all the systems including the Paperspace Console are operating as expected. Any remaining access concerns will be handled on case-by-case basis. We appreciate your patience and understanding.

We are currently investigating an issue that is preventing users from accessing the Paperspace console. Our team is working to resolve the issue as quickly as possible. We apologize for any inconvenience and will provide updates as we progress.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are investigating an issue affecting Core VMs in the NY2 region, which may cause accessibility issues. Our team is working to resolve the issue as quickly as possible. We apologize for any inconvenience and will provide updates as we progress.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are seeing that our API is having degraded performance, also, access to the Paperspace Console might be intermittent. Our engineers are investigating the root cause. We appreciate your patience.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

We've identified the issue and are working on a fix. Some users might still experience issues while we are implementing the fix.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are currently investigating a network issue that is preventing users from interacting with Gradient notebooks and Core VMs.

This incident has been resolved. Thank you for your patience and cooperation.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate the issue preventing users from logging into their Paperspace accounts. We sincerely apologize for the inconvenience this has caused and appreciate your patience as we work toward a resolution.

We are currently investigating an issue that is preventing users from logging into their Paperspace account. We apologize for the inconvenience caused.

Our engineers have applied a fix and the issue is considered resolved. If you continue to experience problems, please contact our Support Team. We apologize for the inconvenience experienced.

We are continuing to monitor for any further issues.

Our engineers have implemented a solution and we are currently monitoring the results.

Our engineers have identified the issue and are actively working on implementing a fix. We appreciate your patience as we work to resolve this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for the issue. We apologize for any inconvenience caused.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue that is preventing users from interacting with their CORE machines in CA1. Our engineers are working on a fix. We apologize for any inconvenience caused.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating a network issue that is preventing users from interacting with Gradient notebooks.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating a network issue that is preventing users from interacting with Gradient notebooks.

This issue has been resolved now. Thank you for your patience and cooperation.

Our engineers implemented a fix and we're monitoring the situation. If you're still experiencing issues, please contact our Support Team.

A fix has been implemented and we're monitoring the results. Thank you for your patience and cooperation.

Our engineers have identified a networking issue affecting Paperspace Bare-Metal services caused by our upstream provided. Our network team is actively working with our upstream provider to mitigate the issue. Further updates will be provided as we implement corrective measures.

We are investigating a Network Issue that's affecting our Bare-Metal users in the NY Region. We apologize for the inconvenience.

Our engineers have resolved the issue. Please contact our Support Team if you encounter any problems with your CA1 CORE Virtual Machine.

Our engineers have implemented a fix and events should start processing as usual. Customers that still experience issues, please wait a few minutes and try again or contact our Support Team.

We are currently investigating an issue that is preventing CORE machines in CA1 from processing events. Our engineers are working on a fix. We apologize for any inconvenience caused.

This incident has been resolved. Thank you for your patience and cooperation.

We are continuing to monitor for any further issues. We appreciate your patience and cooperation.

Our engineering team has implemented a fix and we are monitoring the results. We appreciate your patience and cooperation.

Our engineers have identified the issue and are working on a fix for this issue. We apologize for the inconvenience caused.

The issue has been identified and a fix is being implemented.

We're investigating an issue where users aren't able to interact with the Paperspace Console and other Paperspace Services. We apologize for any inconvenience caused.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Several reports have been done regarding machines not being accessible on both Core and Gradient, we are investigating the root cause of this issue, and your patience is appreciated.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue that is preventing news users and teams from accessing the billing page.

The issue has been resolved. If you experience any problems, please contact our Support Team.

Our engineers implemented a fix and we're monitoring the situation. Users should be able to interact with their VMs as normal. If you experience any issues, please contact our Support Team.

Our engineers are investigating an issue that prevents NY2 Region users from interacting with their Virtual Machines. We apologize for the inconvenience.

The incident was resolved by our engineers. If you continue to experience issues, please contact our Support Team.

Our engineers have implemented a fix. Users should be able to access Gradient assets as normal. If you experience any problems, please contact our Support Team.

We are currently investigating an issue that is preventing users from interacting with Gradient notebooks. We apologize for the inconvenience caused.

The issue affecting both CORE and Gradient users in the NY2 Region has been resolved. If you continue to experience problems, please contact our Support Team.

Our engineers have implemented a fix and we're monitoring the situation. If you continue to experience issues, please let us know.

Our engineers are currently working on a fix. We appreciate your patience.

We're investigating a possible network issue that's affecting CORE and Gradient services in the NY2 Region. We apologize for the inconvenience experienced.

This incident has been resolved, free notebooks are working well. We appreciate your patience and cooperation. Thank you.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

We've identified the issue with Free notebooks being stuck in a pending state and are working on a fix. Some users might still experience issues while we are implementing the fix.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

Our engineers have identified generalized issues with the network connectivity, we are working on a solution to ensure connectivity to all our users. We appreciate your patience.

# Paperspace NY2 network outage Postmortem ## Incident Summary On June 16th, a significant number of virtual machines in the NY2 region became unavailable and went into Read-Only \(RO\) mode, which affected network availability. By 18:20 UTC on the same day, network connectivity was restored to the affected virtual machines. However, affected virtual machines continued to be unavailable until 12:23 UTC on June 17th as they were still in RO mode. The affected virtual machines were restored to Read-Write \(RW\) mode between 5:58 and 12:23 UTC on June 17th. By 12:24 UTC, affected virtual machines were available and network access to customers was restored.  ## Incident Details ### Root Cause The root cause of this unexpected service unavailability is related to a core switch failure in NY2 that caused a large number of machines to go into RO mode. ### Impact As a result of the failure of the core switch, there was a spike in traffic that adversely impacted network performance. The virtual machines went into RO mode as a result of the loss of network access to the Network File System on which they resided. Since virtual machines were in RO mode, customers were unable to perform any write operations on them, resulting in service disruptions around 10:30 UTC on June 16th, 2024.  ## Remediation Actions A number of efforts are underway to try to prevent these types of failures from occurring again, including a network redesign and installation of new equipment. On behalf of Paperspace, we apologize for the disruption to your services and appreciate your understanding.   If you have any questions or concerns, please open a ticket with our[ Customer Support team](https://docs.digitalocean.com/support/paperspace/#open-a-ticket.).

Our engineers have resolved the issue. If you continue to experience issues, please contact our Support Team.

We've implemented a series of fixes and are monitoring the results. While Core is already fixed, the Gradient platform will continue to have degraded performance since it is still taking time to recover. We will continue working on fixes for it.

The issue has been identified, it appears to be an issue with our network in NY2. A fix is being implemented.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are currently investigating a network issue that is preventing users from interacting with Gradient notebooks and Core VMs based in NY2 region.

Our engineers have resolved the issue. If you continue to experience issues, please contact our Support Team.

A fix has been implemented and we are monitoring the results.

We've identified the issue and are working on a fix. Some users might still experience issues while we are implementing the fix.

We are currently investigating an issue that may prevent users from interacting with CORE in the NY2 Region. We apologize for the inconvenience caused.

Our engineers have resolved the issue. If you continue to experience issues, please contact our Support Team.

Our engineers have identified the issue that's preventing some users from interacting with Gradient Notebooks and are in the process of fixing it. Thanks for your patience.

Our engineers are investigating an issue that is affecting some Gradient Notebook users. We apologize for the inconvenience.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an issue that is preventing users from interacting with Gradient.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue that is preventing users from interacting with their Console as expected. This includes logging in, turning their machines on and off, and interacting with the API. We are working on getting this fixed ASAP.

This incident has been resolved.

Our engineers have addressed the issue. Most users should be able to use their CORE Virtual Machines in the AMS1 Region. If you continue to experience issues, please reach out to our Support Team.

Our engineers are working to resolve a networking issue that may prevent some users in the AMS1 Region interacting with their CORE Virtual Machines. We apologize for the inconvenience.

The issue has been resolved by our engineers. We apologize for any inconvenience caused. If you continue to experience issues, please contact our Support Team.

Our engineers have addressed the issue. Most users should be able to use their CORE Virtual Machines in the AMS1 Region. If you continue to experience issues, please reach out to our Support Team.

Our engineers have addressed the issue. Most users should be able to use their CORE Virtual Machines in the AMS1 Region. If you continue to experience issues, please reach out to our Support Team.

Our engineers have identified a potential electrical issue that may be impacting users of the AMS Region CORE Virtual Machine and are currently in the process of rectifying it. We apologize for any inconvenience this has caused you.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

Our engineers are investigating an issue where users may not be able to interact with Gradient Notebooks and Core VMs in NY2

We experienced a brief network disruption affecting GRADIENT services at 20:53 EDT. This issue has been resolved and all systems are back online. We apologize for any inconvenience caused. If you are still seeing issues, please reach out to Support.

At this time we are confident that the incident has been resolved. If you are still seeing issues, please reach out to Support.

We've implemented a fix and are monitoring the results. Users should be able to load the console again.

We are currently investigating an issue that is affecting Core and Gradient users. During this time users will face some issues while attempting to access the resources and may receive 429 errors. Our engineers are working on a fix. We apologize for any inconvenience caused.

This incident has been resolved.

We are monitoring the situation. Some users may experience issues with Gradient and Free instances continue to run at a lower capacity. We apologize for the inconvenience.

We continue to keep an eye on the situation. Gradient Free instances are currently running at a lower capacity. We apologize for the inconvenience.

We continue to monitor the situation. Users may encounter issues starting Free Gradient instances. We apologize for the inconvenience.

We've implemented a fix and are monitoring the results.

Our engineers have identified and working on a fix for GRADIENT. We apologize for the inconvenience caused.

We have implemented a fix for CORE and continue to investigate GRADIENT. We apologize for the inconvenience caused.

We are continuing to work on a fix for this issue.

Our engineers are still working on resolving this issue. We appreciate your patience and apologize for the inconvenience.

Our engineers continue investigating the issue. We apologize for the inconvenience caused.

Our engineers are investigating an issue where users may not be able to interact with Gradient Notebooks. We apologize for the inconvenience caused.

Our engineers have resolved the issue. If you encounter any issues, please contact our Support Team.

Our engineers continue to work on resolving this issue. We apologize for any inconvenience caused.

We are currently investigating an issue that may prevent users from interacting with the CORE API. We apologize for the inconvenience caused.

The issue affecting CORE and GRADIENT users in the NY2 Region has been resolved. If you experience any further issues, please contact our Support Team.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are still investigating a network issue that may prevent users from interacting with CORE and Gradient in the NY2 Region. We apologize for the inconvenience caused.

We are currently investigating a network issue that may prevent users from interacting with CORE and Gradient in the NY2 Region. We apologize for the inconvenience caused.

The issue has been resolved by our engineers. If you need further assistance, please contact our Support Team.

Our engineers have implemented a fix. If you continue to experience issues, please reach out to our Support Team.

Users may experience issues starting or stopping CORE Virtual Machines in the NY2 Region. Our engineers are already working on a fix. We sincerely apologize for any inconvenience that may have been caused.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Our engineers are currently investigating an issue causing slower than usual response times on services in the NY2 Region.

The issue affecting users in the NY2 Region has been resolved. If you experience any further issues, please contact our Support Team.

Our engineers are currently investigating an issue causing slower than usual response times on services in the NY2 Region.

This incident has been resolved.

We are continuing to monitor for any further issues.

A full fix has been implemented for Core and Gradient. We are monitoring the results.

We are continuing to see residual effects on Gradient. Our engineers are continuing to implement a fix.

A fix has been implemented and we are monitoring the results.

We are currently seeing slower performance than usual on Gradient and CORE in NY2, we appreciate your patience while our engineers work on it.