Crusoe

This incident has been resolved.

We have applied a fix which looks to have mitigated the issue. Please let us know if you are still seeing an issue. We are continuing to monitor.

We continue to investigate the issue and have narrowed down the root cause. Our team is actively working towards a resolution and will provide a further update as soon as possible.

We are investigating an issue affecting the Crusoe Container Registry (CCR) in the US South Central (us-southcentral1-a) region. Beginning at approximately 04:07 UTC on July 8, 2026, customers may be unable to push or pull container images, and may see TLS handshake timeouts or authentication errors. Our engineering team is actively investigating. We will provide more information as we investigate the issue.

This incident has been resolved. Network connectivity in our us-east-1-a region has been fully restored. We sincerely apologize for any disruption this may have caused and thank you for your patience.

A fix has been implemented and we are monitoring the results.

We have implemented a fix for the network connectivity issues affecting us-east-1-a. We are seeing improvements across impacted services and are actively monitoring the environment to confirm full recovery. We will provide a further update once we have verified complete resolution. We apologize for any disruption this has caused.

We are currently investigating reports of degraded network connectivity in our us-east-1-a region. Some customers may be experiencing connection timeouts, SSL handshake failures, or degraded network performance for workloads hosted in this region. Our engineers are actively working to mitigate the issue as soon as possible. We will provide further updates as more information becomes available.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We're currently investigating an issue affecting a subset of customer VMs in our us-east2 region . A subset of VMs are showing as UNSPECIFIED in our control plane . We have identified the issue and are actively working on fixing it to bring affected VMs back to a healthy state. We will provide more information as soon as it becomes available.

This incident has been resolved.

A fix has been applied and the outage has been mitigated. Services have returned to normal operations in us-east1-a region. We are continuing to monitor the situation and validate preventive measures. Please contact support@crusoecloud.com if you are still experiencing any issues.

We are continuing to monitor for any further issues.

We have identified and mitigated the underlying cause. Although no further impact is expected, we are continuing to monitor. If you are still experiencing issues, please reach out to us through support@crusoecloud.com and we will investigate further.

We're making significant progress in resolving the virtual machine accessibility issues in the us-east1-a region. We apologize for the continued disruption and thank you for your patience as we work to restore full service. We will provide another update as soon as more information is available or when services are fully restored. You can continue to contact our support team at support@crusoecloud.com, if there are any other concerns to address

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are currently investigating an issue impacting virtual machine accessibility in us-east1-a region. Some users may be unable to access their VMs at this time. We appreciate your patience and will provide updates as more information becomes available.

We are continuing to investigate this issue.

We are currently investigating a potential service interruption that may impact some customers in our us-east1-a region. If you are experiencing issues in this region, please reach out to support@crusoecloud.com

We identified general network errors affecting service connectivity in our us-east1 region. The issue has been resolved and services are operating normally.

This incident has been resolved.

We've identified the cause of the errors when creating/starting VMs or creating scaling CMK clusters in us-east2 region and applied a fix. We'll continue monitoring.

We continue investigating "insufficient capacity" errors when creating/starting VMs or creating scaling CMK clusters in us-east2 region. We'll share an update as soon as we have more information available.

We continue investigating "insufficient capacity" errors when creating/starting VMs or creating scaling CMK clusters in us-east2 region. We'll share an update as soon as we have more information available.

We continue investigating "insufficient capacity" errors when creating/starting VMs or creating scaling CMK clusters in us-east2 region, and have narrowed down a likely cause. We'll share an update as soon as we have more information available.

We are investigating "insufficient capacity" errors when creating/starting VMs or creating scaling CMK clusters in us-east2 region. Existing VMs continue to run normally. We'll share an update as soon as we have more information available.

A subset of resources in us-east1-a experienced intermittent external connectivity disruption during a planned network maintenance window. Affected workloads may have observed elevated latency, connection timeouts, or failures reaching external destinations. Virtual machine availability was not affected. The change was rolled back to restore connectivity, which was confirmed stable following the revert.

This incident has been resolved.

The issue was identified and resolved as of approximately 15:45 PDT. Load balancer connectivity in the ICAT region has been restored.

Beginning at approximately 13:55 PDT, load balancers in the ICAT region became unreachable. Customers relying on load balancer endpoints in this region may be experiencing connectivity failures. Our team is actively investigating and working to restore service. We will provide updates as more information becomes available.

This issue is now resolved.

The issue has been identified and a fix is being implemented.

We have identified an issue affecting custom image creation in us-east1-a, us-southcentral1-a and eu-iceland1-a regions. We have identified the underlying cause and are actively working to restore service. We will provide more information as soon as possible.

This issue has been resolved.

A fix has been implemented and we are monitoring. Please let us know if you are still seeing any network related performance issues. We will keep you updated with any further developments.

We are seeing a network outage in this region, which is currently under investigation. We will keep you posted on further developments as we make progress on this matter

This issue is now resolved.

We are now monitoring the incident and if you see any issues, please reach out to support.

We are seeing intermittent network degradation in this region, which is currently under investigation. We will keep you posted on further developments as we make progress on this matter

This issue has been resolved.

We are seeing an improvement in latency and our teams remain fully focused on driving further recovery. We will provide an update as soon as we have more information available.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are investigating an issue on the eu-iceland1-storage cluster. Customers may experience elevated write latency. We will provide an update as soon as we have more information available.

This incident has been resolved. All affected compute hosts and customer virtual machines in us-east1-a have been restored.

Remediation is ongoing and the number of impacted customers continues to narrow. Engineering remains actively engaged, and we will provide a further update as the work advances.

Engineering continues to make progress on remediation since the last status post. Work remains ongoing for the customers still in progress, We will provide a further update as recovery advances.

Engineering has restored the majority of affected services in us-east1-a and is in the final stages of remediation. A subset of customers remain impacted and are being prioritised for recovery. We will provide a further update shortly.

Work to restore service in us-east1-a continues. Engineering has made significant progress, with the large majority of affected compute hosts now recovered. A small number of hosts remain under active remediation. We will provide a further update shortly.

Work to restore service in us-east1-a continues, with engineering making steady progress across affected compute hosts. A subset of hosts remain impacted as remediation work advances. We will provide a further update shortly.

Engineering continues to make progress on the issue in us-east1-a, though recovery is proceeding more slowly than initially anticipated. A subset of compute hosts in the region remains impacted while remediation work continues. We will provide a further update as recovery advances.

Work to restore service is ongoing as our teams continue to drive remediation in us-east1-a. Engineering remains engaged across affected compute hosts and we will provide a further update shortly.

Engineering continues to work on the issue in us-east1-a. A subset of compute hosts in the region remains impacted while remediation steps are being applied. We will provide a further update once additional progress can be reported

Remediation in us-east1-a is ongoing and continuing to progress. Additional virtual machines have returned to service; however, a subset of compute hosts remains impacted and the region is not yet fully restored. Engineering remains engaged and further updates will follow.

We continue to see progress in us-east1-a as engineering works through mitigation. Additional virtual machines have returned to service, although a subset of compute hosts remains impacted, and the issue is not yet fully mitigated. Engineering remains actively engaged, and we will provide further updates as recovery continues.

We are seeing partial recovery in us-east1-a following mitigation steps applied by engineering. A subset of affected virtual machines have returned to service. Engineering is continuing to validate the mitigation and work toward full restoration. A subset of compute hosts remains impacted at this time. We will provide further updates as progress continues.

We are continuing to work on the issue affecting us-east1-a. Engineering has identified a likely cause and mitigation is in progress. A subset of compute hosts remains unavailable, and Managed Kubernetes and Container Registry operations in the region continue to be impacted. We will continue to provide updates as remediation progresses.

We are continuing to investigate this issue.

We are investigating a networking issue in the us-east1-a region affecting connectivity to a subset of compute hosts. Affected virtual machines may appear in an unspecified state or be unreachable over the network. Managed Kubernetes clusters in the region may also report nodes as Not Ready. Container Registry operations in the region, including image push/pull and repository management, are also impacted. Engineering is engaged and actively working to identify the cause and restore service. We will provide updates as we have more information.

VM creation and start operations in the us-southcentral1-a and eu-iceland1-a regions have been functioning normally for several hours. Our Engineering teams have addressed the underlying issue and a permanent fix is being prepared for deployment to prevent recurrence. Existing running workloads and resources remained fully operational throughout the incident. We apologize for any inconvenience this may have caused.

The issue affecting the ability to create and start Virtual Machines in the us-southcentral1-a and eu-iceland1-a regions has been mitigated. VM creation and start operations are functioning normally. No customer impact has been observed as of 14:00 UTC. Our Engineering teams continue to monitor both regions closely and are working on a permanent fix to prevent recurrence. We apologize for any inconvenience this may have caused.

We have identified the underlying issue affecting the ability to create and start Virtual Machines in the us-southcentral1-a and eu-iceland1-a regions. Our Engineering teams are actively working on recovery. Some customers may still be experiencing impact while this work continues. Existing running workloads and resources remain fully operational and are not impacted. We will provide a further update as soon as more information becomes available.

We are investigating an issue affecting the ability to create and start Virtual Machines in the us-southcentral1-a and eu-iceland1-a regions. Existing running workloads and resources remain fully operational and are not impacted. Our Engineering teams are actively working on recovery. We will provide an update when more information is available.

This incident has been resolved.

Customers using CCR (Crusoe Container Registry) will be unable to push or pull images. We are working on resolving this matter as soon as possible. If you experience any other issues, please reach out to support@crusoe.ai

Service has been fully restored. We have identified the root cause and are taking steps to prevent recurrence.

We are investigating an issue affecting access to Crusoe Managed Inference, including Intelligence Foundry and Bring Your Own Model (BYOM). Customers may be unable to reach inference endpoints or send requests to the service. Engineering is actively working on recovery. We will provide an update when more information is available.

This incident has been resolved.

The issue has been mitigated and project creation requests are completing successfully.

We're investigating failures when creating new projects. Existing workloads and resources remain fully operational. We will provide an update when more information is available.

This incident has been resolved.

We are currently experiencing an issue with storage used disk capacity metrics in our Iceland (eu-iceland1) region. All data access, read/write operations, and service functionality continue to operate normally with no performance impact. Our team is actively working to restore capacity metrics.  We will provide updates as restoration progresses.

This incident has been resolved.

The issues affecting the Crusoe Compute Console, API, and CLI have been fully resolved. Impact Window: The service interruption lasted from 2:42 PM PDT to 3:16 PM PDT. During this window, users may have experienced 503 errors and were unable to view or manage resources via the web console, CLI, or API. Our engineering team identified the issue and implemented a fix to restore all management services. We have verified that all systems are now performing as expected. We are continuing to monitor the platform and will implement additional safeguards and alerting to improve our response and stability moving forward. Note: Active customer workloads and existing infrastructure remained operational and were not impacted by this management-layer interruption.

We are investigating an issue causing 503 errors when accessing the Crusoe Compute Console and API. Users may currently be unable to list instances or manage infrastructure resources. Our engineering team is actively working to identify the root cause. Further updates will be provided as soon as they are available.

This incident has been resolved.

We are currently investigating an issue specifically affecting Persistent Volume (PVC) creation for Crusoe Managed Kubernetes (CMK) customers. While this interruption prevents the provisioning of new block storage volumes, it is strictly limited to block storage and does not impact Shared Disks or other services. Existing volumes and active workloads remain fully operational and unaffected. Our engineering team has identified the root cause and is restoring the service. We expect to provide a final update once provisioning functionality is fully restored.

This incident has been resolved.

A fix has been implemented and we are monitoring the results. If you are experiencing issues, please contact Crusoe Support.

We are investigating an issue in our eu-iceland1 region that may be impacting the ability to manage resources. Our engineering team is actively working to identify the scope and cause. We will provide another update as more information becomes available.

This incident has been resolved.

We are aware of an issue with an upstream dependency that may impact the ability to add new nodes to existing node pools. Existing clusters and running workloads are not affected. We are actively monitoring the situation and will provide updates as it progresses. If you are experiencing issues, please contact Crusoe Support.

This incident has been resolved.

We are investigating an issue affecting Kubernetes CMK control plane operations. Customers may experience elevated latency or timeouts when retrieving cluster credentials via the CLI or console, and may be unable to create new clusters. Existing cluster workloads are not affected. We will provide a further update as our investigation continues.

This incident has been resolved.

We are pleased to report that this incident is now resolved. We have confirmed that control plane functionality for all impacted clusters in the us-southcentral region has been fully restored. The issue was caused by network congestion on a small number of underlying physical hosts. This has been remediated by isolating the source of the traffic. We will be conducting a full post-mortem to prevent a recurrence and apologize for any disruption this incident caused.

We are investigating an issue in our US-Southcentral region that may be impacting the ability to manage cluster resources via the API. Our engineering team is actively working to identify the scope and cause. We will provide another update as more information becomes available.

This incident has been resolved.

The issue has been mitigated and is now operating normally. We are monitoring the system to ensure continued stability.

We are continuing to investigate this issue.

We are currently investigating an issue impacting our Shared Disks in the eu-iceland1-a region. We are actively working with our storage vendor to resolve this issue. We will provide another update as soon as we have more information.

A fix has been implemented which fixed the issue. All operations should have resumed normally.

We are currently investigating an issue impacting our Shared Disks in the eu-iceland1-a region. Customers may not be able to create, read, update, and delete operations. VM operations may also be impacted. We are working with our Storage vendor on a resolution currently.

This issue has been resolved. If you are still experiencing any other issues related to this matter, please contact support@crusoe.ai.

Our storage vendor has successfully mitigated the issue. Control plane operations are now fully operational. We will monitor for some time before declaring the issue resolved.

We are currently investigating an issue affecting control plane operations for shared storage in the eu-iceland1-a region. Existing shared volumes and running VMs are not affected, but creation and deletion of shared volumes may be impacted. We are actively working with our storage vendor to resolve this issue. We will provide another as soon as we have more information.

This issue has been resolved. If you are still experiencing any other issues related to this matter, please contact support@crusoe.ai.

We are currently investigating reports of failures when creating Custom Images in our EU-Iceland (eu-iceland1-a) region. Users may receive "Internal Server Errors" or experience operation timeouts when attempting to create Custom Images from existing disks. If you experience any other issues, please reach out to support@crusoe.ai.

This issue has been resolved. Please contact support@crusoe.ai if you are still experiencing any issues.

Customers using  CCR (Crusoe Container Registry ) will be unable to push or pull images in us-east1, eu-iceland1 and us-southcentral1. We are working on resolving this matter as soon as possible.  If you experience any other issues, please reach out to support@crusoe.ai

Final monitoring of the eu-iceland1-a storage cluster has been successfully completed, and all services remain stable. All planned recovery activities and software upgrades are finished, with performance and VM operations restored to normal levels. This incident is now resolved.

As of this morning, all planned activities to restore storage access have been completed. As we transition the system back to normal operations, we are now moving into a monitoring phase.

As of this morning, all planned activities to restore storage access have been completed. As we transition the system back to normal operations, we are now moving into a monitoring phase.

VAST is performing an additional upgrade to the storage software in our eu-iceland1-a region today, January 1st, at 10:00 AM PT. While we do not anticipate any downtime, you may experience temporary service degradation during this period. We appreciate your continued patience as we work through these final stages of recovery. If you have specific concerns or questions about this maintenance window, please don't hesitate to contact us at support@crusoecloud.com.

The planned VAST storage software upgrade, which began at 9:00 AM PT, completed successfully. The storage cluster is stable, and all services are operating normally.

We are performing an additional upgrade to the VAST storage software today, December 31st, at 9:00 AM PT, to further optimize and accelerate the recovery process. While we do not anticipate any downtime, you may experience temporary service degradation during this period. We appreciate your continued patience as we work through these final stages of recovery. If you have specific concerns or questions about this maintenance window, please don't hesitate to contact us at support@crusoecloud.com.

The planned VAST storage software upgrade, which began at 12:00 PM PT, completed successfully. The storage cluster is stable, and all services are operating normally.

We are upgrading the VAST storage software at 12:00 PM PT today, December 30th, to optimize recovery progress further. This maintenance window is required to deploy critical improvements to the recovery process and has an expected duration of 4 hours. We do not anticipate any downtime; however, you may experience some service degradation. If you have specific concerns or questions regarding this maintenance window, please contact support@crusoecloud.com.

The planned VAST storage software upgrade, which began at 9:00 AM PT, has been completed successfully. The storage cluster is stable, and all services are operating normally.

To ensure the reliability of the system after today's issue, we are postponing the VAST storage software upgrade, initially planned at 6:00 AM PT on December 28th, to 9:00 AM PT. This maintenance window is required to deploy critical improvements to the recovery process and has an expected duration of 4 hours. The new upgrade window is 9:00 PT to 13:00 PT. We do not anticipate any downtime; however, you may experience some service degradation. If you have specific concerns or questions regarding this maintenance window, please reach out to support@crusoecloud.com.

We have restored full service to the eu-iceland1-a storage cluster. The storage node failures have been remediated, and shared disks are once again performing normally. VMs operations have also been stabilized. We sincerely apologize for the continued disruption. Our teams are now working with the vendor on an exhaustive review of the cluster's recent stability to prevent future recurrences. VAST upgrade at 6:00 AM PT will proceed as planned.

We have identified an issue with the underlying storage infrastructure in the eu-iceland1-a region. Several storage components have gone offline unexpectedly, impacting the availability of shared volumes and preventing VMs management operations. We are working urgently with our vendor, VAST, to restore these components and recover service. We understand the impact this has on your active workloads and are treating this with the highest priority. This is unrelated to the VAST upgrade planned at 6:00 AM PT on December 28th, which hasn't started yet.

We are upgrading the VAST storage software at 6:00 AM PT on December 28th to further optimize recovery progress. This maintenance window is required to deploy critical improvements to the recovery process and has an expected duration of 4 hours. We do not anticipate any downtime; however, you may experience some service degradation. If you have specific concerns or questions regarding this maintenance window, please reach out to support@crusoecloud.com.

The planned VAST configuration update has been completed successfully across the cluster. We are now transitioning back to active recovery operations for the remaining volumes requiring remediation. We will continue to monitor the progress and stability of the cluster as these recovery efforts continue.

Our storage vendor will be applying a configuration update to the underlying storage nodes starting at 10:00 AM PT, which will take approximately 15 minutes to complete. This maintenance is designed to optimize and increase the speed of existing recovery workflows. We do not anticipate any downtime; however, you may experience some service degradation. If you have specific concerns or questions regarding this maintenance window, please reach out to support@crusoecloud.com.

We have confirmed that the storage network issue in the eu-iceland1-a region is now resolved. The storage cluster is currently stable. Access to shared volumes should be restored for everyone, and all Virtual Machine (VM) operations—including create, start, stop, and delete—have returned to normal functionality. We are monitoring the environment closely alongside our partner, VAST, to ensure ongoing stability and will keep you posted on any further developments or changes to this status.

We are expanding the scope of impact for the ongoing storage network issue in the eu-iceland1-a region. In addition to the accessibility of shared volumes, this issue is now also impacting Virtual Machine (VM) operations, including create, start, stop, and delete. We are actively working with our storage partner, VAST, to remediate the problem. They are in the process of rebooting the cluster to restore access. Further updates will be provided as we make progress.

We have identified a new storage network issue impacting accessibility of shared volumes in the eu-iceland1-a region. We are currently engaged with our storage partner, VAST, to remediate the issue and restore stable access. Further updates will be provided as we make progress.

The storage network issues in the eu-iceland1-a region have been resolved, and access to shared volumes is restored. Together with our vendor VAST, we have stabilized the network layer and verified that I/O operations are performing normally. We will continue to monitor the environment closely to ensure continued stability.

We have identified a new storage network issue impacting accessibility of shared volumes in the eu-iceland1-a region. We are currently engaged with our storage partner, VAST, to remediate the issue and restore stable access. Further updates will be provided as we make progress.

We have completed our initial review of an approximately 10 minute storage outage in eu-iceland1-a region. The interruption was caused by a manual administrative error by our storage provider (VAST) during the data recovery process. While service was restored within minutes, we recognize that any downtime is unacceptable. Our storage partner, VAST, is conducting a full post incident review and implementing additional automation and safeguards to eliminate reliance on manual steps and reduce the risk of similar issues in the future.

We observed a blip that affected our storage cluster in the eu-iceland1-a region from 2025-12-23 19:52:53 UTC to 2025-12-23 20:03:47 UTC. This would have resulted in read, write, and update operations failing on all shared volumes. The service should be restored now. Our engineering team is working with our storage vendor to investigate the cause of this.

The planned VAST storage software upgrade has been completed successfully across the cluster. We are now transitioning back to active recovery operations for the remaining volumes requiring remediation. With the new automated tooling now in place, engineering teams are resuming the execution of the validated remediation workflows. We will continue to monitor the progress and stability of the cluster as these recovery efforts scale.

To accelerate the VAST recovery progress at scale, we need to deploy an upgrade to the underlying VAST storage software. We are planning a maintenance window at 12:00 am PT on December 23rd. Target Start Time: 12:00 AM PST Expected Duration: 4 Hours Purpose: This VAST upgrade introduces new automated tooling designed to accelerate recovery workflows. Customer Impact: We do not anticipate any downtime; however, you may experience some service degradation (e.g. temporary latency or brief I/O pauses). If you have specific concerns or questions regarding this maintenance window, please reach out to support@crusoecloud.com

The storage cluster continues to operate with stable performance for the majority of workloads. Regarding the subset of volumes requiring further intervention, we are closely coordinating with our storage vendor to execute comprehensive integrity scans and fully map the scope of residual inconsistencies. In parallel, the vendor is currently finalizing the development of automated remediation workflows, modeled on successful manual recovery procedures validated on a limited data subset earlier today. We are closely monitoring this development and will only authorize the execution of these workflows once the solution has been rigorously validated. We will provide further details as we prepare to transition from these ongoing scans to the active recovery phase.

The majority of customers should now observe a return to normal operations and consistent performance within the eu-iceland1-a region. While general cluster stability has been restored, engineering teams are continuing focused recovery efforts for a limited subset of customers who remain more deeply impacted by residual metadata inconsistencies. We are maintaining close oversight of service health and will continue to work directly with our storage vendor to finalize remediation for the remaining impacted volumes. Further updates will be provided as the recovery process nears full completion

Engineering teams have implemented configuration changes to isolate background recovery processes from active customer workloads. Following these adjustments, we are observing signs of stabilization in cluster performance and a reduction in the intermittent connectivity fluctuations reported earlier. We are continuing to monitor service health closely. Please note that while stability has improved, further maintenance operations may be required to finalize the recovery efforts. We are currently evaluating the necessity and timing of these potential steps with the vendor and will provide updates as plans are finalized.

We are continuing to address the ongoing service instability affecting shared volumes in the eu-iceland1-a region. While engineering teams continue the previously reported background recovery efforts, we are observing intermittent performance fluctuations and connectivity drops, which may result in transient errors for active workloads. We are actively engaged with our storage vendor to diagnose the specific drivers of this instability and are working to restore consistent performance to the cluster.

Hello Team, during our monitoring we are observing VM operation failures, as well as shared volume create/update/delete failures due to ongoing impact to the storage control plane. Some customers may still observe issues when accessing existing volumes. A small subset of specific files or directories may remain temporarily inaccessible or return Input/Output (I/O) errors. We are actively engaged with our vendor and pursuing further investigation on this matter.

Engineering teams have completed the critical phase of the integrity verification process and have lifted the broad protective maintenance restrictions. As a result, service availability has been restored for the majority of volumes and data paths in the eu-iceland1-a region. However, a small subset of specific files or directories may remain temporarily inaccessible or return Input/Output (I/O) errors. We have transitioned to a monitoring phase and teams are actively working to restore full access to these remaining items. We will provide a final confirmation once the incident is fully resolved.

Engineering teams are currently executing a targeted integrity verification process to permanently resolve the metadata inconsistencies reported earlier. To ensure strict data safety during this operation, the storage cluster has entered a protective maintenance state that temporarily restricts access to prevent modification while checks are running. As a result of these active safety measures, customers in eu-iceland1-a may experience service unavailability, including mount failures or Input/Output (I/O) errors, for specific volumes, directories, or files. Please be assured that these restrictions are intentional precautions to preserve data integrity. We expect to lift these protective measures once the verification process is complete.

Engineering teams are continuing to actively remediate specific storage metadata inconsistencies and stabilize network connectivity within the eu-iceland1-a region. During this window, customers may continue to experience intermittent write errors, latency spikes, or brief connectivity drops. We are prioritizing work to restore full service performance and expect to provide further details within approximately 2 hours.

We are continuing to observe impact related to the storage performance degradation in eu-iceland1-a. Our engineering teams remain actively engaged with our vendor in focused remediation efforts and are working to address residual intermittent slowness for some customers utilizing Shared Storage services. We will provide another update as soon as we have further developments. Thank you for your continued patience

We have implemented a fix for the storage performance issues in eu-iceland1-a. Storage latency and throughput have returned to normal operating levels. We are actively monitoring the service to ensure continued stability.

We are expanding the scope of impact for this incident. Customers will observe significant slowness and intermittent access issues when utilizing Shared Storage services in eu-iceland1-a region. We will provide another update as soon as we have further developments. Thank you for your patience as we work to restore full service as quickly as possible.

We are currently investigating slowness when using Shared Storage in eu-iceland1-a. We are working with our storage vendor to resolve the issue as soon as possible.

This issue is resolved and connectivity to all Direct NFS mounts in the eu-iceland1 region has been fully restored. The network mitigation is verified, and the system is stable.

We have restored connectivity to all affected Direct NFS mounts by applying a mitigation to the network hardware. Service is now operating normally and we are monitoring the system to ensure continued stability.

It appears that several storage backend VIPs are experiencing connectivity problems. We are actively collaborating with both our networking vendor and our storage vendor to identify and implement a resolution.

We are currently investigating an issue that is causing some Direct NFS mounts to be unavailable in the eu-iceland1 region. Customers utilizing Direct NFS for shared drives may be unable to access their mounts. We are actively diagnosing the problem. We will post another update as soon as more information is available.

A fix has been applied and the issue regarding "Device Not Ready" (DNR) errors in the eu-iceland1-a region is now resolved. All impacted services and Virtual Machines have returned to normal operations.

We are currently experiencing a network instability issue in eu-iceland1-a region, causing a subset of virtual machines to lose access to their boot disk. This is resulting in multiple VMs (Virtual Machines) throwing DNR (Device Not Ready) errors. We will provide another update as soon as we have further developments. Thank you for your patience as we work to restore full service as quickly as possible.

This incident has been resolved.

The issue is resolved and Cloud Console is accessible and functioning as expected.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are currently investigating an issue affecting access to the Cloud Console UI and API. Users may be unable to log in to our cloud services or access the management dashboard.

Issue resolved.

We have mitigated the issue and connectivity is confirmed to be restored. We will monitor overnight.

We have identified the issue and are deploying a fix.

We are continuing to investigate this issue.

We have isolated the issue to a subset of VMs within the Iceland region and determined that it is not a core network transport failure. Our Engineering teams are continuing to actively troubleshoot the issue to restore service as soon as possible.

We are currently investigating an incident affecting VM connectivity and performance in the Iceland region. Our Engineering teams are actively engaged and working to isolate the root cause. We will provide updates as our investigation progresses.

Update: The issue is now resolved. The engineering team has implemented a fix and confirmed that normal provisioning operations have been restored.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We have determined that the problem is sporadic and limited to a specific subset of our host nodes. Existing, currently running VMs are operational. We still strongly recommend customers avoid rebooting critical workloads until the permanent fix is confirmed to be fully deployed. If you are currently experiencing issues with starting or restarting VMs, please open a support ticket so we can immediately address your specific host. We will provide another update once the targeted fix has been deployed to the identified hosts and we have confirmed stability.

We are continuing to investigate this issue.

We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully. Existing, currently running VMs are operational. We advise customers to avoid rebooting critical workloads until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

Update: The issue is now resolved. The engineering team has implemented a fix and confirmed that normal provisioning operations have been restored.

We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on all instance types. Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

This incident is now resolved.

A fix has been implemented, and we are monitoring the environment.

We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on our A100 Infiniband hardware fleet. Any new VM provisioning request for this hardware type will also fail. Additionally, any existing VM on an A100 Infiniband node that is stopped and started (or rebooted) will also fail to come back online. Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

This incident has been resolved. Please contact support@crusoe.ai if you are still experiencing any issues.

A fix has been applied and the outage has been mitigated. Services have returned to normal operations in eu-iceland1-a region. We are continuing to monitor. Please contact support@crusoecloud.com if you are still experiencing any issues.

We have identified and mitigated the issue. Although no further impact is expected, we are continuing to monitor. If you are still experiencing issues, please reach out to us through support@crusoecloud.com and we will investigate further.

We're making significant progress in resolving the virtual machine accessibility issues in the eu-iceland1-a region. We apologize for the continued disruption and thank you for your patience as we work to restore full service. We will provide another update as soon as more information is available or when services are fully restored. You can continue to contact our support team at support@crusoecloud.com, if there are any other concerns.

We are currently investigating an issue impacting virtual machine accessibility in eu-iceland1-a region. Some users may be unable to access their VMs at this time. We appreciate your patience and will provide updates as more information becomes available.

A fix has been implemented. This incident is now resolved

We are currently investigating an issue with machine accessibility and performance issues in the us-southcentral1 region . Some users may be unable to access their VMs at this time or see performance degradation. We appreciate your patience and will provide updates as more information becomes available.

The fix has been deployed, and the user interface now correctly displays the capacity for CMK nodepool creation. You can now use either the UI or the CLI to provision new nodepools.

Please be aware that the user interface is currently not showing the correct capacity for CMK nodepool creation, even when quota is available. This is a UI-specific bug. As a temporary workaround, please use the command-line interface (CLI) to successfully create nodepools. Our team is investigating the root cause and will deploy a fix as soon as possible.

A fix has been implemented. This incident is now resolved

We are currently investigating an issue impacting the ability to start Virtual Machines (VMs) in our us-southcentral1-a region. Our engineering team has identified a potential issue and is actively working on a resolution. Thank you for your patience as we work to restore full functionality.

This incident has been resolved.

We have successfully mitigated the issue affecting the us-east1-a region. The facility power disruption has been addressed, and the impacted Infiniband networking fabric and associated systems have returned to normal operation. All affected services are now stable, and full functionality has been restored. Our teams will continue to monitor the region closely to ensure continued stability. We appreciate your patience during this incident and apologize again for any disruption it may have caused.

We are continuing to investigate and mitigate the service degradation affecting our us-east1-a region, following a facility power disruption at our data center. Our teams remain in close coordination with the data center provider as they work to fully restore services. Recovery of critical systems remains our top priority. We sincerely apologize for the ongoing impact and appreciate your continued patience as we work to resolve the issue.

We're investigating a service degradation in our us-east1-a region, triggered by a facility power disruption at our data center. The primary impact is to the Infiniband networking fabric, which may cause intermittent errors or failures for multi-node, distributed workloads. Some customers may also experience individual virtual machines becoming unavailable. Our teams are working to identify all affected resources. Our engineering teams are actively working to stabilize the affected systems and mitigate the risk of further disruption. We are coordinating with our data center provider to support their remediation efforts and restore full service resiliency as quickly as possible. We apologize for any impact this is causing.

This incident is now resolved

A fix has been implemented, and we are monitoring the environment for now.

The issue has been identified, and we have tested a fix internally. We are working on rolling out the fix to our A100 Infiniband type servers now.

We have identified an issue that is preventing new or restarted Virtual Machines from booting successfully on our A100 Infiniband hardware fleet. Any new VM provisioning request for this hardware type will also fail. Additionally, any existing VM on an A100 Infiniband node that is stopped and started (or rebooted) will also fail to come back online. Existing, currently running VMs are not affected and will continue to operate normally. We advise customers to avoid rebooting critical workloads on this hardware until a resolution is in place. Our engineering teams are actively investigating the root cause and are working to restore normal provisioning operations as quickly as possible.

This incident has been resolved.

A fix has been implemented for the persistent disk storage system. We are observing recovery, and affected virtual machines in eu-iceland1 are returning to a healthy state. Our team will continue to monitor the platform to ensure full service restoration and stability. We will provide the next update once the incident is fully resolved. We appreciate your understanding and support. If you experience any issues, please don't hesitate to contact us at support@crusoecloud.com.

We have identified an issue with our persistent disk storage system for a subset of VM's in eu-iceland1. Our engineers are now working to restore service.

We are currently investigating an issue affecting VM's in eu-iceland1

This incident has been resolved.

A fix has been applied and the outage has been mitigated. Services have returned to normal operations in US-Southcentral region. We are continuing to monitor the situation and validate preventive measures. Please contact support@crusoecloud.com if you are still experiencing any issues.

We've identified a potential issue on the control plane network and we're working to address the issue currently.

We're currently investigating an issue impacting our control plane in the US-Southcentral region. Customers may experience errors or timeouts when making certain API calls, including those for creating new virtual machines. Our engineering team is investigating the underlying cause with the highest priority. We appreciate your patience and will provide another update as soon as we have more information.

This incident has been resolved.

A fix has been applied and the outage has been mitigated. Services have returned to normal operations in us-northcentral1-a region. We are continuing to monitor the situation and validate preventive measures. Please contact support@crusoecloud.com if you are still experiencing any issues.

We are seeing signs of recovery, and some previously impacted VMs in the us-northcentral1-a region are now available. Our teams are closely monitoring the environment and continuing their work with our vendor to restore full service for all remaining affected instances.

We are continuing our dedicated investigation into the intermittent storage connectivity issues impacting VM reachability in us-northcentral1-a. Our teams are working in close collaboration with our storage vendor to analyze the underlying cause. We recognize the impact this is having and have all necessary resources allocated to resolving this.

Our team is continuing its in-depth investigation into the intermittent storage connectivity issues impacting VM reachability in the us-northcentral1-a region, in collaboration with our vendor. We will keep you posted on further developments

We are continuing our in-depth investigation into the intermittent storage connectivity issues impacting VM reachability in the us-northcentral1-a region. We will provide more information as it becomes available.

Our teams are working diligently and believe we have identified the root cause and we are working on implementing a resolution. We will continue to provide updates on this page as further developments occur.

We are currently investigating a potential service interruption that may impact VMs to be unreachable in the us-northcentral1-a region. If you are experiencing issues in this region, please reach out to support@crusoecloud.com

All Crusoe Cloud services have been restored and are fully operational. We will continue to monitor our platform. If you notice any issues please reach out to support@crusoecloud.com

We're starting to see signs of recovery across affected services, and our team is monitoring the situation very closely.

We are expanding the scope of this incident to include the Crusoe console being inaccessible. This means that users are currently unable to log in, view, or manage resources through the Crusoe console interface. We are investigating this alongside the existing issues.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are currently experiencing an outage with our 3rd party service provider, which is intermittently impacting the ability to provision and manage Virtual Machines (VMs) across all of our regions.

We are marking this incident as resolved. Please reach out to support@crusoecloud.com if you experience any issues.

We have identified the issue and a fix has been implemented. Currently, there is no further impact expected to virtual machine accessibility. We are currently monitoring the environment.

We are currently investigating an issue impacting virtual machine accessibility. Some users may be unable to stop and start Virtual Machines.

No further issues have been observed and this incident is now resolved.

We have applied fixes associated with the network performance degradation issue and are monitoring the changes.

We are still continuing to work on remediation steps.

We have identified the issue and are working on remediation steps.

We are currently investigating an issue with our networking in the eu-iceland1 region where customers are experiencing intermittent network degraded performance.

All impacted instances have successfully returned to a running state. A fix has been applied across the us-northcentral1 region to prevent this issue from recurring. Please reach out to support@crusoecloud.com if you experience any issues.

All impacted instances have now successfully returned to a Running state. No further impact is expected. We appreciate your patience and understanding while we worked through this issue.

We are actively working with our storage vendor to address the issue. We have identified the underlying cause and are beginning to see some VMs successfully return to a RUNNING state. Troubleshooting efforts are ongoing, and we will continue to provide updates as progress is made.

We are currently investigating an issue impacting virtual machine accessibility in us-northcentral1-a. Some users may be unable to access their VMs at this time. We appreciate your patience and will provide updates as more information becomes available.