Scale AI Status · StatusForAI

MajorJan 6, 10:11 PM

Degraded performance

Started: Mon, Jan 6, 2025, 07:34:35 PM
Updated: Mon, Jan 6, 2025, 10:11:53 PM
Resolved: Mon, Jan 6, 2025, 10:11:53 PM
Duration: 2h 37m

resolvedMon, Jan 6, 2025, 10:11:53 PM
This incident has been resolved.
investigatingMon, Jan 6, 2025, 07:40:18 PM
We are continuing to investigate this issue.
investigatingMon, Jan 6, 2025, 07:34:35 PM
We are currently investigating this issue.

MinorApr 26, 08:01 PM

Site Outage

Started: Fri, Apr 26, 2024, 07:24:40 PM
Updated: Fri, Apr 26, 2024, 08:01:05 PM
Resolved: Fri, Apr 26, 2024, 08:01:05 PM
Duration: 36m

resolvedFri, Apr 26, 2024, 08:01:05 PM
This incident has been resolved.
investigatingFri, Apr 26, 2024, 07:24:40 PM
We are currently investigating this issue.

MinorMar 19, 01:22 PM

Nucleus Degraded Performance

Started: Tue, Mar 19, 2024, 10:08:14 AM
Updated: Tue, Mar 19, 2024, 01:22:03 PM
Resolved: Tue, Mar 19, 2024, 01:22:03 PM
Duration: 3h 13m

resolvedTue, Mar 19, 2024, 01:22:03 PM
This incident has been resolved.
monitoringTue, Mar 19, 2024, 12:37:45 PM
A fix has been implemented and we are monitoring the results.
investigatingTue, Mar 19, 2024, 10:08:14 AM
Nucleus and Data Engine is currently experiencing degraded performance. Pages and queries will take longer to respond than usual.

CriticalJan 16, 02:13 AM

Cloudflare Outage

Started: Mon, Jan 15, 2024, 07:54:20 PM
Updated: Tue, Jan 16, 2024, 02:13:41 AM
Resolved: Tue, Jan 16, 2024, 02:13:41 AM
Duration: 6h 19m

resolvedTue, Jan 16, 2024, 02:13:41 AM
This incident has been resolved.
investigatingMon, Jan 15, 2024, 07:55:32 PM
Confirmed that the issue has been resolved.
investigatingMon, Jan 15, 2024, 07:55:00 PM
Cloudflare has implemented a fix and seems to be resolving any certificate issues.
investigatingMon, Jan 15, 2024, 07:54:20 PM
We are currently investigating this issue.

CriticalNov 20, 08:31 PM

Donovan Web Application Outage

Started: Mon, Nov 20, 2023, 07:59:31 PM
Updated: Mon, Nov 20, 2023, 08:31:22 PM
Resolved: Mon, Nov 20, 2023, 08:31:22 PM
Duration: 31m

resolvedMon, Nov 20, 2023, 08:31:22 PM
This incident has been resolved.
monitoringMon, Nov 20, 2023, 08:27:09 PM
A fix has been implemented and we are monitoring the results.
investigatingMon, Nov 20, 2023, 07:59:31 PM
We are currently investigating an application issue. We will post updates shortly.

NoneOct 23, 08:38 PM

Site Outage

Started: Mon, Oct 23, 2023, 07:29:43 PM
Updated: Mon, Oct 23, 2023, 08:38:22 PM
Resolved: Mon, Oct 23, 2023, 08:38:22 PM
Duration: 1h 8m

resolvedMon, Oct 23, 2023, 08:38:22 PM
This incident has been resolved.
monitoringMon, Oct 23, 2023, 08:25:13 PM
A fix has been implemented and we are monitoring the results.
investigatingMon, Oct 23, 2023, 07:29:43 PM
We are currently investigating an issue, some operations are experiencing high latency or are failing.

CriticalSep 19, 05:50 PM

Elevated Error Rate

Started: Mon, Sep 18, 2023, 06:35:05 PM
Updated: Tue, Sep 19, 2023, 05:50:25 PM
Resolved: Tue, Sep 19, 2023, 05:50:25 PM
Duration: 23h 15m

resolvedTue, Sep 19, 2023, 05:50:25 PM
This incident has been resolved.
monitoringTue, Sep 19, 2023, 12:52:26 AM
We are continuing to monitor for any further issues.
monitoringTue, Sep 19, 2023, 12:30:48 AM
A temporary fix has been implemented and we are monitoring the status of the platform
identifiedMon, Sep 18, 2023, 11:25:45 PM
We are continuing to work on mitigating the issue. Our website is experiencing a technical issue with the hosting provider. Current ETA for resolution is ~2 hours, or around 6:30 PT.
identifiedMon, Sep 18, 2023, 07:09:07 PM
We are seeing elevated error rates with our cloud provider. We are currently investigating the issue.
investigatingMon, Sep 18, 2023, 06:35:05 PM
We are currently investigating this issue.

CriticalMay 17, 12:27 AM

Spellbook Application Error

Started: Tue, May 16, 2023, 11:12:22 PM
Updated: Wed, May 17, 2023, 12:27:58 AM
Resolved: Wed, May 17, 2023, 12:27:58 AM
Duration: 1h 15m

resolvedWed, May 17, 2023, 12:27:58 AM
The Spellbook application is fully back up and operational.
identifiedTue, May 16, 2023, 11:48:33 PM
The fix for the deployment is in progress now.
identifiedTue, May 16, 2023, 11:12:22 PM
We have identified a platform issue impacting all Spellbook applications. Applications will not appear in the UI or be available over the API. We are deploying a fix for this issue.

CriticalApr 28, 05:40 PM

Catalog Forge Platform Unresponsive

Started: Fri, Apr 28, 2023, 02:26:28 PM
Updated: Fri, Apr 28, 2023, 05:40:47 PM
Resolved: Fri, Apr 28, 2023, 05:40:47 PM
Duration: 3h 14m

resolvedFri, Apr 28, 2023, 05:40:47 PM
This incident has been resolved.
monitoringFri, Apr 28, 2023, 03:59:11 PM
A fix has been implemented and we are monitoring the results.
investigatingFri, Apr 28, 2023, 02:26:28 PM
We are currently investigating this issue.

NoneMar 30, 05:19 PM

ML Pod Startup Failures

Started: Thu, Mar 30, 2023, 02:30:00 PM
Updated: Thu, Mar 30, 2023, 05:19:31 PM
Resolved: Thu, Mar 30, 2023, 02:30:00 PM
Duration: 0m

resolvedThu, Mar 30, 2023, 05:13:28 PM
There was an outage with a service that manages the metadata for models from 7:06am PT to 7:50am PT. It prevented model services from starting up new pods. ML services that received security updates, deployed a new version, or attempted to scale up were impacted. There was an additional 8 to 12 minutes of service degradation after restoring the impacted ML services to catch up on missed requests.

CriticalMar 11, 12:34 PM

Site Outage

Started: Sat, Mar 11, 2023, 10:58:24 AM
Updated: Sat, Mar 11, 2023, 12:34:34 PM
Resolved: Sat, Mar 11, 2023, 12:34:34 PM
Duration: 1h 36m

resolvedSat, Mar 11, 2023, 12:34:34 PM
This incident has been resolved.
monitoringSat, Mar 11, 2023, 12:20:44 PM
A fix has been implemented and we are monitoring the results.
investigatingSat, Mar 11, 2023, 10:58:24 AM
We are currently investigating this issue.

MinorMar 4, 01:41 AM

Elevated latencies

Started: Fri, Mar 3, 2023, 11:59:09 PM
Updated: Sat, Mar 4, 2023, 01:41:20 AM
Resolved: Sat, Mar 4, 2023, 01:41:20 AM
Duration: 1h 42m

resolvedSat, Mar 4, 2023, 01:41:20 AM
This incident has been resolved.
monitoringSat, Mar 4, 2023, 12:15:36 AM
A fix has been implemented and we are monitoring the results.
investigatingFri, Mar 3, 2023, 11:59:09 PM
We are currently investigating this issue.

MajorMar 3, 02:45 AM

Elevated API Error Rates

Started: Fri, Mar 3, 2023, 01:30:13 AM
Updated: Fri, Mar 3, 2023, 02:45:06 AM
Resolved: Fri, Mar 3, 2023, 02:45:06 AM
Duration: 1h 14m

resolvedFri, Mar 3, 2023, 02:45:06 AM
This incident has been resolved.
investigatingFri, Mar 3, 2023, 02:19:13 AM
We're experiencing an elevated level of API errors and are currently looking into the issue.

MajorJan 27, 08:57 PM

Catalog Forge slow or unresponsive

Started: Fri, Jan 27, 2023, 07:31:25 PM
Updated: Fri, Jan 27, 2023, 08:57:12 PM
Resolved: Fri, Jan 27, 2023, 08:57:12 PM
Duration: 1h 25m

resolvedFri, Jan 27, 2023, 08:57:12 PM
This incident has been resolved.
monitoringFri, Jan 27, 2023, 08:42:47 PM
A fix has been implemented and we are monitoring the results.
investigatingFri, Jan 27, 2023, 07:31:25 PM
We are currently investigating this issue.

MajorDec 30, 11:35 PM

Degraded Site Performance

Started: Thu, Dec 29, 2022, 07:08:16 AM
Updated: Fri, Dec 30, 2022, 11:35:21 PM
Resolved: Fri, Dec 30, 2022, 12:32:59 AM
Duration: 17h 24m

resolvedFri, Dec 30, 2022, 11:32:59 PM
This incident has been resolved.
monitoringThu, Dec 29, 2022, 10:36:31 PM
A fix has been implemented and we are monitoring the results.
identifiedThu, Dec 29, 2022, 11:57:06 AM
Many users may still be experiencing slow application loading times and less responsiveness. We have identified that our database has been experiencing internal issues and are continuing to work to resolve them. We have made efforts to mitigate the impact caused and have tuned our database to optimize for the current situation.
investigatingThu, Dec 29, 2022, 07:08:16 AM
We are currently investigating this issue.

NoneDec 29, 09:03 PM

ML Inference Failures

Started: Thu, Dec 29, 2022, 07:00:00 PM
Updated: Thu, Dec 29, 2022, 09:03:00 PM
Resolved: Thu, Dec 29, 2022, 07:00:00 PM
Duration: 0m

resolvedThu, Dec 29, 2022, 09:03:00 PM
Networking issue with ML infra.

CriticalDec 1, 12:27 PM

Platform slow or unresponsive

Started: Thu, Dec 1, 2022, 08:42:35 AM
Updated: Thu, Dec 1, 2022, 12:27:32 PM
Resolved: Thu, Dec 1, 2022, 12:27:32 PM
Duration: 3h 44m

resolvedThu, Dec 1, 2022, 12:27:32 PM
This incident has been resolved.
monitoringThu, Dec 1, 2022, 12:13:27 PM
We are continuing to monitor for any further issues.
monitoringThu, Dec 1, 2022, 11:52:34 AM
A fix has been implemented and we are monitoring the results.
investigatingThu, Dec 1, 2022, 11:41:15 AM
We are continuing to investigate this issue.
investigatingThu, Dec 1, 2022, 08:42:35 AM
We are currently investigating this issue.

CriticalNov 27, 02:09 PM

Platform Slow or Unresponsive

Started: Sun, Nov 27, 2022, 01:24:03 PM
Updated: Sun, Nov 27, 2022, 02:09:06 PM
Resolved: Sun, Nov 27, 2022, 02:09:06 PM
Duration: 45m

resolvedSun, Nov 27, 2022, 02:09:06 PM
This incident has been resolved.
investigatingSun, Nov 27, 2022, 01:24:03 PM
We are seeing elevated error rates that may indicate a slow or unresponsive platform. We are currently investigating the issue.

MajorNov 23, 06:15 PM

Platform slow or unresponsive

Started: Wed, Nov 23, 2022, 05:04:52 PM
Updated: Wed, Nov 23, 2022, 06:15:21 PM
Resolved: Wed, Nov 23, 2022, 06:15:21 PM
Duration: 1h 10m

resolvedWed, Nov 23, 2022, 06:15:21 PM
Incident has been resolved.
investigatingWed, Nov 23, 2022, 05:06:33 PM
We are continuing to investigate this issue.
investigatingWed, Nov 23, 2022, 05:04:52 PM
We are currently investigating this issue.

MinorNov 10, 07:01 PM

Degraded performance

Started: Thu, Nov 10, 2022, 01:38:33 PM
Updated: Thu, Nov 10, 2022, 07:01:35 PM
Resolved: Thu, Nov 10, 2022, 07:01:35 PM
Duration: 5h 23m

resolvedThu, Nov 10, 2022, 07:01:35 PM
This incident has been resolved.
monitoringThu, Nov 10, 2022, 06:45:06 PM
A fix has been implemented and we are monitoring the results.
investigatingThu, Nov 10, 2022, 01:38:33 PM
We are currently investigating an issue with our asynchronous jobs.

CriticalNov 5, 05:00 AM

Site Outage

Started: Sat, Nov 5, 2022, 01:54:13 AM
Updated: Sat, Nov 5, 2022, 05:00:16 AM
Resolved: Sat, Nov 5, 2022, 05:00:16 AM
Duration: 3h 6m

resolvedSat, Nov 5, 2022, 05:00:16 AM
This incident has been resolved.
investigatingSat, Nov 5, 2022, 02:13:50 AM
We are continuing to investigate this issue.
investigatingSat, Nov 5, 2022, 01:54:13 AM
We are currently investigating the issue.

CriticalNov 1, 01:50 AM

Site Outage

Started: Tue, Nov 1, 2022, 12:22:41 AM
Updated: Tue, Nov 1, 2022, 01:50:21 AM
Resolved: Tue, Nov 1, 2022, 01:50:21 AM
Duration: 1h 27m

resolvedTue, Nov 1, 2022, 01:50:21 AM
This incident has been resolved.
monitoringTue, Nov 1, 2022, 01:40:05 AM
We are starting to see signs of recovery and are monitoring for issues.
investigatingTue, Nov 1, 2022, 12:22:41 AM
We are currently investigating an issue, jobs may be affected across all build types.

NoneOct 14, 07:38 PM

Error processing signed S3 attachment URLs

Started: Fri, Oct 14, 2022, 06:00:00 PM
Updated: Fri, Oct 14, 2022, 07:38:39 PM
Resolved: Fri, Oct 14, 2022, 06:00:00 PM
Duration: 0m

resolvedFri, Oct 14, 2022, 07:38:39 PM
A fix has been implemented and we are monitoring the results.

CriticalOct 13, 04:45 AM

ML Infrastructure Latency Issues

Started: Thu, Oct 13, 2022, 12:01:25 AM
Updated: Thu, Oct 13, 2022, 04:45:39 AM
Resolved: Thu, Oct 13, 2022, 04:45:39 AM
Duration: 4h 44m

resolvedThu, Oct 13, 2022, 04:45:39 AM
This incident has been resolved.
monitoringThu, Oct 13, 2022, 04:34:18 AM
A fix has been implemented and we are monitoring the results.
identifiedThu, Oct 13, 2022, 02:34:15 AM
The issue has been identified and a fix is being implemented.
investigatingThu, Oct 13, 2022, 12:01:25 AM
We are currently investigating this issue.

MajorOct 10, 07:23 PM

Site incident

Started: Mon, Oct 10, 2022, 05:20:02 PM
Updated: Mon, Oct 10, 2022, 07:23:11 PM
Resolved: Mon, Oct 10, 2022, 07:23:11 PM
Duration: 2h 3m

resolvedMon, Oct 10, 2022, 07:23:11 PM
This incident has been resolved.
monitoringMon, Oct 10, 2022, 06:47:58 PM
We are continuing to monitor for any further issues.
monitoringMon, Oct 10, 2022, 06:33:53 PM
A fix has been implemented and we are monitoring the results.
investigatingMon, Oct 10, 2022, 06:28:26 PM
We are continuing to investigate this issue.
investigatingMon, Oct 10, 2022, 05:20:02 PM
We are currently investigating this issue.

MajorOct 5, 04:10 PM

Platform slow or unresponsive

Started: Wed, Oct 5, 2022, 02:55:31 PM
Updated: Wed, Oct 5, 2022, 04:10:31 PM
Resolved: Wed, Oct 5, 2022, 04:10:30 PM
Duration: 1h 14m

resolvedWed, Oct 5, 2022, 04:10:30 PM
This incident has been resolved.
monitoringWed, Oct 5, 2022, 03:25:28 PM
A fix has been implemented and we are monitoring the results.
investigatingWed, Oct 5, 2022, 02:55:31 PM
We are currently investigating this issue.

CriticalSep 15, 11:05 PM

Platform Slow or Unresponsive

Started: Thu, Sep 15, 2022, 10:24:33 PM
Updated: Thu, Sep 15, 2022, 11:05:04 PM
Resolved: Thu, Sep 15, 2022, 10:29:03 PM
Duration: 4m

resolvedThu, Sep 15, 2022, 10:29:03 PM
Scale AI is now operational
investigatingThu, Sep 15, 2022, 10:24:34 PM
We are seeing elevated error rates that may indicate a slow or unresponsive platform. We are currently investigating the issue.

CriticalMay 21, 01:39 AM

Platform Slow or Unresponsive

Started: Sat, May 21, 2022, 12:50:22 AM
Updated: Sat, May 21, 2022, 01:39:53 AM
Resolved: Sat, May 21, 2022, 01:39:53 AM
Duration: 49m

resolvedSat, May 21, 2022, 01:39:53 AM
This incident has been resolved.
monitoringSat, May 21, 2022, 01:21:35 AM
A fix has been implemented and we are monitoring the results.
identifiedSat, May 21, 2022, 01:08:54 AM
The issue has been identified and a fix is being implemented.
investigatingSat, May 21, 2022, 12:50:22 AM
We are currently investigating this issue.

CriticalApr 22, 03:40 AM

Increased site latency & error rate

Started: Fri, Apr 22, 2022, 02:17:53 AM
Updated: Fri, Apr 22, 2022, 03:40:04 AM
Resolved: Fri, Apr 22, 2022, 03:40:04 AM
Duration: 1h 22m

resolvedFri, Apr 22, 2022, 03:40:04 AM
This incident has been resolved.
monitoringFri, Apr 22, 2022, 03:21:28 AM
A fix has been implemented and we are monitoring the results.
identifiedFri, Apr 22, 2022, 03:01:50 AM
We are continuing to work on a fix for this issue.
identifiedFri, Apr 22, 2022, 02:17:54 AM
The issue has been identified and a fix is being implemented.

CriticalApr 18, 07:20 PM

Platform Slow or Unresponsive

Started: Mon, Apr 18, 2022, 05:57:31 PM
Updated: Mon, Apr 18, 2022, 07:20:45 PM
Resolved: Mon, Apr 18, 2022, 07:20:45 PM
Duration: 1h 23m

resolvedMon, Apr 18, 2022, 07:20:45 PM
This incident has been resolved.
monitoringMon, Apr 18, 2022, 07:07:55 PM
A fix has been implemented and we are monitoring the results.
identifiedMon, Apr 18, 2022, 07:03:34 PM
The issue has been identified and a fix is being implemented.
investigatingMon, Apr 18, 2022, 06:44:27 PM
We are actively working with our underlying service providers to resolve the issues impacting Scale's service.
investigatingMon, Apr 18, 2022, 05:57:31 PM
We are seeing elevated error rates that may indicate a slow or unresponsive platform. We are currently investigating the issue and its potential impact.

CriticalApr 15, 10:16 PM

Platform Slow or Unresponsive

Started: Fri, Apr 15, 2022, 06:20:54 PM
Updated: Fri, Apr 15, 2022, 10:16:01 PM
Resolved: Fri, Apr 15, 2022, 10:16:01 PM
Duration: 3h 55m

resolvedFri, Apr 15, 2022, 10:16:01 PM
This incident has been resolved.
monitoringFri, Apr 15, 2022, 09:57:44 PM
A fix has been implemented and we are monitoring the results.
identifiedFri, Apr 15, 2022, 09:35:54 PM
We are continuing to work with our underlying service providers to resolve the issues impacting Scale's service.
identifiedFri, Apr 15, 2022, 09:00:29 PM
We are actively working with our underlying service providers to resolve the issues impacting Scale's service.
investigatingFri, Apr 15, 2022, 08:09:33 PM
We are continuing to investigate the issue.
investigatingFri, Apr 15, 2022, 06:20:54 PM
We are seeing elevated error rates that may indicate a slow or unresponsive platform. We are currently investigating the issue.

CriticalApr 15, 03:00 AM

Platform Slow or Unresponsive

Started: Fri, Apr 15, 2022, 01:35:43 AM
Updated: Fri, Apr 15, 2022, 03:00:29 AM
Resolved: Fri, Apr 15, 2022, 03:00:29 AM
Duration: 1h 24m

resolvedFri, Apr 15, 2022, 03:00:29 AM
This incident has been resolved.
monitoringFri, Apr 15, 2022, 02:21:28 AM
A fix has been implemented and we are monitoring the results.
investigatingFri, Apr 15, 2022, 01:35:43 AM
We are seeing elevated error rates that may indicate a slow or unresponsive platform. We are currently investigating the issue.

CriticalApr 4, 06:41 AM

Platform Slow or Unresponsive

Started: Mon, Apr 4, 2022, 12:55:17 AM
Updated: Mon, Apr 4, 2022, 06:41:27 AM
Resolved: Mon, Apr 4, 2022, 06:41:27 AM
Duration: 5h 46m

resolvedMon, Apr 4, 2022, 06:41:27 AM
This incident has been resolved.
monitoringMon, Apr 4, 2022, 06:36:27 AM
We are continuing to monitor for any further issues.
monitoringMon, Apr 4, 2022, 04:37:43 AM
Site is unstable again and we're trying to implement a fix
monitoringMon, Apr 4, 2022, 03:38:13 AM
A fix has been implemented and we are monitoring the results.
investigatingMon, Apr 4, 2022, 12:55:17 AM
We are currently investigating this issue.

MajorMar 4, 11:52 PM

Platform Slow or Unresponsive

Started: Fri, Mar 4, 2022, 10:00:11 PM
Updated: Fri, Mar 4, 2022, 11:52:02 PM
Resolved: Fri, Mar 4, 2022, 11:52:02 PM
Duration: 1h 51m

resolvedFri, Mar 4, 2022, 11:52:02 PM
This incident has been resolved.
monitoringFri, Mar 4, 2022, 11:51:58 PM
We are continuing to monitor for any further issues.
monitoringFri, Mar 4, 2022, 10:39:55 PM
A fix has been implemented and we are monitoring the results.
identifiedFri, Mar 4, 2022, 10:03:50 PM
The issue has been identified and a fix is being implemented.
investigatingFri, Mar 4, 2022, 10:00:11 PM
We are currently investigating this issue.

CriticalFeb 28, 10:49 PM

Platform Slow or Unresponsive

Started: Mon, Feb 28, 2022, 07:00:09 PM
Updated: Mon, Feb 28, 2022, 10:49:33 PM
Resolved: Mon, Feb 28, 2022, 10:49:32 PM
Duration: 3h 49m

resolvedMon, Feb 28, 2022, 10:49:32 PM
This incident has been resolved.
monitoringMon, Feb 28, 2022, 10:15:15 PM
A fix has been implemented and we are monitoring the results.
identifiedMon, Feb 28, 2022, 08:46:13 PM
The issue has been identified and a fix is being implemented.
investigatingMon, Feb 28, 2022, 08:45:09 PM
We are currently investigating this issue.

CriticalFeb 15, 02:31 PM

Platform availability issues

Started: Tue, Feb 15, 2022, 01:00:18 PM
Updated: Tue, Feb 15, 2022, 02:31:48 PM
Resolved: Tue, Feb 15, 2022, 02:29:30 PM
Duration: 1h 29m

resolvedTue, Feb 15, 2022, 02:29:30 PM
This incident has been resolved.
monitoringTue, Feb 15, 2022, 02:24:45 PM
A fix has been implemented and we are monitoring the results.
investigatingTue, Feb 15, 2022, 02:17:39 PM
We are continuing to investigate this issue.
investigatingTue, Feb 15, 2022, 02:16:58 PM
We are continuing to investigate this issue.
investigatingTue, Feb 15, 2022, 02:16:18 PM
We are currently investigating this issue.

CriticalFeb 11, 10:25 PM

Platform availability issues

Started: Fri, Feb 11, 2022, 09:00:38 PM
Updated: Fri, Feb 11, 2022, 10:25:42 PM
Resolved: Fri, Feb 11, 2022, 09:45:45 PM
Duration: 45m

resolvedFri, Feb 11, 2022, 10:12:45 PM
This incident has been resolved.
investigatingFri, Feb 11, 2022, 09:28:38 PM
We are currently investigating this issue.

CriticalJan 29, 12:57 AM

Platform Outage

Started: Sat, Jan 29, 2022, 12:20:38 AM
Updated: Sat, Jan 29, 2022, 12:57:42 AM
Resolved: Sat, Jan 29, 2022, 12:57:42 AM
Duration: 37m

resolvedSat, Jan 29, 2022, 12:57:42 AM
We have identified an issue related to database index performance. We have completed deploying a fix for this issue. The platform is up and running with non-elevated error rates.
investigatingSat, Jan 29, 2022, 12:41:16 AM
The platform is experiencing a degraded service level and we are continuing to investigate. API and Website error rates are elevated.
monitoringSat, Jan 29, 2022, 12:24:24 AM
The platform has recovered. We are actively monitoring and root causing the issue.
investigatingSat, Jan 29, 2022, 12:20:38 AM
We are currently experiencing a platform outage. All services are impacted, including the Scale Dashboard, API, and associated services. We are currently reviewing this issue.

MajorDec 25, 02:49 PM

Elevated error rates & increased latency

Started: Sat, Dec 25, 2021, 02:09:52 PM
Updated: Sat, Dec 25, 2021, 02:49:24 PM
Resolved: Sat, Dec 25, 2021, 02:49:24 PM
Duration: 39m

resolvedSat, Dec 25, 2021, 02:49:24 PM
This incident has been resolved.
monitoringSat, Dec 25, 2021, 02:43:24 PM
A fix has been implemented and we are monitoring the results.
identifiedSat, Dec 25, 2021, 02:24:12 PM
The issue has been identified and a fix is being implemented.
investigatingSat, Dec 25, 2021, 02:10:21 PM
We are continuing to investigate this issue.
investigatingSat, Dec 25, 2021, 02:09:52 PM
We are currently investigating this issue.

MajorDec 15, 05:25 PM

Elevated error rates

Started: Wed, Dec 15, 2021, 10:12:37 AM
Updated: Wed, Dec 15, 2021, 05:25:48 PM
Resolved: Wed, Dec 15, 2021, 12:15:05 PM
Duration: 2h 2m

resolvedWed, Dec 15, 2021, 05:25:05 PM
This incident has been resolved.
identifiedWed, Dec 15, 2021, 10:12:37 AM
We are experiencing elevated error rates across the platform

MajorNov 13, 01:08 AM

Platform Slow or Unresponsive

Started: Sat, Nov 13, 2021, 12:15:23 AM
Updated: Sat, Nov 13, 2021, 01:08:01 AM
Resolved: Sat, Nov 13, 2021, 01:00:54 AM
Duration: 45m

resolvedSat, Nov 13, 2021, 01:07:54 AM
This incident has been resolved.
monitoringSat, Nov 13, 2021, 01:07:42 AM
A fix has been implemented and we are monitoring the results.
investigatingSat, Nov 13, 2021, 01:07:23 AM
We are currently investigating this issue.

CriticalNov 5, 07:55 AM

Nucleus Database Offline

Started: Fri, Nov 5, 2021, 06:21:22 AM
Updated: Fri, Nov 5, 2021, 07:55:41 AM
Resolved: Fri, Nov 5, 2021, 07:55:41 AM
Duration: 1h 34m

resolvedFri, Nov 5, 2021, 07:55:41 AM
This incident has been resolved.
identifiedFri, Nov 5, 2021, 07:55:02 AM
We are continuing to work on a fix for this issue.
identifiedFri, Nov 5, 2021, 06:21:22 AM
The upgrade appears stuck. We're restoring the database.

MajorOct 29, 05:11 AM

Intermittent site errors

Started: Fri, Oct 29, 2021, 01:35:02 AM
Updated: Fri, Oct 29, 2021, 05:11:01 AM
Resolved: Fri, Oct 29, 2021, 05:11:01 AM
Duration: 3h 35m

resolvedFri, Oct 29, 2021, 05:11:01 AM
This incident has been resolved.
monitoringFri, Oct 29, 2021, 02:15:24 AM
We are continuing to monitor for any further issues.
monitoringFri, Oct 29, 2021, 02:15:07 AM
A fix has been implemented and we are monitoring the results.
identifiedFri, Oct 29, 2021, 01:35:03 AM
The issue has been identified and a fix is being implemented.

CriticalOct 14, 12:22 AM

Site down

Started: Wed, Oct 13, 2021, 11:40:34 PM
Updated: Thu, Oct 14, 2021, 12:22:29 AM
Resolved: Thu, Oct 14, 2021, 12:21:49 AM
Duration: 41m

resolvedThu, Oct 14, 2021, 12:21:49 AM
This incident has been resolved at 4:56PM PST
investigatingWed, Oct 13, 2021, 11:41:29 PM
We are investigating the issue.
investigatingWed, Oct 13, 2021, 11:41:16 PM
Site is not accessible around 3:55PM PST
investigatingWed, Oct 13, 2021, 11:40:34 PM
We are currently investigating this issue.

CriticalSep 16, 06:22 PM

Platform Slow or Unresponsive

Started: Thu, Sep 16, 2021, 05:39:34 AM
Updated: Thu, Sep 16, 2021, 06:22:37 PM
Resolved: Thu, Sep 16, 2021, 09:15:43 AM
Duration: 3h 36m

resolvedThu, Sep 16, 2021, 03:26:43 PM
This incident has been resolved.
monitoringThu, Sep 16, 2021, 09:10:39 AM
A fix has been implemented and we are monitoring the results.
identifiedThu, Sep 16, 2021, 08:09:12 AM
The issue has been identified and a fix is being implemented.
monitoringThu, Sep 16, 2021, 05:56:16 AM
A fix has been implemented and we are monitoring the results.
investigatingThu, Sep 16, 2021, 05:39:34 AM
We are currently investigating this issue.

CriticalSep 15, 04:00 PM

Platform Slow or Unresponsive

Started: Wed, Sep 15, 2021, 06:13:08 AM
Updated: Wed, Sep 15, 2021, 04:00:17 PM
Resolved: Wed, Sep 15, 2021, 09:00:55 AM
Duration: 2h 47m

resolvedWed, Sep 15, 2021, 03:59:55 PM
This incident has been resolved.
monitoringWed, Sep 15, 2021, 08:34:28 AM
A fix has been implemented and we are monitoring the results.
identifiedWed, Sep 15, 2021, 07:55:40 AM
The issue has been identified and a fix is being implemented.
investigatingWed, Sep 15, 2021, 06:13:08 AM
We are currently investigating this issue.

CriticalSep 14, 10:30 AM

Platform Slow or Unresponsive

Started: Tue, Sep 14, 2021, 09:45:45 AM
Updated: Tue, Sep 14, 2021, 10:30:00 AM
Resolved: Tue, Sep 14, 2021, 10:30:00 AM
Duration: 44m

resolvedTue, Sep 14, 2021, 10:30:00 AM
This incident has been resolved.
monitoringTue, Sep 14, 2021, 10:00:52 AM
A fix has been implemented and we are monitoring the results.
identifiedTue, Sep 14, 2021, 10:00:45 AM
The issue has been identified and a fix is being implemented.
investigatingTue, Sep 14, 2021, 09:45:45 AM
We are currently investigating this issue.

NoneSep 9, 05:02 AM

Platform Slow or Unresponsive

Started: Thu, Sep 9, 2021, 03:32:24 AM
Updated: Thu, Sep 9, 2021, 05:02:31 AM
Resolved: Thu, Sep 9, 2021, 05:02:31 AM
Duration: 1h 30m

resolvedThu, Sep 9, 2021, 05:02:31 AM
This incident has been resolved.
monitoringThu, Sep 9, 2021, 04:07:01 AM
A fix has been implemented and we are monitoring the results.
investigatingThu, Sep 9, 2021, 03:32:25 AM
We are currently investigating this issue. Instability starts around 7:15PM PST

NoneSep 7, 05:31 AM

Platform Slow or Unresponsive

Started: Tue, Sep 7, 2021, 04:39:32 AM
Updated: Tue, Sep 7, 2021, 05:31:03 AM
Resolved: Tue, Sep 7, 2021, 05:31:03 AM
Duration: 51m

resolvedTue, Sep 7, 2021, 05:31:03 AM
This incident has been resolved.
monitoringTue, Sep 7, 2021, 04:52:33 AM
We've identified the cause and error rate has returned to normal around 9:40PM. We'll keep monitoring the issue.
investigatingTue, Sep 7, 2021, 04:39:32 AM
We are seeing significantly higher latencies around 9PM and are investigating.

CriticalAug 7, 03:44 PM

Platform Slow or Unresponsive

Started: Sat, Aug 7, 2021, 02:54:45 AM
Updated: Sat, Aug 7, 2021, 03:44:24 PM
Resolved: Sat, Aug 7, 2021, 03:44:24 PM
Duration: 12h 49m

resolvedSat, Aug 7, 2021, 03:44:24 PM
This incident has been resolved.
monitoringSat, Aug 7, 2021, 05:18:19 AM
A fix has been implemented and we are monitoring the results.
investigatingSat, Aug 7, 2021, 02:54:45 AM
We are currently investigating this issue.

Overview

Components11

Incidents50

History 50

Degraded performance

Site Outage

Nucleus Degraded Performance

Cloudflare Outage

Donovan Web Application Outage

Site Outage

Elevated Error Rate

Spellbook Application Error

Catalog Forge Platform Unresponsive

ML Pod Startup Failures

Site Outage

Elevated latencies

Elevated API Error Rates

Catalog Forge slow or unresponsive

Degraded Site Performance

ML Inference Failures

Platform slow or unresponsive

Platform Slow or Unresponsive

Platform slow or unresponsive

Degraded performance

Site Outage

Site Outage

Error processing signed S3 attachment URLs

ML Infrastructure Latency Issues

Site incident

Platform slow or unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Increased site latency & error rate

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform availability issues

Platform availability issues

Platform Outage

Elevated error rates & increased latency

Elevated error rates

Platform Slow or Unresponsive

Nucleus Database Offline

Intermittent site errors

Site down

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Platform Slow or Unresponsive

Details

Related in Frontier LLMs