Baseten Status · StatusForAI

Major14h ago

Kimi K3 Model API outage

Started: Sun, Sep 20, 2026, 07:19:01 AM
Updated: Sun, Sep 20, 2026, 08:13:44 AM
Resolved: Sun, Sep 20, 2026, 08:13:44 AM
Duration: 54m

resolvedSun, Sep 20, 2026, 08:13:44 AM
This incident has been resolved.
monitoringSun, Sep 20, 2026, 08:04:39 AM
A fix has been implemented and we are monitoring the results.
identifiedSun, Sep 20, 2026, 07:59:08 AM
Calls to the US-hosted endpoint are resulting in errors. A fix is in progress.
identifiedSun, Sep 20, 2026, 07:30:12 AM
~20% of calls are resulting in errors. A fix is in progress.
identifiedSun, Sep 20, 2026, 07:19:01 AM
The issue has been identified and a fix is being implemented.

Minor5d ago

Elevated errors for models in a single India-based cluster

Started: Tue, Sep 15, 2026, 01:22:38 AM
Updated: Tue, Sep 15, 2026, 02:07:38 AM
Resolved: Tue, Sep 15, 2026, 02:07:38 AM
Duration: 44m

resolvedTue, Sep 15, 2026, 02:07:38 AM
This incident has been resolved.
monitoringTue, Sep 15, 2026, 01:39:27 AM
A fix has been implemented and we are monitoring the results.
investigatingTue, Sep 15, 2026, 01:22:38 AM
We are currently investigating this issue.

Minor6d ago

Partial outage on Kimi K3 model API inference (non-US only)

Started: Mon, Sep 14, 2026, 04:21:34 PM
Updated: Mon, Sep 14, 2026, 05:03:20 PM
Resolved: Mon, Sep 14, 2026, 05:03:20 PM
Duration: 41m

resolvedMon, Sep 14, 2026, 05:03:20 PM
This incident has been resolved.
investigatingMon, Sep 14, 2026, 04:51:34 PM
We are currently investigating this issue.

Minor11d ago

Partial outage on Nemotron Ultra Model API. Dedicated Inference is not affected

Started: Wed, Sep 9, 2026, 05:42:43 PM
Updated: Wed, Sep 9, 2026, 05:52:20 PM
Resolved: Wed, Sep 9, 2026, 05:49:06 PM
Duration: 6m

resolvedWed, Sep 9, 2026, 05:52:06 PM
This incident has been resolved.
monitoringWed, Sep 9, 2026, 05:45:56 PM
A fix has been implemented and we are monitoring the results.
investigatingWed, Sep 9, 2026, 05:42:43 PM
We are currently investigating this issue.

Major20d ago

Elevated inference 500s for models in a single US-based cluster

Started: Mon, Aug 31, 2026, 06:38:34 AM
Updated: Mon, Aug 31, 2026, 07:02:44 AM
Resolved: Mon, Aug 31, 2026, 07:02:44 AM
Duration: 24m

resolvedMon, Aug 31, 2026, 07:02:44 AM
This incident has been resolved.
monitoringMon, Aug 31, 2026, 06:57:28 AM
Fix is in place and traffic is back.
investigatingMon, Aug 31, 2026, 06:38:34 AM
We are currently investigating this issue.

Minor24d ago

Elevated errors for models in a single US cluster due to provider outage

Started: Thu, Aug 27, 2026, 09:17:41 PM
Updated: Thu, Aug 27, 2026, 09:31:55 PM
Resolved: Thu, Aug 27, 2026, 09:31:55 PM
Duration: 14m

resolvedThu, Aug 27, 2026, 09:31:55 PM
This incident has been resolved.
monitoringThu, Aug 27, 2026, 09:23:50 PM
Auto-recovery is in progress.
investigatingThu, Aug 27, 2026, 09:17:41 PM
We are currently investigating this issue.

Minor26d ago

Elevated errors for deepseek-ai/DeepSeek-V4-Flash-0731 on Model APIs. Dedicated Inference is not affected

Started: Tue, Aug 25, 2026, 04:17:49 PM
Updated: Tue, Aug 25, 2026, 04:59:02 PM
Resolved: Tue, Aug 25, 2026, 04:59:02 PM
Duration: 41m

resolvedTue, Aug 25, 2026, 04:59:02 PM
This incident has been resolved.
monitoringTue, Aug 25, 2026, 04:43:08 PM
A fix is in place and we are monitoring
investigatingTue, Aug 25, 2026, 04:17:49 PM
We are currently investigating this issue.

Minor26d ago

Elevated inference 500s for models in an EU cluster due to provider outage

Started: Tue, Aug 25, 2026, 12:34:49 PM
Updated: Tue, Aug 25, 2026, 01:15:33 PM
Resolved: Tue, Aug 25, 2026, 01:15:33 PM
Duration: 40m

resolvedTue, Aug 25, 2026, 01:15:33 PM
This incident has been resolved.
monitoringTue, Aug 25, 2026, 12:53:22 PM
A fix has been implemented and we are monitoring the results.
identifiedTue, Aug 25, 2026, 12:34:49 PM
Failover in progress.

MinorAug 19, 02:00 AM

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. Dedicated Inference is not affected

Started: Wed, Aug 19, 2026, 01:23:41 AM
Updated: Wed, Aug 19, 2026, 02:00:12 AM
Resolved: Wed, Aug 19, 2026, 02:00:12 AM
Duration: 36m

resolvedWed, Aug 19, 2026, 02:00:12 AM
This incident has been resolved.
monitoringWed, Aug 19, 2026, 01:45:35 AM
A fix has been implemented and we are monitoring the results.
investigatingWed, Aug 19, 2026, 01:23:41 AM
We are currently investigating this issue.

MinorAug 18, 08:21 PM

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. Dedicated Inference is not affected

Started: Tue, Aug 18, 2026, 07:04:18 PM
Updated: Tue, Aug 18, 2026, 08:21:42 PM
Resolved: Tue, Aug 18, 2026, 08:21:42 PM
Duration: 1h 17m

resolvedTue, Aug 18, 2026, 08:21:42 PM
This incident has been resolved.
monitoringTue, Aug 18, 2026, 08:00:05 PM
A fix has been implemented and we are monitoring the results.
investigatingTue, Aug 18, 2026, 07:04:18 PM
We are currently investigating this issue.

MinorAug 18, 05:43 PM

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. Dedicated Inference is not affected

Started: Tue, Aug 18, 2026, 05:24:01 PM
Updated: Tue, Aug 18, 2026, 05:43:58 PM
Resolved: Tue, Aug 18, 2026, 05:35:50 PM
Duration: 11m

resolvedTue, Aug 18, 2026, 05:43:50 PM
This incident has been resolved.
investigatingTue, Aug 18, 2026, 05:24:01 PM
We are currently investigating this issue.

MinorAug 18, 05:14 PM

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. Dedicated Inference is not affected

Started: Tue, Aug 18, 2026, 04:27:04 PM
Updated: Tue, Aug 18, 2026, 05:14:44 PM
Resolved: Tue, Aug 18, 2026, 05:14:44 PM
Duration: 47m

resolvedTue, Aug 18, 2026, 05:14:44 PM
This incident has been resolved.
investigatingTue, Aug 18, 2026, 04:27:04 PM
We are currently investigating this issue.

MinorAug 17, 05:41 PM

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic

Started: Mon, Aug 17, 2026, 04:56:06 PM
Updated: Mon, Aug 17, 2026, 05:41:45 PM
Resolved: Mon, Aug 17, 2026, 05:41:45 PM
Duration: 45m

resolvedMon, Aug 17, 2026, 05:41:45 PM
This incident has been resolved.
investigatingMon, Aug 17, 2026, 04:56:06 PM
We are currently investigating this issue.

MinorAug 17, 01:35 PM

Metrics degradation in UI and API. Inference not affected

Started: Mon, Aug 17, 2026, 01:11:10 PM
Updated: Mon, Aug 17, 2026, 01:35:10 PM
Resolved: Mon, Aug 17, 2026, 01:35:10 PM
Duration: 24m

resolvedMon, Aug 17, 2026, 01:35:10 PM
This incident has been resolved.
monitoringMon, Aug 17, 2026, 01:30:09 PM
A fix has been implemented and we are monitoring the results.
investigatingMon, Aug 17, 2026, 01:11:10 PM
We are currently investigating this issue.

MinorAug 2, 12:56 AM

Elevated inference 502s for a subset of models in a single US cluster

Started: Sat, Aug 1, 2026, 11:57:54 PM
Updated: Sun, Aug 2, 2026, 12:56:47 AM
Resolved: Sun, Aug 2, 2026, 12:41:11 AM
Duration: 43m

resolvedSun, Aug 2, 2026, 12:56:11 AM
This incident has been resolved.
identifiedSun, Aug 2, 2026, 12:06:27 AM
The issue has been identified and a fix is being implemented.
investigatingSat, Aug 1, 2026, 11:57:54 PM
This is due to a networking failure in a cloud provider.

MinorAug 1, 12:20 AM

Elevated inference 502s for a subset of models in a single US cluster

Started: Fri, Jul 31, 2026, 11:52:09 PM
Updated: Sat, Aug 1, 2026, 12:20:55 AM
Resolved: Sat, Aug 1, 2026, 12:20:55 AM
Duration: 28m

resolvedSat, Aug 1, 2026, 12:20:55 AM
This incident has been resolved.
monitoringSat, Aug 1, 2026, 12:07:09 AM
A fix has been implemented and we are monitoring the results.
investigatingFri, Jul 31, 2026, 11:52:09 PM
We are currently investigating this issue.

MinorJul 31, 04:58 PM

Slower than usual scale-ups for B200s

Started: Fri, Jul 31, 2026, 04:46:29 PM
Updated: Fri, Jul 31, 2026, 04:58:01 PM
Resolved: Fri, Jul 31, 2026, 04:58:01 PM
Duration: 11m

resolvedFri, Jul 31, 2026, 04:58:01 PM
This incident has been resolved.
investigatingFri, Jul 31, 2026, 04:46:29 PM
We are currently investigating this issue.

MinorJul 22, 06:22 PM

Elevated inference failures for models in an EU cluster

Started: Wed, Jul 22, 2026, 05:49:51 PM
Updated: Wed, Jul 22, 2026, 06:22:45 PM
Resolved: Wed, Jul 22, 2026, 06:22:45 PM
Duration: 32m

resolvedWed, Jul 22, 2026, 06:22:45 PM
This incident has been resolved.
identifiedWed, Jul 22, 2026, 06:05:22 PM
We are continuing to work on a fix for this issue.
identifiedWed, Jul 22, 2026, 05:49:51 PM
Failure in GPU provider's cooling system. Models are automatically migrating over to other EU clusters.

MinorJul 22, 06:04 PM

Slower than usual scale-ups in 1 us-central cluster

Started: Wed, Jul 22, 2026, 05:45:12 PM
Updated: Wed, Jul 22, 2026, 06:04:30 PM
Resolved: Wed, Jul 22, 2026, 06:04:30 PM
Duration: 19m

resolvedWed, Jul 22, 2026, 06:04:30 PM
This incident has been resolved.
identifiedWed, Jul 22, 2026, 05:45:12 PM
The issue has been identified and a fix is being implemented.

MajorJul 18, 11:12 PM

GLM 5.2 Model API partial outage due to provider network failure

Started: Sat, Jul 18, 2026, 10:40:12 PM
Updated: Sat, Jul 18, 2026, 11:12:32 PM
Resolved: Sat, Jul 18, 2026, 11:12:32 PM
Duration: 32m

resolvedSat, Jul 18, 2026, 11:12:32 PM
This incident has been resolved.
monitoringSat, Jul 18, 2026, 11:05:51 PM
We are continuing to monitor for any further issues.
monitoringSat, Jul 18, 2026, 11:05:22 PM
A fix has been implemented and we are monitoring the results.
identifiedSat, Jul 18, 2026, 10:40:12 PM
Escalated Dedicated inference is *not* affected.

MinorJul 18, 12:28 AM

Elevated inference errors in 1 Canada cluster due to provider network outage

Started: Sat, Jul 18, 2026, 12:22:55 AM
Updated: Sat, Jul 18, 2026, 12:28:34 AM
Resolved: Sat, Jul 18, 2026, 12:28:34 AM
Duration: 5m

resolvedSat, Jul 18, 2026, 12:28:34 AM
This incident has been resolved.
identifiedSat, Jul 18, 2026, 12:22:55 AM
The issue has been identified and a fix is being implemented.

MinorJul 16, 10:25 AM

Elevated errors in accessing the Baseten UI. Inference is not affected

Started: Thu, Jul 16, 2026, 09:06:26 AM
Updated: Thu, Jul 16, 2026, 10:25:13 AM
Resolved: Thu, Jul 16, 2026, 10:25:13 AM
Duration: 1h 18m

resolvedThu, Jul 16, 2026, 10:25:13 AM
This incident has been resolved.
monitoringThu, Jul 16, 2026, 10:14:07 AM
We are starting to see recovery and are continuing to monitor.
investigatingThu, Jul 16, 2026, 10:07:06 AM
We are continuing to investigate this issue.
investigatingThu, Jul 16, 2026, 09:06:26 AM
We are currently investigating this issue.

MajorJul 13, 02:10 AM

Partial Outage on Model APIs

Started: Mon, Jul 13, 2026, 12:04:01 AM
Updated: Mon, Jul 13, 2026, 02:10:34 AM
Resolved: Mon, Jul 13, 2026, 02:10:34 AM
Duration: 2h 6m

resolvedMon, Jul 13, 2026, 02:10:34 AM
Kimi-K2.6 recovered.
monitoringMon, Jul 13, 2026, 01:46:44 AM
Identified an infiniband outage in one of our providers. GLM-5.2 Recovered. Kimi-K2.6 recovering.
identifiedMon, Jul 13, 2026, 12:04:01 AM
Affecting Kimi-K2.6 and GLM-5.2.

MinorJul 9, 02:38 PM

Elevated inference 500s for models deployed in 1 US-east cluster

Started: Thu, Jul 9, 2026, 02:07:51 PM
Updated: Thu, Jul 9, 2026, 02:38:26 PM
Resolved: Thu, Jul 9, 2026, 02:28:16 PM
Duration: 20m

resolvedThu, Jul 9, 2026, 02:38:16 PM
This incident has been resolved.
monitoringThu, Jul 9, 2026, 02:24:42 PM
The provider had a network outage. Affected models are automatically moving to other clusters.
identifiedThu, Jul 9, 2026, 02:07:51 PM
The issue has been identified and a fix is being implemented.

MinorJul 8, 11:03 PM

Metrics degradation in UI and API. Inference not affected

Started: Wed, Jul 8, 2026, 04:50:15 PM
Updated: Wed, Jul 8, 2026, 11:03:09 PM
Resolved: Wed, Jul 8, 2026, 06:45:29 PM
Duration: 1h 55m

resolvedWed, Jul 8, 2026, 11:02:29 PM
This incident has been resolved.
investigatingWed, Jul 8, 2026, 11:02:15 PM
Incorrect values recorded for some metrics

MinorJul 6, 09:22 PM

Elevated inference 500s — network outage in our cloud provider in Delaware

Started: Mon, Jul 6, 2026, 09:08:27 PM
Updated: Mon, Jul 6, 2026, 09:22:15 PM
Resolved: Mon, Jul 6, 2026, 09:22:15 PM
Duration: 13m

resolvedMon, Jul 6, 2026, 09:22:15 PM
This incident has been resolved.
identifiedMon, Jul 6, 2026, 09:08:27 PM
The issue has been identified and a fix is being implemented.

MajorJul 3, 04:25 AM

Elevated errors for GLM 5.2 on Model APIs. Dedicated Inference is not affected

Started: Fri, Jul 3, 2026, 02:59:34 AM
Updated: Fri, Jul 3, 2026, 04:25:22 AM
Resolved: Fri, Jul 3, 2026, 04:25:22 AM
Duration: 1h 25m

resolvedFri, Jul 3, 2026, 04:25:22 AM
This incident has been resolved.
monitoringFri, Jul 3, 2026, 03:56:09 AM
A fix has been implemented and we are monitoring the results.
investigatingFri, Jul 3, 2026, 03:06:57 AM
We are continuing to investigate this issue.
investigatingFri, Jul 3, 2026, 02:59:34 AM
We are currently investigating this issue.

MinorJul 3, 12:36 AM

Elevated inference 500s for a subset of models deployed in 1 cluster

Started: Fri, Jul 3, 2026, 12:21:01 AM
Updated: Fri, Jul 3, 2026, 12:36:48 AM
Resolved: Fri, Jul 3, 2026, 12:36:48 AM
Duration: 15m

resolvedFri, Jul 3, 2026, 12:36:48 AM
This incident has been resolved.
identifiedFri, Jul 3, 2026, 12:36:42 AM
We are continuing to work on a fix for this issue.
identifiedFri, Jul 3, 2026, 12:21:01 AM
The issue has been identified and a fix is being implemented.

MinorJul 2, 11:13 PM

Degraded metrics performance in dashboard and API. Inference is not affected

Started: Thu, Jul 2, 2026, 10:00:12 PM
Updated: Thu, Jul 2, 2026, 11:13:51 PM
Resolved: Thu, Jul 2, 2026, 11:13:51 PM
Duration: 1h 13m

resolvedThu, Jul 2, 2026, 11:13:51 PM
This incident has been resolved.
monitoringThu, Jul 2, 2026, 10:53:00 PM
A fix has been implemented and we are monitoring the results.
investigatingThu, Jul 2, 2026, 10:35:12 PM
We are currently investigating this issue.

MajorJul 1, 08:53 PM

Elevated errors in a single EU cluster

Started: Wed, Jul 1, 2026, 10:30:46 AM
Updated: Wed, Jul 1, 2026, 08:53:51 PM
Resolved: Wed, Jul 1, 2026, 11:30:11 AM
Duration: 59m

resolvedWed, Jul 1, 2026, 08:52:11 PM
This incident has been resolved.
investigatingWed, Jul 1, 2026, 08:51:58 PM
We are continuing to investigate this issue.
investigatingWed, Jul 1, 2026, 08:51:46 PM
Elevated errors due to degradation in a single EU cluster

MinorJun 29, 08:11 PM

Elevated error rates on GLM 5.2 Model APIs. Dedicated Inference is not affected

Started: Mon, Jun 29, 2026, 06:26:41 PM
Updated: Mon, Jun 29, 2026, 08:11:15 PM
Resolved: Mon, Jun 29, 2026, 07:01:52 PM
Duration: 35m

resolvedMon, Jun 29, 2026, 08:08:52 PM
This incident has been resolved.
investigatingMon, Jun 29, 2026, 08:08:41 PM
Elevated error rates on GLM 5.2 Model APIs. **Dedicated Inference is not affected**

MinorJun 27, 06:50 AM

Elevated inference 500s for models in 1 cluster due to cloud provider outage

Started: Sat, Jun 27, 2026, 06:35:31 AM
Updated: Sat, Jun 27, 2026, 06:50:31 AM
Resolved: Sat, Jun 27, 2026, 06:50:31 AM
Duration: 15m

resolvedSat, Jun 27, 2026, 06:50:31 AM
This incident has been resolved.
monitoringSat, Jun 27, 2026, 06:38:02 AM
Models are auto-moving to other clusters
identifiedSat, Jun 27, 2026, 06:35:31 AM
The issue has been identified and a fix is being implemented.

MinorJun 27, 12:55 AM

Elevated model deployment failures. Inference is not affected

Started: Fri, Jun 26, 2026, 11:41:14 PM
Updated: Sat, Jun 27, 2026, 12:55:40 AM
Resolved: Sat, Jun 27, 2026, 12:55:40 AM
Duration: 1h 14m

resolvedSat, Jun 27, 2026, 12:55:40 AM
The incident has resolved. New model deployments are successfully coming up.
investigatingFri, Jun 26, 2026, 11:41:14 PM
New model deployments failing to become ready

MajorJun 26, 10:15 PM

Elevated errors for dedicated inference in a subset of clusters

Started: Fri, Jun 26, 2026, 10:00:38 PM
Updated: Fri, Jun 26, 2026, 10:15:23 PM
Resolved: Fri, Jun 26, 2026, 10:15:04 PM
Duration: 14m

resolvedFri, Jun 26, 2026, 10:15:04 PM
This incident has been resolved.
identifiedFri, Jun 26, 2026, 10:11:52 PM
We are continuing to work on a fix for this issue.
identifiedFri, Jun 26, 2026, 10:09:18 PM
Nearly all clusters have recovered.
identifiedFri, Jun 26, 2026, 10:05:19 PM
The issue has been identified and a fix is being implemented.
investigatingFri, Jun 26, 2026, 10:03:38 PM
We are currently investigating this issue.

MinorJun 23, 12:54 AM

Slower than usual scale-ups. High failure rate for new model deploys. Existing replicas aren't impacted.

Started: Tue, Jun 23, 2026, 12:10:14 AM
Updated: Tue, Jun 23, 2026, 12:54:43 AM
Resolved: Tue, Jun 23, 2026, 12:54:43 AM
Duration: 44m

resolvedTue, Jun 23, 2026, 12:54:43 AM
This incident has been resolved.
identifiedTue, Jun 23, 2026, 12:16:25 AM
We are continuing to work on a fix for this issue.
identifiedTue, Jun 23, 2026, 12:10:14 AM
The issue has been identified and a fix is being implemented.

MinorJun 18, 02:30 PM

Elevated errors for models in 1 Austin-based cluster affecting some models on B200s

Started: Thu, Jun 18, 2026, 12:46:34 PM
Updated: Thu, Jun 18, 2026, 02:30:17 PM
Resolved: Thu, Jun 18, 2026, 02:30:17 PM
Duration: 1h 43m

resolvedThu, Jun 18, 2026, 02:30:17 PM
This incident has been resolved.
monitoringThu, Jun 18, 2026, 01:40:51 PM
A fix has been implemented and we are monitoring the results.
identifiedThu, Jun 18, 2026, 12:46:34 PM
GLM 5.1 and Kimi k2.6 Model APIs are also affected.

MinorJun 17, 06:03 AM

Elevated errors for models in 1 Sweden-based cluster

Started: Wed, Jun 17, 2026, 04:40:04 AM
Updated: Wed, Jun 17, 2026, 06:03:13 AM
Resolved: Wed, Jun 17, 2026, 06:03:13 AM
Duration: 1h 23m

resolvedWed, Jun 17, 2026, 06:03:13 AM
This incident has been resolved.
identifiedWed, Jun 17, 2026, 04:44:00 AM
The issue has been identified and a fix is being implemented.
investigatingWed, Jun 17, 2026, 04:40:04 AM
We are currently investigating this issue.

MinorJun 15, 04:58 PM

Elevated errors for models in a single EU cloud provider

Started: Mon, Jun 15, 2026, 04:14:00 PM
Updated: Mon, Jun 15, 2026, 04:58:16 PM
Resolved: Mon, Jun 15, 2026, 04:58:16 PM
Duration: 44m

resolvedMon, Jun 15, 2026, 04:58:16 PM
This incident has been resolved.
monitoringMon, Jun 15, 2026, 04:54:36 PM
A fix has been implemented and we are monitoring the results.
investigatingMon, Jun 15, 2026, 04:14:00 PM
We are currently investigating this issue.

MinorJun 13, 07:09 PM

Elevated errors for models in a single cloud provider

Started: Sat, Jun 13, 2026, 06:25:24 PM
Updated: Sat, Jun 13, 2026, 07:09:52 PM
Resolved: Sat, Jun 13, 2026, 07:09:52 PM
Duration: 44m

resolvedSat, Jun 13, 2026, 07:09:52 PM
This incident has been resolved.
monitoringSat, Jun 13, 2026, 06:54:05 PM
A fix has been implemented and we are monitoring the results.
investigatingSat, Jun 13, 2026, 06:25:24 PM
Network outage in single cloud provider (https://status.vultr.com/), elevated errors for some models

MajorJun 5, 03:36 AM

Elevated errors in a single US-East cluster

Started: Fri, Jun 5, 2026, 02:10:16 AM
Updated: Fri, Jun 5, 2026, 03:36:17 AM
Resolved: Fri, Jun 5, 2026, 02:29:32 AM
Duration: 19m

resolvedFri, Jun 5, 2026, 03:31:32 AM
This incident has been resolved.
investigatingFri, Jun 5, 2026, 03:31:16 AM
We are currently investigating this issue.

MinorJun 4, 09:59 PM

Elevated latencies for a subset of models in a US-East cluster

Started: Thu, Jun 4, 2026, 08:53:22 PM
Updated: Thu, Jun 4, 2026, 09:59:44 PM
Resolved: Thu, Jun 4, 2026, 09:59:44 PM
Duration: 1h 6m

resolvedThu, Jun 4, 2026, 09:59:44 PM
This incident has been resolved.
monitoringThu, Jun 4, 2026, 09:09:32 PM
A fix has been implemented and we are monitoring the results.
identifiedThu, Jun 4, 2026, 09:05:02 PM
The issue has been identified and a fix is being implemented.
investigatingThu, Jun 4, 2026, 08:53:22 PM
We are currently investigating this issue.

MinorJun 3, 06:49 PM

Slower than usual scale-ups for some models using H100s

Started: Wed, Jun 3, 2026, 02:56:57 PM
Updated: Wed, Jun 3, 2026, 06:49:26 PM
Resolved: Wed, Jun 3, 2026, 05:11:42 PM
Duration: 2h 14m

resolvedWed, Jun 3, 2026, 06:48:42 PM
This incident has been resolved.
identifiedWed, Jun 3, 2026, 02:56:57 PM
The issue has been identified and a fix is being implemented.

MinorMay 29, 08:04 PM

Elevated async inference errors in a single cluster

Started: Fri, May 29, 2026, 07:31:24 PM
Updated: Fri, May 29, 2026, 08:04:13 PM
Resolved: Fri, May 29, 2026, 08:04:13 PM
Duration: 32m

resolvedFri, May 29, 2026, 08:04:13 PM
This incident has been resolved.
investigatingFri, May 29, 2026, 07:31:24 PM
We are currently investigating this issue.

MinorMay 26, 06:55 PM

Elevated errors in a single cluster

Started: Fri, May 15, 2026, 01:00:00 AM
Updated: Tue, May 26, 2026, 06:55:13 PM
Resolved: Fri, May 15, 2026, 03:00:00 AM
Duration: 2h

resolvedTue, May 26, 2026, 06:53:13 PM
This incident has been resolved.
investigatingTue, May 26, 2026, 06:52:57 PM
Elevated errors in our us-east1 cluster

MinorMay 26, 04:35 PM

Slower than usual scale-ups in 1 India cluster

Started: Tue, May 26, 2026, 04:21:51 PM
Updated: Tue, May 26, 2026, 04:35:27 PM
Resolved: Tue, May 26, 2026, 04:35:27 PM
Duration: 13m

resolvedTue, May 26, 2026, 04:35:27 PM
This incident has been resolved.
monitoringTue, May 26, 2026, 04:33:58 PM
A fix has been implemented and we are monitoring the results.
identifiedTue, May 26, 2026, 04:21:52 PM
The issue has been identified and a fix is being implemented.

MinorMay 20, 01:54 AM

Elevated errors for models in single region

Started: Wed, May 20, 2026, 01:40:54 AM
Updated: Wed, May 20, 2026, 01:54:13 AM
Resolved: Wed, May 20, 2026, 01:54:13 AM
Duration: 13m

resolvedWed, May 20, 2026, 01:54:13 AM
This incident has been resolved.
investigatingWed, May 20, 2026, 01:40:54 AM
We are investigating elevated errors and latency for models in a single region

NoneMay 18, 03:59 PM

Slower than usual scale-ups for L4s in US-East

Started: Mon, May 18, 2026, 03:20:36 PM
Updated: Mon, May 18, 2026, 03:59:17 PM
Resolved: Mon, May 18, 2026, 03:59:07 PM
Duration: 38m

resolvedMon, May 18, 2026, 03:59:07 PM
This incident has been resolved.
monitoringMon, May 18, 2026, 03:58:58 PM
A fix has been implemented and we are monitoring the results.
investigatingMon, May 18, 2026, 03:20:36 PM
We are currently investigating this issue.

NoneMay 16, 06:07 PM

Slower than usual scale-ups in EU due to a provider failure

Started: Sat, May 16, 2026, 03:32:02 PM
Updated: Sat, May 16, 2026, 06:07:08 PM
Resolved: Sat, May 16, 2026, 05:05:02 PM
Duration: 1h 32m

resolvedSat, May 16, 2026, 06:05:02 PM
This incident has been resolved.
identifiedSat, May 16, 2026, 03:32:02 PM
Root-cause has been identified and a fix is in progress.

NoneMay 16, 01:39 AM

Higher latencies for a subset of models in one cluster. Root caused and a fix is in progress

Started: Sat, May 16, 2026, 01:02:31 AM
Updated: Sat, May 16, 2026, 01:39:09 AM
Resolved: Sat, May 16, 2026, 01:39:06 AM
Duration: 36m

resolvedSat, May 16, 2026, 01:39:06 AM
The incident is resolved.
monitoringSat, May 16, 2026, 01:10:22 AM
Fix is in place. Monitoring.
identifiedSat, May 16, 2026, 01:02:31 AM
The issue has been identified and a fix is being implemented.

MajorMay 10, 03:26 AM

Training cluster is down due to a provider failure

Started: Sun, May 10, 2026, 12:30:56 AM
Updated: Sun, May 10, 2026, 03:26:54 AM
Resolved: Sun, May 10, 2026, 03:24:36 AM
Duration: 2h 53m

resolvedSun, May 10, 2026, 03:24:36 AM
The issue has been resolved.
monitoringSun, May 10, 2026, 03:18:45 AM
Monitoring the fix. Most nodes are back online, but not all.
identifiedSun, May 10, 2026, 03:08:56 AM
Root-caused and fix is underway. Inference is not affected.

Overview

Components6

Incidents50

History 50

Kimi K3 Model API outage

Elevated errors for models in a single India-based cluster

Partial outage on Kimi K3 model API inference (non-US only)

Partial outage on Nemotron Ultra Model API. **Dedicated Inference is not affected**

Elevated inference 500s for models in a single US-based cluster

Elevated errors for models in a single US cluster due to provider outage

Elevated errors for deepseek-ai/DeepSeek-V4-Flash-0731 on Model APIs. **Dedicated Inference is not affected**

Elevated inference 500s for models in an EU cluster due to provider outage

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. **Dedicated Inference is not affected**

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. **Dedicated Inference is not affected**

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. **Dedicated Inference is not affected**

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. **Dedicated Inference is not affected**

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic

Metrics degradation in UI and API. **Inference not affected**

Elevated inference 502s for a subset of models in a single US cluster

Elevated inference 502s for a subset of models in a single US cluster

Slower than usual scale-ups for B200s

Elevated inference failures for models in an EU cluster

Slower than usual scale-ups in 1 us-central cluster

GLM 5.2 Model API partial outage due to provider network failure

Elevated inference errors in 1 Canada cluster due to provider network outage

Elevated errors in accessing the Baseten UI. **Inference is not affected**

Partial Outage on Model APIs

Elevated inference 500s for models deployed in 1 US-east cluster

Metrics degradation in UI and API. **Inference not affected**

Elevated inference 500s — network outage in our cloud provider in Delaware

Elevated errors for GLM 5.2 on Model APIs. **Dedicated Inference is not affected**

Elevated inference 500s for a subset of models deployed in 1 cluster

Degraded metrics performance in dashboard and API. **Inference is not affected**

Elevated errors in a single EU cluster

Elevated error rates on GLM 5.2 Model APIs. **Dedicated Inference is not affected**

Elevated inference 500s for models in 1 cluster due to cloud provider outage

Elevated model deployment failures. **Inference is not affected**

Elevated errors for dedicated inference in a subset of clusters

Slower than usual scale-ups. High failure rate for new model deploys. Existing replicas aren't impacted.

Elevated errors for models in 1 Austin-based cluster affecting some models on B200s

Elevated errors for models in 1 Sweden-based cluster

Elevated errors for models in a single EU cloud provider

Elevated errors for models in a single cloud provider

Elevated errors in a single US-East cluster

Elevated latencies for a subset of models in a US-East cluster

Slower than usual scale-ups for some models using H100s

Elevated async inference errors in a single cluster

Elevated errors in a single cluster

Slower than usual scale-ups in 1 India cluster

Elevated errors for models in single region

Slower than usual scale-ups for L4s in US-East

Slower than usual scale-ups in EU due to a provider failure

Higher latencies for a subset of models in one cluster. Root caused and a fix is in progress

Training cluster is down due to a provider failure

Details

Related in Fast Inference

Partial outage on Nemotron Ultra Model API. Dedicated Inference is not affected

Elevated errors for deepseek-ai/DeepSeek-V4-Flash-0731 on Model APIs. Dedicated Inference is not affected

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. Dedicated Inference is not affected

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. Dedicated Inference is not affected

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. Dedicated Inference is not affected

Elevated errors for moonshotai/Kimi-K3 for US-pinned traffic on Model APIs. Dedicated Inference is not affected

Metrics degradation in UI and API. Inference not affected

Elevated errors in accessing the Baseten UI. Inference is not affected

Metrics degradation in UI and API. Inference not affected

Elevated errors for GLM 5.2 on Model APIs. Dedicated Inference is not affected

Degraded metrics performance in dashboard and API. Inference is not affected

Elevated error rates on GLM 5.2 Model APIs. Dedicated Inference is not affected

Elevated model deployment failures. Inference is not affected