Turbopuffer

From 7:21-7:41 UTC, the turbopuffer dashboard was unavailable due to an issue with our backing datastore. This issue has been resolved and dashboard access has been restored.

The turbopuffer dashboard is currently experiencing an outage. turbopuffer regions are fully available. Writes and queries are not affected.

The issue causing elevated 5xx errors for some namespaces has been resolved. We repaired the affected namespaces, and error rates have returned to normal. We will continue monitoring the service closely.

Some namespaces are experiencing an elevated rate of 5xx errors. We have identified the issue and are working to restore full service.

Service has been fully restored.

We’ve identified the problem and rolled out a fix. We’re closely monitoring the situation.

Full outage identified in gcp-southeast1

Our authentication provider has resolved the incident on their end, and dashboard logins have been fully operational since 10:08 UTC. Existing sessions and the database were unaffected throughout.

Logins are succeeding again as of 10:08 UTC. Our authentication provider is still working through their incident, so some logins may be slower than usual. We'll continue monitoring until it's fully resolved.

Dashboard logins (Google, SSO, and email) are intermittently failing due to an ongoing incident at [our authentication provider](https://status.workos.com/incidents/b52fkqjzcl20). We're monitoring and will update here.

We're aware of issues affecting some customers who are logging into the dashboard. We're investigating and will provide an update shortly. Note that the database is not affected - this issue is limited to the dashboard.

Starting at ~17:55 UTC, we experienced an elevated 500 rate affecting delete_by_filter, and filtered queries as a result of a recent change deployed to filtering. This change was reverted, and errors fully subsided by ~18:30 UTC. This incident affected approximately 38.2% of `delete_by_filter` requests in aws-us-east-1, and a small percentage of filtered queries in other regions.

Investigating elevated 500 error rate on filtered queries, delete_by_filter operations.

Our deploy has rolled out and affected namespaces have recovered.

Our team has identified an issue with a deploy in aws-us-west-2, causing queries and writes to 500 for affected namespaces. We are deploying a fix.

This incident is resolved. Dashboard access is working as usual. As noted earlier, this incident only affected the dashboard and had no impact on turbopuffer regions or namespaces!

Traffic has returned to normal. We're monitoring closely!

We're seeing recovery with traffic returning to normal. We're monitoring closely and will share another update shortly.

This issue is currently affecting the dashboard and API key management. **turbopuffer regions and namespaces are all operating normally**, and existing API keys continue to authenticate as expected. The root cause is a dependency service that we use for metadata. We'll share an update as we make progress.

We've confirmed reports of dashboard errors and are actively investigating. Note that the database is not affected - this issue is limited to the dashboard.

We were impacted by [an issue with one of our providers](https://www.vercel-status.com/incidents/5r9bp5y8rql2), which we were able to resolve. We will continue to monitor for regressions.

We've confirmed reports of dashboard errors and our team is investigating.

We've observed a complete recovery of traffic.

Our team has taken remedial action and is observing recovery, we will continue to monitor.

The gcp-us-east4 cluster is experiencing elevated rates of 408 and 429s affecting both reads and writes affecting approximately 4% of traffic. Our team is investigating.

The region is stable and has fully recovered. We will continue to monitor.

Latencies have returned to normal and 429s have fully recovered. The issue was related to transient failures in the underlying infrastructure. Our engineers have applied mitigations, and we are continuing to monitor.

We are looking into an issue causing increased latency and 429s in us-east4.

The region has stabilized and returned to normal operation. We are continuing to monitor!

A fix has been rolled out and 5xx errors in the region have reduced. We are continuing to monitor the situation!

We are investigating an increase 500's in the region!

Full recovery as of 12:24 PM UTC

We are seeing degraded performance again as latency to Google Cloud Storage. We are continuing to monitor and work with the GCP team.

Full recovery as of 10:44am UTC. We are investigating the root cause with the GCP team.

Substantial improvement starting 10:35am UTC. We are continuing to investigate and work directly with GCP to resolve this fully.

8:30am UTC - 9:25am UTC: Degraded performance on queries and writes in gcp-us-east4 from increased latency to Google Cloud Storage. 9:25am UTC: Escalated to partial outage both write and read paths due to climbing latency and timeouts from Google Cloud Storage. * Queries may see elevated rates of 429s, and 500s. * Writes may observe elevated rates of 408s, and 500s. Our team is working to restore full system performance, and we are working with GCP to restore service as soon as possible.

AWS has reported that the upstream incident is resolved.

The turbopuffer service remains healthy in the aws-us-east-1 region. We continue to closely monitor the situation while AWS continues to report service disruption in the region.

AWS is experiencing a major disruption in region us-east-1 impacting multiple services (<https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-east-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_BA540_514A652BE1A>). This caused turbopuffer to become unavailable in that region starting at 7:50 UTC. We've rolled out mitigations which resulted in a full recovery of turbopuffer at 8:20 UTC (despite the ongoing AWS outage). An existing mitigation to guard against EC2 instance launch issues has prevented the long-running EC2 degradation from further affecting turbopuffer availability. The AWS incident is ongoing. We will continue monitoring the cluster's health.

We have deployed mitigations and are monitoring the situation.

[AWS us-east-1 is experiencing a major outage and many services are affected. ](https://health.aws.amazon.com/health/status) turbopuffer clients in aws-us-east-1 will see elevated error rate and latencies during this outage. We're looking at mitigating the impact.

Service is stable. We are continuing to investigate the root cause with high priority.

We have observed an increase in timeouts and a slow down in indexing starting at 9:38 UTC. The service has returned to normal at 10:00 UTC. We are monitoring and investigating the root cause.

we're currently observing degraded performance in gcp-us-east4 leading to elevated rates of 408s and 429s. Our team is currently investigating.

The upstream incident is resolved.

Dashboard error rates have subsided. We continue to monitor the situation while we wait for confirmation from our authentication provider that the incident has resolved.

Some users are reporting issues accessing the turbopuffer dashboard due to a partial outage with our authentication provider: <https://status.workos.com/incidents/rdz189wnjhyn> turbopuffer APIs are unaffected.

We have taken steps to remediate the 500s, which have now subsided. We're investigating the root cause.

We're investigating an increase in 500 responses in the region.

we have not seen any re-occurences of the issue.

we've removed problematic query nodes from rotation and are monitoring for further issues.

some namespaces in aws-us-east-1 may experience degraded performance and request timeouts. our team is working to rectify the issue.

Service is stable.

Some of our query nodes restarted repeatedly due to OOMs caused by a code path that was producing very large allocations. We have deployed a fix, and the service has recovered. We are monitoring.

The problematic queries have been isolated and traffic has been restored to normal.

Some query nodes are currently experiencing excessive load, leading to 429s for certain namespaces. We have identified the problem and have a fix in preparation.

Service has been fully restored.

The affected certificates were manually renewed. Connections to aws-us-east-1.turbopuffer.com should once again pass TLS validation.

We've identified that a stuck ACME challenge prevented cert-manager from automatically renewing the TLS certificate for aws-us-east-1.turbopuffer.com.

We are investigating reports of connections to aws-us-east-1.turbopuffer.com failing TLS validation.

incident was resolved

misconfiguration broke indexer auto-scaling, causing a backlog of jobs

Latencies have remained stable.

The region has fully transitioned back to Intel hardware and API response latency has returned to normal. We're monitoring to ensure that latencies remain stable.

We've identified a pathological performance issue when running on ARM hardware. We've shifted the region to run entirely on Intel hardware.

We're investigating reports of degraded performance in aws-us-west-2.

The issue has been resolved. We deployed a fix and confirmed that affected nodes have returned to normal operation.

We identified an issue causing high CPU utilization on a subset of query nodes in us-west-2. Some customers experienced 429 errors and increased query latency during this time. We have identified the root cause and are deploying a fix. We are actively monitoring the situation and implementing additional safeguards to prevent this from recurring.

Service has returned to normal. The cause has been identified and is being addressed.

We've identified the problem and taken action to stop customer impact.

Customers may notice a small number of their requests receiving a 500 response due to transient unavailability of turbopuffer query nodes. We have identified a probable cause and are moving to confirming and addressing it.

Service has been fully restored. We'll be sharing a full RCA with affected customers as soon as possible.

We've identified the problem and rolled out a fix. We're closely monitoring the situation.

We are investigating reports of an outage in aws-us-west-2.