PlanetScale

We've identified an issue with database connectivity for Vitess databases in GCP. We've taken steps to resolve the issue and are making immediate improvements to address root-causes and detection. Databases connectivity is now restored.

We've identified an issue in our networking and deployed mitigation. Connectivity is back to normal. We're continuing to monitor.

We're currently experiencing issues with connectivity to Vitess databases in GCP. We're investigating the issue.

The incident has been resolved. Between 12:17 and 12:24 UTC, the dashboard and API were returning 500 errors for postgres database management pages and API endpoints.

For postgres databases, API requests and the dashboard were returning 500 errors. We've deployed a fix.

The incident has been resolved.

Mitigations are in place and we're seeing signs of recovery. We continue to monitor the health of the system.

Insights metrics are underreporting for Postgres and Vitess databases. Database traffic and connectivity are unaffected. We are actively investigating the issue and are deploying mitigations.

Query Insights functionality is restored on all Postgres development branches.

Query Insights is gradually recovering across all Postgres development branches.

Query Insights is recovered across all branches except for Postgres development branches, where recovery is happening gradually. We are working to accelerate recovery of Query Insights Postgres development branches.

We are seeing recovery of Query Insights across all branches, except for Postgres development branches which where we disabled Query Insights to relieve pressure on an overloaded component. We are working to restore Query Insights to development branches as quickly as possible.

We have temporarily disabled Query Insights in Postgres development branches to relieve an overloaded component in the Query Insights processing pipeline. We are working to restore Query Insights to all branches as quickly as possible.

We are continuing to work towards remediation.

We have identified the cause of reduced Query Insights metrics reporting and are working on remediations.

Insights metrics are underreporting for Postgres and Vitess databases. Database traffic and connectivity are unaffected. We are actively investigating the Insights issue.

Connection issues have been resolved. If you experience any further issues, please contact our support team.

We remediated the issue, and clients can now connect to their database branches in us-east-1 normally. We will continue monitoring the situation.

We are observing connectivity issues on Postgres in the us-east-1 region. We are remediating.

API and dashboard service is restored.

We have increased the memory available to the API and are monitoring the performance and stability of the API and dashboard to confirm service is restored.

We have identified the cause of the errors - a pattern of requests causing the application to run out of memory and crash - and are working to better handle this traffic pattern now.

We are aware that both our API and dashboard are returning errors. We are investigating the cause and will restore service as soon as possible.

The issue is resolved. SSO is working fine again.

SSO login to the PlanetScale website and API is unavailable. We are investigating. Database operations are unaffected.

All impacted branches are recovered. PlanetScale upgraded development Vitess branches with Vectors enabled from MySQL 8.0.x to 8.4.x. PlanetScale runs an internal fork containing support for PlanetScale's MySQL Vectors indexes. A bug in the 8.4.x version of our Vectors builds resulted in the upgrade immediately failing on startup. Because Vitess development branches only have one instance - the primary - impacted development branches did not have a primary during the impact window. The incident was resolved by reverting impacted branches to their previous MySQL version. PlanetScale will fix the bug in the MySQL Vectors 8.4.x build before re-attempting the upgrade at a later date.

Branches are recovering and we are monitoring the situation.

We have identified the issue and are rolling impacted development branches from 8.4 back to 8.0.

PlanetScale initiated a system wide upgrade of branches with Vectors enabled and running on MySQL 8.0 to MySQL 8.4. A number of development branches were disrupted and have not yet recovered. We are investigating and remediating.

The changes we applied are effectively preventing further unexpected restarts to the high availability component shared by Postgres clusters. We have not seen further restarts to this component nor unplanned failovers in Postgres clusters since applying these changes.

We applied changes to prevent unexpected restarts of the high availability component shared by Postgres clusters. We have not seen further restarts of this component nor unplanned Postgres failovers since applying these changes. We are actively monitoring the situation.

We have identified the likely cause of unexpected restarts to the high availability component used by Postgres clusters, and are applying changes to prevent recurrences.

A high availability component that Postgres clusters depend on is restarting unexpectedly, triggering unplanned failovers in Postgres clusters.

PlanetScale has remediated the issue and it is fully resolved.

We've identified the issue and have taken steps to mitigate it. Database operations will successfully complete again. We continue to monitor the situation.

We're aware of a degradation in eu-west-2 where database operations such as resizes, branch creations or parameter changes may not complete. We're investigating the problem.

PlanetScale is not observing any database impact resulting from the AWS use1-az4 thermal event. Please contact support if you have any questions or notice any disruption.

PlanetScale is aware of the AWS use1-az4 thermal event. We expect databases to be unaffected. Please contact support if you have any questions or notice any disruption.

The errors have been resolved.

We've made a fix and are seeing webhooks trigger properly again.

We're investigating an issue with webhook deliveries failing.

Some percentage of new credential creates, updates, and deletes errored or were very slow for approx 35 minutes.

We have applied a fix and see recovery.

All existing credentials still work and will continue to work. We are investigating an issue affecting only modifying or creating new ones.

We've resolved the issue. SSO/SAML is working again.

We've identified a potential fix, testing, and the preparing to roll it out.

We are continuing to investigate. We've confirmed an issue with our SSO integration, but are still searching for a root cause.

PlanetScale is aware of issues logging in via SSO/SAML. We are currently investigating.

No further disruption is expected, please contact support with any further issues.

All databases have recovered, but we are continuing to watch the situation closely.

We are continuing to investigate

PlanetScale experienced the failure of VMs across multiple availability zones, causing a small number of branches to need rescheduling before recovery was possible. We expect all branches in this state have recovered, and we are continuing to monitor for further failures.

We are continuing to investigate and recover.

A small number of databases in GCP us-central1 are stuck with "primary is not serving, there may be a reparent operation in progress".

PlanetScale is continuing to monitor the condition of AWS us-east-1. At this point we do not see any further impact, and have re-enabled database and branch creation in the region.

PlanetScale is continuing to monitor the condition of AWS us-east-1. We have seen networking conditions and compute availability in us-east-1 continue to improve. Out of caution, database and branch creation continues to be disabled in AWS us-east-1.

PlanetScale is continuing to monitor the condition of AWS us-east-1. At this time database and branch creation continues to be disabled, and we are seeing some degraded network performance between components inside of us-east-1 and inbound connectivity into the region.

PlanetScale is continuing to monitor the condition of AWS us-east-1. At this time database and branch creation continues to be disabled, and connectivity may be disrupted into the AWS region.

PlanetScale is seeing degraded network connectivity across AWS us-east-1 components reaching both MySQL and PostgreSQL databases. We are currently monitoring the situation and will continue to update the status page.

Database creation in AWS us-east-1 is currently disabled.

Dashboard and API are back fully operational

There is a [AWS us-east-1 outage](https://health.aws.amazon.com/health/status). Existing databases are unaffected - new databases cannot be created.

SSO is affected

We are investigating elevated error rates with Dashboard and API

Datadog metrics are back up-to-date.

During the ongoing incident, some branches are seeing delays in metrics via the Datadog integration.

We've made a fix and the errors have resolved.

The dashboard and API are currently returning 500 errors. We are rolling out a fix.

This incident is resolved.

We've deployed a fix and are seeing error rates resolve.

We're continuing to work on a fix and will share an update once it has been deployed.

We're aware of an issue impacting new branch creation, deploy requests and cluster size changes for Vitess databases deployed to GCP. We are working on a fix.

Insights data ingestion encountered an error due to an unexpected data format. The issue was identified and fixed, and ingestion resumed.

Insights data ingestion is delayed. This underlying issue has identified and the Insights data backlog is being processed

The issue has been resolved.

A fix has been applied, and we are monitoring recovery.

We are investigating internal errors on some API endpoints. Database connectivity is unaffected.

Query Insights is fully operational.

Insights service is restored and we are blasting through the backlog of data.

There is an incident with a downstream provider. [https://status.clickhouse.com/incidents/01KG9R4QYASZW9VXHG3S2YD4WD](https://status.clickhouse.com/incidents/01KG9R4QYASZW9VXHG3S2YD4WD "https://status.clickhouse.com/incidents/01KG9R4QYASZW9VXHG3S2YD4WD")

We are investigating an issue with Query Insights

Problem has been resolved. We saw three nodes fail across three availability zones simultaneously. Impacting databases hosted on those nodes. All databases have been moved to healthy nodes and we're investigating how we can further reduce impact from similar events in the future.

Impacted databases are recovering, we're continuing to monitor.

We're working on a fix for the impacted databases.

We are currently investigating impact.

Vitess branch creation is available once again.

We're rolling out a mitigation now that will allow Vitess branch creation to succeed while we continue to investigate. No database operations or anything about Postgres database creation are impacted. Next update by 12:00 UTC.

New Vitess databases are being accepted by the PlanetScale API but we are unable to provision the infrastructure to serve them. We've confirmed this impact and are investigating the cause. We will provide further updates by 11:00 UTC.

This incident has been resolved.

We're continuing to monitor. We've updated the status history to reflect the time periods where customers experienced connection issues.

At **12:51 PM UTC** and **1:38 PM UTC** some Vitess databases in us-east-1 were impacted by connection errors for approximately 5 minutes due to an issue in our edge network. We resolved the issue and are no longer seeing errors. We will be taking action to protect customers from this issue happening again in the future and apologize for the error.

We're investigating an issue with connection errors to Vitess databases in us-east-1.