Braintrust

The issue causing slow or failed data loading in our US-hosted service has been resolved. We're sorry for the disruption.

We've applied a mitigation for the data-loading issue affecting our US-hosted service and are monitoring performance. We'll share another update once we've confirmed stability.

We're investigating slow or failed data loading affecting some customers on our US-hosted service. Affected pages may load slowly, hang, or show errors.

The issue causing elevated errors in the Braintrust control plane APIs has been resolved.

We have mitigated the issue causing elevated errors in the Braintrust control plane APIs, we are continuing to monitor closely.

We are investigating elevated errors affecting the Braintrust control plane APIs. Some users may experience failures when accessing projects, prompts, settings, or creating new resources.

The load spike has subsided and 5xxs are mitigated

The gateway encountered a load spike leading to 5xx errors.

The deploy was rolled back and gateway 5xxs were mitigated.

A fresh gateway deploy was attempted and led to 5xx errors, and then was immediately rolled back.

The faulty deploy was rolled back and 5xxs were mitigated.

A deployment caused the gateway to begin returning 5xxs for a portion of traffic.

The upstream APT package mirror in AWS us-east-1 has recovered and we're no longer seeing impact when starting new brainstore instances.

The upstream APT package mirror on AWS us-east-1 is degraded. **There is currently no downtime** and we are looking into alternative package mirrors in the meantime. Please note that for **both the Braintrust-hosted data plane and self-hosted data planes:** EC2 instance replacement due to healthcheck failures or deploys may result in failed instance initialization and downtime until the upstream issue is resolved.

The issue has been resolved and service has remained stable following the rollback. We will continue to monitor internally, but this incident is now considered resolved.

The issue has been resolved. We are monitoring the service to confirm normal operation.

We identified an issue causing 5XXs introduced by a recent Gateway deployment and are currently rolling it back.

We have reverted to an older configuration to avoid an AWS outage with Cloudfront.

The hosted EU data plane is currently experiencing a very high rate of 504 errors.

A configuration change was identified as the root cause and reverted.

Our US data plane is encountering issues.

We have pushed a fix and the loading issues have been remediated.

We're experiencing issues loading logs and some other pages in Braintrust. Our team is actively investigating and working to resolve.

We determined the issue was caused by a role used by the automation service losing required permissions following a configuration change. The issue was reported on May 15 at approximately 1:00 PM PT and resolved on May 16 at 11:50 AM PT. To help prevent similar incidents, we have improved our alerting and added audit logging for role and permission changes.

On May 15, some automations stopped running which impacted scorer executions.

Our auth provider has recovered.

Our upstream auth provider is having issues. Fresh logins to Braintrust will fail with an error.

The issue was caused by our firewall provider unexpectedly blocking traffic despite safeguards we had in place to prevent this behavior. We are actively working with the vendor on additional mitigation measures to reduce the likelihood of this occurring again.

We are no longer seeing issues.

We are actively monitoring the fix.

We have identified the cause and are working with an upstream provider on a fix.

We're observing a high rate of errors on the production control plane which is causing errors on braintrust.dev and data plane operations.

Provider has recovered. Error levels back to zero.

Our upstream auth provider is having an outage. Users unable to login to Braintrust.

We're no longer seeing impact from the connectivity issues and we are still investigating the root cause.

We've updated our database connection method and are observing recovery. We're monitoring to look for remaining impact.

We're currently experiencing an issue with database connectivity. We're currently attempting to mitigate by connecting through an alternative method.

Performance quickly returned to normal when the problematic query was identified and remediated, and we are implementing safeguards to prevent similar future issues.

We experienced a brief period of degraded performance on the Braintrust hosted control plane due to elevated database load. The issue was caused by an internal traffic spike to a specific query pattern.

Error rates have dropped and functionality is restored.

We're seeing a high rate of 5xx errors

Load issues have subsided

DB upgrade complete. CPU usage appears to be at normal levels. Monitoring for now.

Database size is being increased. This will cause a short outage.

Error rates have recovered by database usage is still high.

The control plane is experiencing very high load.

The authorization provider has completed mitigation work and is reporting a full recovery in their systems. We are no longer observing impact to users.

The authorization provider has applied a mitigation and we're seeing signs of recovery. We're continuing to monitor until all errors have subsided.

Degraded performance of our authorization provider is causing issues with accessing the control plane.

The root cause appeared to be an issue with an upstream dependency (database connection pooler), which we have replaced. We will be writing a full post-mortem with more details.

We are no longer seeing issues

DB upgrades have completed, and we have seen error rates go down. Continuing to monitor.

We are upgrade DB resources, which may result in brief downtime

We are repairing the DB which may result in brief downtime.

We're observing a high rate of errors on the production control plane which is causing errors on braintrust.dev and data plane operations.

The issue has been resolved and you should be seeing no slowness in the Braintrust Control Plane UI. Please let us know if you have any issues.

At the current moment in time we have reports of hanging in the UI particularly in dataset or trace views. We are currently narrowing down the code change causing the issue. Please stand by we are aware and currently working to resolve.

We have not observed any recurrence of issues and our authorization provider has reported full recovery and stable performance across their entire system.

Impact for Braintrust users appears to have been resolved for the past 30 minutes. We'll continue to monitor until our authorization provider reports full resolution.

We've observed improvement in authorization errors but our authorization provider is still reporting a degraded state. We're continuing to monitor.

Degraded performance of our authorization vendor is causing issues with user authorization accessing the control plane. The vendor has identified the issue and is working on a fix.

Affected a small subset of users.

S3 automations have stopped running on the expected schedule they were set to.

The authorization provider has completed mitigation work and is reporting a full recovery in their systems. We are no longer observing impact to users.

The authorization provider has applied a mitigation and we're seeing signs of recovery. We're continuing to monitor until all errors have subsided.

Degraded performance of our authorization provider is causing issues with accessing the control plane. The service provider is investigating the root issue.

Provider recovered

An upstream provider had a brief outage that caused high errors in our control plane