Labelbox

We have confirmed the fix was successful and are moving the status to resolved.

A fix has been successfully deployed to production. We are closely monitoring system performance and metrics to ensure complete stability. Normal operations have resumed.

There is an ongoing orchestration incident that has resulted in labeling queues in some projects to be degraded. The issue has been identified and a fix will be rolled out shortly after further testing.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue

Labeling task queues may experience delays in reflecting new changes. Tasks, queue assignments, and priority updates may take longer than usual to appear. Existing labeling work in progress is not affected.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently experiencing increased response times across parts of our API infrastructure. Some users may notice slower request processing and delayed responses. Our engineering team is actively investigating the issue and working to restore normal performance. We will provide updates as soon as more information becomes available.

The incident has been resolved

A fix has been implemented and we are monitoring the results

The issue has been identified and a fix is being applied

We are continuing to investigate this issue.

We are noticing issues logging into Labelbox and are actively investigating the issue

This incident has been resolved.

Our service is impacted by a global Cloudflare outage. We’re awaiting recovery on their side and will update you as soon as things are back online

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We’re currently experiencing a service disruption due to an ongoing AWS outage affecting one of our critical dependencies. Our engineering team is actively monitoring the situation and will restore full functionality as soon as AWS service is recovered. We’ll continue to share updates as they become available. Thank you for your patience and understanding.

This incident has been resolved.

A fix has been implemented and we are monitoring the results

We have identified the issue affecting Azure cloud storage integrations and are actively working to resolve it. We will provide updates as progress is made.

We are experiencing issues with Azure cloud storage integrations. Users may encounter errors when accessing Azure-hosted datasets or validating Azure integrations. Our team is working to restore full functionality.

This incident has been resolved.

The issue has been identified. A fix is being implemented.

Incident is resolved.

We have identified the root cause, which is being mitigated. Monitoring.

We are continuing to investigate this issue.

We are investigating general slowness and lag in various parts of the labeling workflow.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue with query loads for the project-level performance dashboard and workspace-level monitor pages. These have currently been disabled while we investigate and fix the problem.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The Monitor page is undergoing maintenance. Workspace-level production monitoring is temporarily unavailable.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are investigating an issue causing latency in data exports

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating an incident that may impact Annotate and Catalog

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Dear Customers, We wanted to bring to your attention a recent labeling operation service degradation on the Labelbox platform. We will provide details on the cause and remediation as well as assurance on what actions we’re taking to prevent such incidents happening in the future. **Summary:** On Decembre 19, 2024, around 3:00 PM UTC, we experienced service degradation with labeling operations. This manifested mostly as general with logout message in the labeling editor. An investigation was launched and the root cause was determined. Subsequently a fix was implemented and deployed within an hour by 4:00 PM UTC. **Root Cause:** * Customers encountered authentication failures and were logged out while using the editor. Reports indicate that image and video editor labelers were particularly impacted. This was caused by authentication changes. **Resolution:** Upon identifying the root cause, fixes were implemented. The incident was mitigated when the changes for the CFW endpoint were rolled back. Monitor will be put in place to avoid similar situation to occur going forward. ‌ Sincerely, The Labelbox team

We are getting reports that logout messages are disrupting labeling operation.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are actively working on a fix and understand the urgency of this issue. We sincerely apologize for the delay and appreciate your patience as we resolve it.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Fixed and monitoring.

We have applied a fix and monitoring further.

We are currently investigating the issue.

This incident has been resolved.

We are currently experiencing issues with the Labelbox queue system impacting task assignment and processing times.

This incident has been resolved.

We are investigation an issue preventing user to login to Labelbox.

This incident has been resolved. We continue to investigate an additional fix to make the platform even more resilient. If we observe future occurrence of this issue due to same or similar root cause a new status incident will be opened. We will also update here once the additional fix is rolled out.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Dear Customers, We wanted to bring to your attention a recent labeling operation service degradation on the Labelbox platform. We will provide details on the cause and remediation as well as assurance on what actions we’re taking to prevent such incidents happening in the future. **Summary**: On April 25, 2024, around 9:25 AM UTC, we experienced service degradation with labeling operations. This manifested mostly as general slowness in the product. An investigation was launched and the root cause was determined. Subsequently a fix was implemented and deployed within a few hours by 2:59 PM UTC. **Root Cause**: * Task queues were overwhelmed around the same time data rows were deleted. Reconciling these deletions for data consistency resulted in a database deadlock. * Corrupt data was introduced into our global data processing pipeline causing it to halt and impact all services at Labelbox. **Resolution**: Upon identifying the root cause, fixes were implemented. * Data row deletion reconciliation was decoupled from task consumer. Size of messages batch was also reduced to simplify and reduce the size of database query thus greatly lowering the probability of any deadlock. * The corrupt data was identified and eliminated. We have implemented a code-patch to prevent similar corrupt data from impacting our global data processing pipeline. We apologize for this inconvenience. We are here to make Labelbox the most reliable and responsive data-centric AI platform, and will provide support in any way we can for your continued use. ‌ Sincerely, The Labelbox team

This incident has been resolved.

The issue is resolved, but the system is still catching up with processing the data. During this time, the Data Rows tab may display inaccurate details.

The fix has been implemented, and we're observing improvements in the system's health. Our teams are actively monitoring the situation, and we anticipate full resolution within the next 30 minutes.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

The issue has been identified and we are working on a resolution.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are currently investigating this issue.

Dear Customers, We wanted to bring to your attention a recent latency issue with export functionality on the Labelbox platform. We’ll provide details on the cause and remediation as well as assurance on what actions we’re taking to prevent such incidents happening in the future. **Summary:** On April 15, 2024, around 14:00 UTC, we experienced latency issues with export functionality. An investigation was launched and the root cause was determined. Subsequently a fix was implemented and deployed within hours. There were some sporadic occurrences of login issues while the increased export latency mitigation was in progress. **Root Cause:** The root cause of the incident was due to high resource usage on the primary database and a resultant backlog in internal systems due to a sudden surge in exports, even as systems scaled to meet the increased load. **Resolution**: Upon identifying the root cause, a fix was implemented that directed export traffic to a replica thereby reducing load on the primary database. We have also updated the batch processing logic for exports to reduce the probability of overload. Systems were monitored to confirm problem mitigation. We have also identified and scheduled the following additional corrective actions to prevent the occurrence of these types of issues in the future. * Enhancing the concurrent message processing logic. * Improving the retry handling mechanism to prevent a backlog of unprocessed messages. We apologize for this inconvenience. We are here to make Labelbox the most reliable and responsive data-centric AI platform, and will provide support in any way we can for your continued use. ‌ Sincerely, The Labelbox team

Exports latency is resolved. We'll be posting a post-mortem analysis of the incident within a week.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented; we are monitoring the resolution.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Incident resolved.

We've monitoring a fix and waiting on Exports V2 Non-Streamable to return back to normal.

We are continually working on identifying a fix, next update within 15-30 minutes if not sooner.

We've identified the cause for the slowdown and are working towards a fix.

We are currently investigating a slowdown related to Exports V2 Non-Streamable.

This incident has been resolved.

You can monitor the status of the third-party vendor here: https://www.readmestatus.com/

This incident is resolved.

Projects are now accessible. We are monitoring any further latency.

The issue has been identified and we are looking into a fix.

This incident has been resolved. All stalled export tasks are now completed.

A fix has been implemented and we are monitoring the results.

We have identified the issue and are working towards a resolution.

We are continuing to investigate this issue.

The issue is only affecting non-streamable exports. Streamable exports work as expected

This incident has been resolved.

We have identified the issue and working on deploying a solution.

Dataset creation through the SDK is currently unavailable for users with AWS integrations.

We've applied a fix, and resolved the issue. Customers are able to log into Labelbox with Google SSO.

The issue is impacting customers logging into Labelbox with Google SSO. We've identified a fix and are currently testing it.

We are currently investigating this issue.

All systems are back to normal operations.

Fix identified and is being deployed.

We are currently investigating this issue.

This incident has been resolved.

Labelbox is now operational, this issue was caused by a third-party service.

Labelbox is currently not accessible

This incident has been resolved.

Users are now able to log in. We are monitoring the results.

Users are currently unable to login to the platform. Engineering is investigating the issue.

**Summary** An issue in our automated deployment tooling caused Labelbox services to become unavailable. The malfunction did not affect any data storage, database, network, security, or other infrastructure. A change to the automated deployment tooling was reverted and services were automatically restored.  **Details** A configuration change was made Friday, October 20th at approximately 15:45 PT \(22:45 UTC\) that was believed to be benign. However, when this change propagated through the configuration system it had unintended consequences. This resulted in Labelbox services becoming unavailable for approximately one hour. **The malfunction did not affect any data storage, database, network, security, or other infrastructure, and no data was lost or affected by the outage.** The change to the automated deployment tooling was reverted and services were automatically restored. The restoration of the services took some time to stabilize. The incident was deemed fully resolved at 16:56 PT \(23:56 UTC\) ‌ Labelbox has already completed several changes to prevent this type of issue from being able to occur again.

This incident has been resolved. We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

A fix has been made and services are returning to a normal state. This will take some time before the platform is fully available.

We've identified an issue with our CD system and are working on a fix.

We are currently investigating this issue.

This incident is resolved. Thank you for your patience.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are currently investigating the issue.

This issue has been resolved, the rollback is now complete. Exports for v1 are fully operational.

To all affected customers, if you are not using custom editors, it is recommended to use Exports V2 endpoint. Exports V2 provides better functionality and is more resilient with load bursts. Learn more: 1. Exports V2: https://docs.labelbox.com/reference/export-v2-glossary 2. Limits: https://docs.labelbox.com/reference/export-v2-glossary

We are currently investigating the performance degradation regarding label exports v1 taking a long time to complete and in some cases not completing. The engineering team has attempted a fix that completes exports v1 resulting in a higher success rate. We are looking into restoring the full functionality soon. We appreciate your patience.

This incident has been resolved.

We are continuing to investigate this issue.

We are currently investigating this issue.

This incident has been resolved.

This issue has been identified and we are actively looking into a fix. Exports V2 is still working.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.