Inngest

The incident is now resolved and the system is full operational.

A fix has been deployed and we're monitoring the system to ensure the system is fully operational.

Monitoring revealed additional issues, and we're actively working on further fixes to restore normal system operation.

A fix has been deployed and we're monitoring the system to ensure the system is fully operational.

We are actively investigating an issue with function execution. We will provider further updates as we identify the cause and resolve the issue.

The incident is now resolved and the system is full operational.

We have identified the cause of the issue and we're monitoring the system as it is currently recovering. If users experienced any failed runs, you are able to replay failed runs in the UI. More info can be found here: <https://www.inngest.com/docs/platform/replay#how-to-create-a-new-replay>

We are actively investigating an issue with execution. Users may experience failed runs or delayed scheduling of runs from incoming events. We will provider further updates as we identify the cause and resolve the issue.

We are actively investigating an issue with function execution. We will provider further updates as we identify the cause and resolve the issue.

Between August 26 00:31 to 01:13 UTC, a subset of customers using Inngest Connect experienced delayed and failed function runs due to an internal networking issue that disrupted connectivity between Connect services. We reverted the change and restored service. Although Connect buffers worker responses, stalled internal API calls delayed acknowledgements and subsequent replies. In some cases, request forwarding failed even after a buffered response had been received. We fixed the root cause, improved Connect’s failure handling, and enhanced our internal observability so we can detect and respond more quickly to similar problems in the future. Affected customers should review failed Connect runs during the incident window and replay them after confirming that doing so is safe and idempotent.

Between August 24 at 23:38 UTC and August 25 at 00:08 UTC, a subset of customers using Inngest Connect experienced delayed and failed function runs due to an internal networking issue that disrupted connectivity between Connect services. We reverted the change and restored service. Although Connect buffers worker responses, stalled internal API calls delayed acknowledgements and subsequent replies. In some cases, request forwarding failed even after a buffered response had been received. We fixed the root cause, improved Connect’s failure handling, and enhanced our internal observability so we can detect and respond more quickly to similar problems in the future. Affected customers should review failed Connect runs during the incident window and replay them after confirming that doing so is safe and idempotent.

Run history delay has caught up and the incident is now resolved.

A fix has been deployed and we're monitoring the system to ensure the system is fully operational.

The incident is now resolved and the system is full operational.

A fix has been deployed and we're monitoring the system to ensure the system is fully operational.

We are actively investigating an issue with delays with function execution. We have a fix that has been released and are monitoring the system.

Execution latencies are back to normal across all customers, and the incident is now resolved.

We’re investigating execution slowness affecting a subset of customers, where steps may be delayed by a few seconds to a couple of minutes in the worst cases, caused by elevated memory usage on one of our shards.

We observed temporary function execution delays for a period of 15 minutes as we performed emergency maintenance to manage disk capacity to several nodes.

The incident is now resolved and the system is full operational. Delays began around 14:03 UTC caused by an issue publishing to a new Kafka topic. Recovery began around 14:40 UTC as the team was able to isolate this topic. The consuming services which schedules new function runs was scaled out and began to consume the backlog. No events were dropped during this incident, all functions were scheduled, but with delays during this time window. No action is needed for manual recovery.

New function execution has resumed and nearly caught up from the incurred backlog. Wait for event and cancel on handlers are still in a backlog and are catching up with processing. No events were dropped during this issue, all functions will be executed, but with delays. The root cause has been confirmd and changes are underway to prevent reoccurrence.

We have identified the cause of the issue - the team has deployed a fix and is scaling to catch up on function delays.

We are actively investigating an issue with function execution that is leading to delays starting function runs after events are received. We will provider further updates as we identify the cause and resolve the issue.

Metrics in the UI are now restored. Users may notice metrics data continue to be unavailable during the time of the incident, from the timeframe 7/16/26 15:45 UTC until approximately 7/16/26 21:07 UTC

A fix has been deployed and we're monitoring the system to ensure the system is fully operational. We confirmed all executions were functioning as expected, so this only impacts the metrics on the dashboard.

We are actively investigating an issue with our metrics service. Metrics displayed in the Inngest UI may not be up to date, but core platform functionality is unaffected and function execution is operating normally. We will provide further updates as we identify the cause and resolve the issue.

Throughput and latency have recovered and the system should be fully operational again. The core issue is related to the part of the system that persists run state. Some operations experienced and increase in errors and retries, causing a backlog in some parts of the system and some failed operations like checkpoints or signals. We are continuing a thorough post-mortem investigation to prevent reoccurrance.

We identified issues in the system and have deployed changes to mitigate the issues. Throughput has increased across the system. We continue to monitor latency as well as we expect it to reduce with the throughput increases. We are continuing to investigate for any additional issues that may have occurred.

We are actively investigating an issue with latency and execution errors. We will provider further updates as we identify the cause and resolve the issue.

The incident is now resolved and the system is full operational.

We are continuing monitoring the system. Users affected by this incident may need to replay impacted events to ensure their associated functions are executed.

We've made updates to the system, and function throughput is continuing to improve for users affected. The backlog is actively being processed, and we're closely monitoring recovery to ensure processing returns to normal.

We are investigating a partial outage affecting function triggering from incoming events. Some events may experience delays before triggering functions, or functions may not trigger as expected. Our team is actively investigating and will provide updates as more information becomes available.

The incident is now resolved and the system is full operational.

A fix has been deployed and we're monitoring the system to ensure the system is fully operational.

We are actively investigating degraded function execution performance. We will provider further updates as we identify the cause and resolve the issue.

The incident is now resolved and the system is full operational.

We’ve recovered the queue shard and it is now operational. We’re continuing to monitor it closely.

We’ve identified that one of the queue shards is down and are currently working to recover it. Only a subset of customers is affected.

The Support Portal is now available.

We are currently investigating an issue causing the Inngest support portal to be unavailable. Our team is working on a fix, and in the meantime, any support questions can be emailed directly to [hello@inngest.com](mailto:hello@inngest.com).

The incident is now resolved.

A fix has been deployed via Clerk and we're monitoring the system to ensure the system is fully operational.

We have identified the cause of the issue and are waiting for the downstream provider to implement their fix

The previous incident with the observability database has beens resolved, but there is a separate incident with a downstream provider.

A fix has been deployed and we're monitoring the system to ensure the system is fully operational.

We are actively investigating an issue with our backing database for several of our dashboard pages. REST API and Function execution are not currently impacted.

The incident is now resolved and the system is fully operational.

We're monitoring the system to ensure the system is fully operational.

A failover event in our data store provider changed the query plan behavior for some queries negatively and we had to adjust those plans to restore performance. Runs and event detail lookups are again performing normally. We're monitoring the system to ensure the system is fully operational.

We are actively investigating elevated latency with the backing data store powering runs and event lookups for the dashboard at [https://app.inngest.com/](https://app.inngest.com/. "https://app.inngest.com/.") as well as our REST APIs. We will provider further updates as we identify the cause and resolve the issue.

We are actively investigating elevated latency with the backing data store powering runs and event lookups for the dashboard at https://app.inngest.com/. We will provider further updates as we identify the cause and resolve the issue.

Performance on all shards is back to normal levels. The degradation was caused by a change that aimed to improve concurrency metrics for Inngest accounts. The change, while out for a full 24 hours and fairly benign, began to compound and produced slowness for some queue shards within our system earlier today. We have reverted that change and are working to understand the performance impact of this change.

Function execution latency has returned to normal for affected customers. Some customer shards were affected and we've pinpointed the cause of a slow degradation that compounded over time. We are working on adding new monitoring to catch performance regressions in this part of the system more quickly.

We are actively investigating increased function execution latency on a subset of customer shards starting around 5pm UTC. We will provider further updates as we identify the cause and resolve the issue.

The incident is now resolved and the system is full operational.

We have fixed the issue affecting a subset of users utilizing our REST API for retrieving runs and events data for recent updates and are monitoring for issues with availability of historical data over the REST API.

We are currently experiencing degraded performance affecting a subset of users utilizing our REST API for retrieving runs and events data. Responses may be delayed or temporarily return incomplete data, with recent updates not appearing immediately. We have identified the cause of the issue. We're actively working on implementing a fix to resume normal operation of the REST API. Function execution is not affected and is working as expected.

The incident is now resolved and the system is full operational.

Async "pause" operations (step.waitForEvent, step.invoke, cancelOn) should be running again at typical throughput. Event batching still has an backlog that we are actively working through. Status: • Function scheduling - Running as expected • Function execution - Running as expected, no queue backlogs • Async "pause" opts (waitForEvent, invoke, cancelOn) - Running as expected • Event batching - Significant backlog

The function run backlog for functions has been resolved as of 11:10AM PT. Batched functions and `step.waitForEvent` may face delays as the backlog continues to process.

Function execution scheduling and processing throughput is at normal levels. Async "pause" operations (step.waitForEvent, step.invoke, cancelOn) are severely backlogged which may cause delays in any of these operations from completing. This may cause issues with your function execution if you rely on them. The team is working on fixes and clear this backlog and fix the key issues. We do not yet have an ETA on resolving this specific issue.

Function scheduled delays should be caught up. With events processed and new runs scheduled, your system my still see backlogs based on your function's flow control (e.g. concurrency) config and your account's concurrency. We still see backlogs in processing step.waitForEvent, step.invoke and cancelOn event expressions. We are continuing to work on this. We also are continuing our rollout of isolated batch processing as previously mentioned to further isolate parts of our system. EDIT - This was edited to include step.invoke as well for completeness.

The system is consuming the event backlog as fast as possible, with an ETA of ~10-15 minutes until function scheduling is caught up. After function scheduling is caught up, function execution in your account may still be limited by your account concurrency or a given function's own flow control settings (concurrency, rate limit, etc.). We will continue to share more updates as soon as we can.

There is an increase in throughput since 15:37 UTC (~15 min ago). We are continuing to apply changes and prepare a larger change to decouple parts of the system. **Event observability**: Events may be delayed when appearing in the dashboard as the database ingestion for these events is also related to this part of the system that handles function scheduling. Events continue to be ingested and the Event API remains unaffected.

Changes have increase throughput, but not yet to typical levels. We are actively testing the new system change to decouple batch processing before enabling it for all accounts.

We're deploying an in-memory optimization within the the part of the system the schedules new function runs. This optimization will alleviate pressure on underlying systems and increase throughput. The change will be rolled out momentarily. We're also working in parallel on a system change to create a dedicated service for processing for event batching which is the cause of the overall backlog on the system.

We have scaled up several resources across the system and to handle a large increase in scale within the system. Services are scaled up and we have also added new function state shards, but rollout of those new shards can take up to ~30m. We are also working on networking improvements to improve efficiency of the system with this significantly higher load.

We are actively investigating delays with function run scheduling for a subset of customers. We will provide further updates as we identify the cause and resolve the issue.

The incident is now resolved and the system is full operational. Related to the Vercel incident, we updated our dashboard to Node 22.x to solve the issue. We continue to monitor Vercel's incident and react accordingly. https://www.vercel-status.com/incidents/5r9bp5y8rql2

A fix has been deployed and the dashboard is back online. We will continue to monitor the Vercel status page for any changes to the incident itself.

We area pushing a hotfix to the dashboard as recommended by Vercel's incident report. The rest of the Inngest system, function execution, API, etc. all remain functional.

The Inngest dashboard is down due to an issue with our downstream provider. Vercel. We are working quickly to bring this back up

The incident is now resolved and the system is full operational. During this incident step.fetch and step.ai.infer were failing due to a bug causing empty request bodies to be returned. The root cause was determined, the system was rolled back and a fix will be rolled out today.

A fix has been deployed for step.fetch and step.ai.infer and we're monitoring the system to ensure the system is fully operational. We continue to investigate the root cause.

We are actively investigating an issue with proxied requests via step.fetch or step.ai.infer. We will provider further updates as we identify the cause and resolve the issue. If you are not using these features your system should remain unaffected.

The incident is now resolved and the system is full operational. This was related to an issue caused by the part of the system powering the debounce feature. The internal event backlog is fully caught up and the two mitigations deployed have addressed the issue. The team is preparing a post-mortem to ensure this issue does not reoccur.

The mitigations have fixed the issue and the event backlog is now caught up. We continue to monitor and evaluate other short and long term mitigations to add.

We have deployed an additional hot fix. The earlier change rolled out have addressed the core issue and the system is now processing the backlog. The backlog is decreasing. We will provide another update as we have an estimate on time to recovery.

We have identified the cause of the issue affecting a core system queue. We are rolling our a mitigation now and preparing follow up changes.

We are actively investigating delays with function run scheduling. We will provider further updates as we identify the cause and resolve the issue.

After an extended monitoring period, we are resolving this incident. The system is full operational.

Function execution has returned to normal levels as of 11:12 UTC. We are actively looking into the root cause and taking further measures to stabilize the system.

We are actively investigating an issue with internal networking. We will provider further updates as we identify the cause and resolve the issue. Function execution is picking up again and was degraded between 11:02 and 11:10 AM UTC.

We are actively investigating an issue with function execution and other core system health. We will provider further updates as we identify the cause and resolve the issue.