Fly.io

This incident has been resolved.

A fix has been implemented and we are monitoring builds. Standard flyctl builds should be working again for customers in all regions.

We have identified an issue with deploying via Depot for users connecting through our SYD and JNB regions. Affected customers in these regions can deploy successfully using the --depot=false or --buildkit arguments to flyctl.

We are investigating reports of Depot builds failing for some customers. Affected customers can deploy successfully using the --depot=false or --buildkit arguments to flyctl.

This incident has been resolved.

Private networking between Fly Machines is resolved, and most outbound connections are healthy. We're continuing to monitor the network, and some issues will still be expected from clients physically located in US West until upstream transit issues are resolved.

We are investigating upstream network issues from US West Coast (SJC, LAX). Apps hosted in US West regions may experience higher latency or packet loss, and requests from clients physically located in US West may experience higher latency.

This incident has been resolved.

Error rates have decreased. We are continuing to monitor the API health.

We're aware of a problem affecting a subset of Sprites users. We are investigating the source of the issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We're updating the affected region list to also include SJC since this seems to be a wider upstream issue in US West Coast.

We have put in some temporary mitigations along with our providers. However since the root cause of this issue lies within a bigger upstream transit provider, you may continue to see some elevated latency and connection issues in/around affected regions. We're still working closely with them to resolve the root cause.

We have observed an upstream network issue in LAX. Connections to some destinations may see elevated latency and packet loss. We're working with our upstream to resolve this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating an issue with the background job runner for our API. Actions that require a background job, such as creating apps, assigning IP addresses, or creating/renewing certificates, may fail at this time.

A configuration update caused temporary failures for incoming HTTP/2 traffic for Fly Machines located on a subset of hosts for a few minutes. This incident has since been resolved. Managed Postgres depends on HTTP/2 and some control plane ops may have been affected as well. However, downstream Postgres connections were unlikely to have been affected by this since they do not use the HTTP2 handler.

This incident has been resolved.

Packet loss in ORD is improving and impacted services are recovering; we’re continuing to monitor for intermittent issues

Due to an upstream provider, we are seeing ~50% packet loss on a subset of hosts in ORD. Some MPG clusters in ORD are slow to replicate as a result.

We saw Sprite deletion jobs failing between 21:18 and 22:05 UTC. This issue has been resolved.

This incident has been resolved.

Our upstream provider has implemented a fix. Network performance in GRU has normalized.

We are seeing a recurrance in networking issues in GRU. Some apps in the region may experience increased latency or packet loss. We are working with our upstream networking provider to resolve.

Networking performance in GRU has normalized and we are no longer seeing issues. We are continuing to monitor to ensure a full recovery.

We are investigating networking issues impacting some hosts in GRU (São Paulo, Brazil) region. Some apps in GRU may experience increased latency or packet loss.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

Our testing and monitoring indicates gateways should be back to normal; if you are still having problem using `flyctl ssh console`, try restarting the `flyctl` agent by `flyctl agent restart`.

A fix has been implemented and we are monitoring the results.

We are investigating issues with our WireGuard gateways. Some CLI commands like `flyctl ssh console` or `flyctl proxy` may not work at this time. Apps continue to run.

This is now resolved

All hosts have caught up with metrics and we're monitoring the situation

We are currently experiencing some metrics lag on servers in some regions. We are provisioning more metric processing instances to accommodate the backlog and catch up.

This incident has been resolved.

Upstream networking issues have resolved.

We are investigating network issues in the Los Angeles region. Apps may experience higher latency or be unreachable at this time.

A BGP configuration error caused our Anycast DNS to route to some nodes without the proper DNS infrastructure. The issue was temporary and was resolved as soon as we removed that node from BGP.

This incident has been resolved.

A fix has been deployed and this error should no longer be occurring. We're monitoring to ensure full recovery.

We have identified an issue causing authentication errors for some operations from `flyctl`. These operations are failing with an error like: `This endpoint no longer accepts legacy OAuth tokens (starting with `fo1_`). Please use a macaroon token (starting with `fm2_`) instead. We have identified the issue and are rolling out a fix

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We had an issue with the ord-0 Fly Kubernetes cluster, and many MPG clusters are failing to restart. Our MPG team is actively working on it.

6PN networking issues between some machines in YYZ during a rollout which was rolled back once we noticed errors. During this time some machines were unable to talk to internal resources like other DBs, other apps or MPG clusters.

The capacity issue in the ARN region has been resolved.

New machines may fail to create in ARN because we lack capacity.

This incident has been resolved.

We have failed over the secrets database to a replica, and the Machines API appears healthy now. We are monitoring for any further issues.

We are working to recover our secrets service after a failed deployment. Apps continue to run, but it is not possible to create new apps or update secrets at this time.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

We are currently investigating degraded ipv6 networking on a subset of hosts

This incident has been resolved.

A fix has been implemented and we are monitoring the results

We’ve deployed an additional mitigation to further reduce Corrosion retry pressure and are seeing improvement; we’re continuing to monitor while remaining affected nodes catch up.

We’ve applied a mitigation to reduce the impact from Corrosion batch insertion retries and are continuing to monitor while affected nodes catch up.

We have identified the issue as failed insertions in a subset of Corrosion batches. These failures trigger retries, which can cause timeouts for other batches.

We are currently investigating app-not-found errors returned by Machines API calls made shortly after creating new applications.

Etcd is stable. Services are back to normal.

High CPU pressure on a shared etcd instance is causing lags on MPG creation in the IAD region

This incident has been resolved.

We tweaked hosts to allow for more machine allocation. We'll be monitoring the region over the next hours.

New MPG clusters may fail to create in GRU because we lack capacity.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have the size of TLS certificate issuance backlog under control, but are still seeing some remaining issues and are currently working to clean up the edge cases.

We believe we have identified the issue and are releasing a fix.

We're currently investigating issues related to certificate issuance. Certificates may be delayed for new custom domains.

A BGP misconfiguration while provisioning new edge capacity caused most traffic from Europe endpoints to be dropped, between 14:50 UTC and 15:02 UTC. The misconfiguration has been fixed and we are implementing safeguards against this kind of issue in the future.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating an issue with the control plane for Managed Postgres v2 in the IAD region. Creating new v2 clusters in the IAD region may fail at this time. Existing clusters continue to run, but may experience lagging backups.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Creating machines in CDG region may fail at this time with an "no capacity available in cdg" message. Existing apps continue to run.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Our dashboard is failing to send outbound email. Emails such as new account verification or password reset may fail to send at this time.

This incident has been resolved.

We are investigating some database issues causing high latency on some API endpoints and dashboard operations. You may experience intermittent "503 service unavailable" errors at this time. Currently deployed apps continue to run.

This incident has been resolved.

We are still working on fixing degraded Managed Postgres clusters.

Some Managed Postgres v1 clusters are degraded. We are working on fixing them. Managed Postgres v2 is unaffected.

A fix has been implemented and we are monitoring the results.

We've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience.

We've identified an internal service providing authentication to our Machines API has failed, our team is currently looking at our options for restoring this service. Existing Machines/Apps will continue to run as normal. Thank you for your patience.

We are continuing to investigate this issue.

Existing machines are unaffected. We are investigating the issue.

This incident has been resolved.

We have identified an upstream issue that is preventing egress IPv6 addresses in BOM from reaching parts of the internet, and we're currently working with an upstream provider to resolve this issue. Normal IPv6 addresses remain unaffected.

A bad deployment momentarily caused issues with Flycast connectivity, and, by extension, MPG, in our YYZ region. The deployment was immediately rolled back and connectivity was restored shortly after.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating increased connection latency and "connection reset" errors from our edge proxy. Apps continue to run, but requests may experience increased connection latency or fail at this time.

This incident has been resolved.

An issue with our Machines API is causing app creations to fail in some cases. We are working on a fix.

This incident has been resolved.

A fix has been implemented and we are monitoring the results. Apps should be reachable at this point in time.

A subset of hosts in SJC are currently offline. Some apps may be unreachable at this time.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

A subset of hosts in DFW are currently offline, and we're investigating the issue.

This incident has been resolved.

A fix has been implemented and we are seeing improvements in builder performance, latency, and error rates. We are continuing to monitor for a full recovery. Customers still seeing issues can trigger a fly-hosted builder based deploy with `fly deploy --depot=false`.

We have identified an issue causing delays or failures when starting depoting builders located in the IAD region. Customers with builders in IAD may see delays or timeouts starting builds during `fly deploy`. We are working on a fix. In the meantime customers can trigger a fly-hosted builder based deploy with `fly deploy --depot=false`. You can also change your builder region away from IAD via the `settings` tab of your Fly dashboard. We recommend DFW or ORD as alternate builder regions at this time. Please note, your builder region can differ from the region your machines are in.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The fly.io registry is currently experiencing capacity constraints that reduced performance and may lead to temporary high latency or failed pushes. We are currently working to add capacity and restore service.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We've identified the issue as a networking hardware failure impacting a subset of hosts at one of our Upstream providers in ORD. We are working with our provider to restore connectivity.

We are investigating an issue with one of our upstream providers in ORD. Machines across a subset of hosts may be unreachable or not running correctly. Deploys with machines on these hosts may fail at this time. Some Managed Postgres clusters in ORD region may be unavailable or see connectivity issues at this time.

This incident has been resolved.

Customer workloads are now starting and we're monitoring the affected hosts. Affected Managed Postgres instances will be investigated.

Power restoration is ongoing and we're making sure the hosts are healthy before starting customer workloads to avoid issues. Customer impact remains and updates to come.

Power restoration work in a subset of ORD is still in progress and impact remains ongoing for a subset of hosts and some Managed Postgres clusters.

Our provider has advised us their facilities team is working on restoring the power, we'll provide another update as soon as we learn more.

We've identified and reported power issues with one of our upstream providers in ORD. We're waiting for an update from our upstream for a resolution. Some Managed Postgres clusters in ORD will be unavailable due to placement.

We are investigating an issue with one of our upstream providers in ORD. Machines across a subset of hosts may be unreachable or not running correctly. Deploys with machines on these hosts may fail at this time. Some Managed Postgres clusters in ORD region may be unavailable at this time.

This incident has been resolved.

A fix has been implemented upstream and certificates are being issued successfully. We will continue to monitor.

The issue has been identified and we are awaiting a fix.

We are currently investigating errors when issuing new SSL certificates for hostnames.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are investigating issues with static egress IPv6 addresses in NRT region. Apps using static egress IPs may experience connectivity failures to some destinations.

This incident has been resolved.

Background jobs have caught up and the API is fully operational. We are continuing to monitor service health.

A fix has been put in place, and we are no longer seeing elevated API errors. Some dashboard actions will be delayed while background processing catches up.

The cause of the errors has been identified and we are working on a fix.

We are investigating elevated errors with our GraphQL API and background job processing

This incident has been resolved.

Almost all metrics have caught up aside from a small handful in `sin` and `syd`. We're continuing to monitor and expect these to complete in the next few hours.

Backlogged metrics are still being processed. We're bringing extra processing capacity online to speed up the process.

Backlogged metrics are still being processed and ingestion delays persist for some customers

Backlogged metrics are still being processed and ingestion delays persist for some customers, but we’re continuing to see gradual improvement as the backlog continues to drain.

Backlogged metrics data is still being processed, continue to monitor the situation while the system catches up.

Metric ingestion is still delayed for some customers. We’re seeing gradual improvement and continue to monitor the situation while the system catches up.

We are still in the process of increasing the metrics cluster throughput to catch up with metric backlog.

We are continuing to see delayed metrics due to resource contention on a subset of metrics ingestion hosts, and we are working to rebalance ingestion traffic and reduce the backlog.

We are continuing to see delayed metric exports from a number of hosts. Users will see delayed or missing metrics for machines on impacted hosts at this time.

We've identified an issue on multiple hosts causing delayed metric ingestion into our hosted fly-metrics.net dashboards. We are working on a fix.

We are investigating issues with customer facing metrics in the fly-metrics.net dashboard. Users may see delayed or missing metrics at this time.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are aware of egress IP issues in SIN and NRT and are working on a fix. Some machines in SIN and NRT using egress IPs may temporarily lose connectivity or otherwise see degraded performance.

This incident has been resolved.

We are currently investigating an issue with our metrics cluster.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

One of our upstream providers is experiencing IPv6 network connectivity problems in EWR. Apps with machines on affected hosts may have impacted connectivity to certain IPv6 destinations while they investigate and resolve this issue.

This incident has been resolved.

We are seeing improvements in Depot builder provision times and are switching the default deploy strategy back to them. We will continue to monitor builder performance closely. Users with a preference can trigger a Fly builder based deploy with `fly deploy --depot=false` or use `fly deploy --depot=true` to force a depot-based deployment.

We are investigating delays provisioning Depot backed builders for deploys. We have switched the default `fly deploy` strategy to use fly hosted builders at this time. Users can still trigger a depot based deploy with `fly deploy --depot=true`

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We're addressing elevated control plane latency and saturation affecting the BOM and NRT regions. Apps with machines in this region might experience longer response times and possible timeouts (502 errors).

This incident has been resolved.

Some 6PN Private Networking traffic remains impacted into and out of our LAX region, pending upstream resolution.

Most networking is largely healthy between primary North American regions. Some Machines may see ongoing packet loss and higher latency communicating with other Machines on certain routes. We're continuing to monitor the backbone health upstream.

We are currently investigating degraded network performance between sites in NA due to an upstream incident