Our Consul cluster is now stable again. We've re-enabled Nomad deployments and are monitoring for any further issues.
Posted 2 years ago. Mar 16, 2023 - 00:58 UTC
Update
The Consul cluster rebuild is nearly completed and we are working to restore service.
Posted 2 years ago. Mar 16, 2023 - 00:04 UTC
Update
The Consul cluster rebuild is nearly completed and we are working to restore service.
Posted 2 years ago. Mar 15, 2023 - 23:53 UTC
Update
We are working to build a new Consul cluster with 10x the RAM. We aren't yet sure, but believe a routine DNS change might have created a thundering herd problem causing Consul servers to immediately increase RAM usage by 500%. This is not ideal.
Posted 2 years ago. Mar 15, 2023 - 22:42 UTC
Identified
We're still working on recovering our Consul cluster.
Posted 2 years ago. Mar 15, 2023 - 21:10 UTC
Monitoring
Our Consul cluster is experiencing an outage. This impacts queries to our API, including creating and modifying apps, as well as incoming network requests for recently deployed apps.
Posted 2 years ago. Mar 15, 2023 - 19:20 UTC
This incident affected: Machines API and Deployments.