From 7:34 to 7:47 PT our API experienced an outage causing 503 and 504 responses to be served for most requests. The outage was caused by a memory leak in the API which caused processes to crash. We have put mitigations into place and will continue to monitor and improve our memory management systems to prevent this from reoccurring. We apologize for the inconvenience.
Posted Feb 08, 2022 - 12:34 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 08, 2022 - 08:09 PST
Identified
We have identified the source of the issue and begun mitigation.
Posted Feb 08, 2022 - 07:55 PST
Investigating
We are currently investigating an API outage causing 503/504s to be served from percy API.