At this time, all delayed messages have been sent, and data processing latency has returned to normal levels. Additionally, Amazon has posted that they have now fully restored power to all instances and devices in the affected data center.
We're considering this incident to be resolved. No data or messages were lost.
Posted Dec 22, 2021 - 09:06 EST
Monitoring
Between 7:10 and 8:25 am we experienced an issue causing increased latency with our message sending and data processing on our US01 and US03 clusters. We also saw some increased REST API error rates.
This was related to an ongoing issue with AWS in one of their data centers. Amazon has posted that they are recovering in the affected data center and we are seeing recovery.
At this time, most delayed messages have been sent. We are now able to spin up additional servers to process the remaining backlogs of work that is in the previously affected AWS availability zone. No data or message sending was lost.
Posted Dec 22, 2021 - 08:40 EST
This incident affected: US 01 Cluster (SDK Data Collection, Data Processing, Outbound Messaging) and US 03 Cluster (SDK Data Collection, Data Processing, Outbound Messaging).