Still all clear! Thanks for the patience today as we recovered from a failure in our hosting infrastructure. Have a great day!
Posted Nov 14, 2019 - 20:01 UTC
Monitoring
Our data processing is fully operational again.
We identified a number of internal services that restarted due to failures at our cloud infrastructure provider starting at approx. 19:00 UTC. Some of these services, chief among them being our track.customer.io API workers did not restart gracefully.
At 19:15 UTC our SRE team restarted these API workers which unblocked our delayed data processing and we are monitoring now to ensure our system remains stable.
If we don't identify additional issues by 20:00 UTC we will resolve this incident.
Posted Nov 14, 2019 - 19:28 UTC
Investigating
We're investigating a sudden spike in errors for our data processing infrastructure. This causes a delay in inbound API processing and outbound message delivery.
We'll provide an update by 19:30 UTC as we learn more.
Posted Nov 14, 2019 - 19:15 UTC
This incident affected: Data Processing and Message Sending.