The streaming destinations pipeline has fully recovered and all events are now back to real time.
Posted Jul 13, 2022 - 14:18 PDT
Monitoring
A hardware failure in our streaming destinations pipeline caused a delay of up to 25 minutes for 7% of cloud mode destinations. For the affected destinations, 33% of data did not experience a delay.
We manually replaced the underlying hardware and the affected destinations began to recover. We will continue to monitor until all destinations are back to real-time.
Our systems are designed to be resilient to hardware failures and recover automatically where possible but in this case, it did not recover automatically. We are investigating the root cause and will put fixes in place to ensure that we can automatically recover from more kinds of hardware failures.
Posted Jul 13, 2022 - 13:20 PDT
This incident affected: Streaming Destinations (Streaming Destinations (US)).
