Summary

Starting form Tuesday, Jan 24, 2023, 05:15 UTC, All data sent to our data platform on EU region failed to be ingested. No exception was thrown.

The event was caused by an internal Azure platform component malfunction.

Once the issue was reported we notified to the vendor and they have fixed it internally.

Incident Timeline

• Tuesday, Jan 24, 2023, 05:15 UTC – Data sent to data platform stopped being ingested to EU region.
• Tuesday, Jan 24, 2023, 09:00 UTC – Complaints started flowing on no new data available on EU region. Data platform engineers started investigating.
• Tuesday, Jan 24, 2023, 10:00 UTC – Issue root cause found to be Azure platform related and vendor was notified
• Tuesday, Jan 24, 2023, 10:45 UTC – Issue resolved by vendor, Data started being ingested regularly
• Tuesday, Jan 24, 2023, 11:00 UTC – EU cluster scaled out to handle the data backlog accumulated while ingestion was down.
• Wednesday, Jan 25, 2023, 02:00 UTC – Data backlog consumed.
• Wednesday, Jan 25, 2023, 05:30 UTC – EU Cluster scaled back to normal sizing.

Root Cause Analysis

Internal Azure platform error caused data not being ingested and no error message was thrown. The problem was fixed internally by the vendor engineering team. After the issue was resolved it took several hours to consume the data backlog accumulated while ingestion was down.

Steps Done

Dedicated alert was added to both vendor’s platform and to checkpoint data platform.