Data Processing is now fully caught up for realtime end points. Batch & Vacuum data is lagging by 4-5hrs which we expect to catch up in next couple of hours.
Posted Sep 28, 2020 - 21:29 PDT
Update
Data Processing is recovering from lag, now around 30% of the data is seeing lag up to 40min.
Posted Sep 28, 2020 - 20:19 PDT
Update
Data Processing is caught up on some devices and still catching up on others. Some data is delayed by up to 2 hrs
Posted Sep 28, 2020 - 17:43 PDT
Monitoring
Deployed fix to relieve load on the databse. Data Processing is catching up on the backlog right now. Currently data processing is delayed by up to 3 hrs
Posted Sep 28, 2020 - 16:17 PDT
Identified
We identified root cause to be an overloaded database. We are still working to resolve the issue. Currently data processing is delayed by ~2hrs.
Posted Sep 28, 2020 - 14:57 PDT
Investigating
Our data processing systems are delayed. This incident started at 1 pm PST.
Current status: a) Our ingestion systems are working as expected and we are not losing data. Our processing systems are not processing newly ingested data. b) All of our customers are impacted. c) Impacted customers will see delayed metrics for the last 60 mins.
We are investigating the issue and will post an update in an hour or earlier if we identify the issue.