This incident has been resolved. We will continue to monitor the fix we have put in place and analyze other improvements we can make to our job processing pipeline.
Posted Sep 04, 2020 - 17:19 PDT
Monitoring
The backlog of jobs has been cleared and we have put in place a permanent fix for a queue contention issue. We are continuing to monitor the results.
Posted Sep 04, 2020 - 16:43 PDT
Identified
We have identified the source of the issue related to a sidekiq queue backlog, and have put a fix in place. Job latency is decreasing and we are continuing to work on the issue.
Posted Sep 04, 2020 - 15:24 PDT
Investigating
We are investigating an issue related to a backlog of jobs that is causing builds to be slow to process.