Performance was degraded in our US data center for around 2 hours between 16:00 and 18:00 UTC. This was down to our primary database struggling under load.
Steps were taken to remove any background processes to reduce the load as much as possible to aid the system to return to regular operation. API responses may have been slower than usual during this period, and background processing such as synchronizing calendar data will also have been slower than usual with messages taking up to 3 minutes to be picked up at the peak of the incident.
We will be bringing forward the maintenance to upgrade this database cluster from this coming Sunday 18th December to tomorrow Friday 16th December. A notice for this maintenance change will be posted shortly.
Posted Dec 15, 2022 - 18:36 UTC
Monitoring
The queued work has now been processed and we can see that performance is no longer degraded. We are continuing to monitor the situation.
Posted Dec 15, 2022 - 17:58 UTC
Identified
Our US database is experiencing slower than usual disk performance. We have taken steps to ease the pressure, such as temporarily disabling maintenance tasks. The amount of queued work is reducing. We're continuing to work to bring performance levels back to the usual level.
We have also added bigger database nodes to the cluster in case we need to failover to those. However, this would require a short outage and so we are holding off on failing over to those just yet.
Posted Dec 15, 2022 - 17:28 UTC
Update
We have taken steps to ease the pressure on our US database. This has resulted in better, but still degraded, performance.
We're continuing to investigate the root cause
Posted Dec 15, 2022 - 16:34 UTC
Investigating
We are investigating degraded performance in our US data center
Posted Dec 15, 2022 - 16:11 UTC
This incident affected: Scheduler, API, Background Processing, and Developer Dashboard.