At 04:40 UTC on September 23, 2021, the Wasabi database service in EU-Central-1 region was impacted for about 57 minutes during which customers experienced connection errors.
At 05:37 UTC on September 23, 2021, the system was returned to fully operational service.
Outage Details
Wasabi uses an internal audit process that runs in the background and validates the metadata from an accounting perspective. It issues queries to the database to collect appropriate information needed for reconciliation of data. The cause of the disruption was an inefficient query which put one of the database servers in a highly stressed state. Please note that Wasabi uses multiple “shards”, for its database operations, and this was seen on one of those. While the majority of operations worked, Buckets on the affected shard experienced read and write failures.
Database on the impacted shards was restarted, as part of a recovery. The audit process has been disabled until the inefficient query can be rewritten.