Requests to Wasabi are failing
Incident Report for Wasabi Technologies
Postmortem

Summary

On 12 July 2021 21:36 UTC, there was a service event that caused an interruption in service across all Wasabi regions. The service event was a result of the unavailability of a working host on the primary Wasabi Global Database (GDB) infrastructure. Under normal circumstances, there are multiple secondary GDB instances, but these backup resources were not available at the time. To resolve the service event, the entire GDB instance was required to be restarted, taking approximately 46 minutes to complete. The GDB instance was restored at 22:22 UTC and service was returned to operational. 

Event Details

On Monday 12 July 2021 the primary GDB instance became unavailable, leading to errors on connections via S3, IAM, STS and the Wasabi Console from all regions. The Wasabi Operations team immediately moved to restore service to the primary GDB instance. Immediate attempts to restore the individual service failed and thus the GDB instance needed to be shut down and restarted. This consumed the majority of the downtime period. When the database service resumed, activity from all regions resumed and service was restored to normal. 

Action Items:

  1. Multiple new replicas of the GDB are now available in case of any future issue with the primary. 
  2. Procedures are being put in place to increase the frequency of replication to ensure consistency. 
  3. Regional replications will be deployed in coming weeks for additional redundancy
Posted Jul 16, 2021 - 17:41 UTC

Resolved
This incident has been resolved and service is restored to normal operating levels. Thank you for your patience while this was investigated. Please reach out to support@wasabi.com if you continue to see any issues.
Posted Jul 12, 2021 - 23:31 UTC
Monitoring
This issue has been resolved and we will be continuing to monitor the service.
Posted Jul 12, 2021 - 22:29 UTC
Investigating
We are currently experiencing service request timeouts. Our Engineering team is working quickly to resolve the issue. We will post regular updates as they become available.
Posted Jul 12, 2021 - 22:11 UTC
This incident affected: US-Central-1 (Texas), US-East-1 (N. Virginia), US-East-2 (N. Virginia), US-West-1 (Oregon), EU-Central-1 (Amsterdam), AP-Northeast-1 (Tokyo), Wasabi Account Control API, Wasabi Account Control Manager Console, and Wasabi Management Console.