Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-October-26 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday October 26th 2023, 17:03 - 17:30 UTC
What happened:
For the duration of the incident, customers could not upload new files or list files/groups through the Web UI. Some users received an HTTP error code of 429 due to them being rate-limited.
Why it happened:
Customers began to have issues uploading assets to our storage service due to a high number of concurrent uploads. This spike in upload requests caused a delay in acquiring a backend connection for the storage service.
How we fixed it:
We recycled the storage service pods, which cleared the connectivity issue with our backend service.
What we are doing to prevent it from happening again:
We are looking at the following areas:
- Determining how better to handle periods of increased load for our storage service.
- Refining our alerting for our storage service to detect performance issues customers may experience when interacting with it.
Resolved
Live and Automated tests on Real Devices/Emulators/Simulators have remained stable on our US-West Data Center. All services are fully operational.
Monitoring
Real device and Emulator/Simulator tests are running successfully in our US West data center. We continue to monitor.
Investigating
We are aware of an ongoing issue related to Application Management impacting RDC and Emulator/Simulator Live and Automated tests on our US-West Data Center. We're investigating
This incident affected: Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), API Testing (US-West), and Native Framework Mobile App Testing (US-West).