Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-September-20 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday September 20th 2021, 11:30 - 19:50 UTC
What happened:
Sauce Labs UI pages and Real Device testing were not fully functional in our US-West-1 instance.
Why it happened:
Due to an increased amount of test report data for real devices, one of our long running queries caused the database to become unresponsive, resulting in real device testing and the Sauce Labs UI not being fully functional.
How we fixed it:
We improved the efficiency of the database query for large amounts of data.
What we are doing to prevent it from happening again:
We are improving our data storage limitations and optimizing the database setup, as well as extending our monitoring capabilities.
Resolved
We have identified the issue and deployed a fix. Real Device tests are working as expected.
Identified
We have identified the issue and taken corrective action. Live and Automated Real Device tests are now running correctly but we are still experiencing issues displaying Real Device test history. We continue to work on the issue.
Investigating
Live and Automated Real Device Testing and the Sauce Labs dashboard in our US-West-1 data center are running at a degraded level of performance, and errors are intermittently being experienced. We are investigating. This is a continuation of the issue experienced earlier today.
This incident affected: Automated Real Device Testing (US-West), Live Real Device Testing (US-West), and Sauce Labs Dashboard (US-West).