Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-April-13 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates
Tue April 13th, 2021 19:20-20:24 UTC
What happened
Customers using Sauce Headless experienced elevated wait times and an increased rate of tests unexpectedly terminating.
Why it happened
Our Google Kubernetes Engine (GKE) cluster underwent automated upgrades during the incident window, causing services to unexpectedly terminate.
How we fixed it
The automated cluster upgrade was completed and service returned to normal operations.
What we are doing to prevent it from happening again
We’re increasing system robustness by adding graceful shutdown mechanisms and creating a highly available setup for identified services.
Resolved
We have taken remedial action. us-headless-1 (Headless Cloud) should be running jobs without error.
Investigating
us-east-1 (Headless Cloud) jobs are failing intermittently. We are taking remedial action.