Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-September-19 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Sunday September 19th 2021, 15:42 - 16:35 UTC
What happened:
Automated and Live Mac and iOS tests in our EU-Central-1 data center were failing to start.
Why it happened:
We experienced a combination of router software failure caused by a third-party bug as well as with a failure to properly route traffic.
How we fixed it:
The issue was resolved by initiating a manual failover of traffic to the secondary circuit.
What we are doing to prevent it from happening again:
We are investigating potential options to avoid the failure to automatically failover traffic routing in the future. Additionally, we are removing a configuration from our edge routers to ensure we do not hit this particular software bug again.
Resolved
We have taken remedial measures and all Mac testing is back and running. All services are fully operational.
Investigating
Mac tests (Live and Automated) are not starting in our EU data center. We are investigating.
This incident affected: Automated Browser Testing (US-West) and Live Browser Testing (US-West).