Dates:

Tuesday March 15th 2022 16:15 - 23:00 UTC
Wednesday March 16th 2022 11:00 - 22:00 UTC
Thursday March 17th 2022 10:00 - 12:00 UTC

What happened:

On Tuesday and Wednesday our services located in our US data center (including website saucelabs.com, app.saucelabs.com, and services going through it like our REST API, Sauce Connect, saucectl, and API Testing) were not fully operational and some customers reported general slowness or were redirected to an error page.

On Thursday we noticed similar issues in our EU data center.
Both issues happened during rush hours and did not affect customers outside of them.

Why it happened:

While migrating traffic to a new set of servers, our CPU limits were set too low and resulted in our services not being able to process all traffic.

How we fixed it:

We reconfigured our server configurations to better handle the traffic sent to our services

What we are doing to prevent it from happening again:

We reviewed our monitoring policies and added additional checks to detect CPU over throttling in the future before it causes issues to our services. We’re also reviewing our system tests in order to understand how to detect and address latency/slowness going forward.