Monday, March 8, 18:39 - 23:30 UTC
We were seeing elevated wait times for automated and manual MacOS tests in our US-West-1 data center. Customers may have seen MacOS and iOS Simulator tests taking a long time to start, or failing to start. Windows, Linux and Android tests were unaffected.
Due to high sustained peak usage combined with intensive CPU operations from a customer application our Mac Cloud CPU performance became degraded.
Additional data center caching servers were deployed to improve performance and distribution of application storage. We also implemented throttling of targeted jobs to reduce the CPU load on our US-West Mac Cloud.
In the future more aggressive throttling will be implemented on a per customer basis to keep the performance of our services within established baselines. A Sev1 issue will be opened with affected customers to work with them to resolve the issue specific to their usage.