On June 29, 2021 at 11:40 AM ET, the Widgets service began serving a large number of errors. By 12:20 PM ET, we implemented a fix, and the error rate began to normalize, reaching full recovery by 12:40 PM ET.
As part of our normal security patching process, we migrated the Widgets microservices to new physical application servers to perform security updates on the original application servers. The Widgets services are unique in that they use intermediate caching load balancers. These intermediate load balancers were not aware of the new application servers, and thus believed the Widgets microservices to be down.
We have improved our service location code to ensure that the intermediate load balancers identify these migrations as they occur.