Summary

On January 19th, 2021, beginning at 2:07 p.m. ET, the Photos, Answers, and Live API services in Sandbox began experiencing elevated error rates. Engineers were notified and began investigation. Mitigation measures were implemented by 4:05 p.m. ET, at which point error rates began returning to normal. All Sandbox services were fully restored by 5:18 p.m. ET.

No production services were disrupted during this time.

Root Cause

A routine operation to patch and upgrade server hardware failed to add the upgraded servers to the load balancers. Adding the new machines to the load balancers allowed backend services to resume normal operation.

Remediation

We will be adding checks to our upgrade process to verify the correctness of load balancer changes.

Posted Jan 22, 2021 - 17:16 EST

Resolved

This incident has been resolved.

Posted Jan 19, 2021 - 19:04 EST

Monitoring

We have restored service in Sandbox and we will begin monitoring for any regressions.

Posted Jan 19, 2021 - 16:30 EST

Identified

We have identified the cause of the issue and have uncovered other network issues which may be preventing access to other flows in Sandbox at this time. We are working to mitigate and will post as soon as we have more updates.

Posted Jan 19, 2021 - 16:13 EST

Investigating

We are currently investigating reports of photo upload issues in the Sandbox environment. Some flows may not be available in Hitchhikers and the Sandbox Customer Portal at this time.

Production environments are unaffected by this incident.

Posted Jan 19, 2021 - 15:58 EST

This incident affected: Sandbox.