This incident has been resolved as of last week. We have improved internal systems and monitoring for this issue, and will continue to watch it closely.
Posted Oct 06, 2021 - 13:32 PDT
Monitoring
After root cause analysis and some infrastructure changes to address the issue, we have not seen any similar high latency issues in >12 hours. We will continue to monitor closely.
Posted Sep 29, 2021 - 18:35 PDT
Investigating
We have identified an issue with redis latency, which is causing bursts of 503/504s from Percy's API.
We do not have a fix at this time but are investigating further and will post updates as we have them.
We appreciate your patience and apologize for the inconvenience. This ops issue has been a tricky issue to resolve and we are coordinating with our redis provider on a resolution. We will leave this ticket open until we are sure the problem has been fully resolved.