On January 12th, 2021, at 1:20 p.m. ET, a backend database server storing analytical data for Answers started experiencing elevated latencies. This resulted in timeouts and errors for some users on the Answers Overview Page. Engineers restored service at 1:35 p.m. ET.
A large query was executed on the database server in question at a time of high load, which degraded its performance to the point of requests timing out.
In order to prevent such queries causing outages in future, we will tune these database servers to improve resiliency, as well as adjusting the query in question to improve its own performance, and exploring ways to filter potentially problematic queries.