At 19:58 UTC on 14 April, 2021 we observed elevated latencies and errors in our Europe point of presence which affected History, Channel Groups, Push device registration, and Presence joins. We added additional server capacity and the issue was resolved at 20:27 UTC. This issue occurred because of an atypical combination of usage patterns causing CPU over-utilization, ultimately resulting in the latencies and errors we observed.
We added server capacity above our typical overhead to prevent this from reoccurring in the short term but in the coming weeks, we will be analyzing the causal usage patterns and working to separate the affected services with the goal of mitigating possible recurrence of these multiple process failures.