At 14:02 UTC on 2021/11/12, we observed increased latency with the Presence service in all of our PoPs. This was a repeat issue from the prior day so we already knew the quickest path to mitigating the effects of this incident. We contacted our service provider to provide insights while we restarted our presence services to clear the backlog of requests. and the issue was resolved at 15:01 UTC.
This issue occurred because there wasn't sufficient alerting between our system and our service provider when there is a rapid increase in Presence requests.
To prevent a similar issue from occurring in the future our service provider has upgraded the infrastructure to handle higher network throughput.