Elevated latency for the Presence service in US-West
Incident Report for PubNub
Postmortem

Problem Description, Impact, and Resolution

A sudden and drastic increase in presence traffic led to a presence database resource contention that negatively impacted the latency of processing presence events which ultimately led to false presence timeouts.

Mitigation Steps and Recommended Future Preventative Measures

At 05:25 UTC (21:25 PST) we increased the capacity of the presence database resources and presence latency and errors quickly returned to normal.

Posted Jan 06, 2021 - 21:36 UTC

Resolved
The latency has remained at normal levels for over an hour so we are resolving this incident.
Posted Dec 23, 2020 - 06:49 UTC
Monitoring
Latency has returned to normal levels since 05:38 UTC (21:38 PST) and we will continue to monitor.
Posted Dec 23, 2020 - 06:04 UTC
Identified
We have identified the issue and taken action to mitigate the latencies. We have seen the latencies dropping over the past 30 minutes.
Posted Dec 23, 2020 - 05:48 UTC
Investigating
We are currently investigating elevated latency for the Presence service in the US-West region.
Posted Dec 23, 2020 - 05:00 UTC
This incident affected: Points of Presence (North America Points of Presence) and Realtime Network (Presence Service).