History and Presence experiencing elevated latencies and errors in the Europe PoP
Incident Report for PubNub
Postmortem

Problem Description, Impact, and Resolution 

At 19:58 UTC on 14 April, 2021 we observed elevated latencies and errors in our Europe point of presence which affected History, Channel Groups, Push device registration, and Presence joins. We added additional server capacity and the issue was resolved at 20:27 UTC. This issue occurred because of an atypical combination of usage patterns causing CPU over-utilization, ultimately resulting in the latencies and errors we observed.

Mitigation Steps and Recommended Future Preventative Measures

We added server capacity above our typical overhead to prevent this from reoccurring in the short term but in the coming weeks, we will be analyzing the causal usage patterns and working to separate the affected services with the goal of mitigating possible recurrence of these multiple process failures.

Posted Apr 16, 2021 - 21:15 UTC

Resolved
Services have remained operational since 20:32 UTC. We are resolving this incident and we'll follow up with a post-mortem after our analysis is complete.

We apologize for any impact this may have had on your services. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you wish to discuss any impact you experienced.
Posted Apr 14, 2021 - 21:48 UTC
Update
Services have remained operational since 20:32 UTC. We'll continue to monitor for the next 30 minutes.
Posted Apr 14, 2021 - 21:17 UTC
Monitoring
A fix has been deployed and we see improvements since 20:32 UTC. We'll continue to monitor for the next 30 mins.
Posted Apr 14, 2021 - 20:46 UTC
Update
We are continuing to investigate this issue.
Posted Apr 14, 2021 - 20:19 UTC
Investigating
At about 20:03 UTC, History and Presence services began to experience elevated latencies and errors in the Europe PoP. PubNub Technical Staff is investigating and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).
Posted Apr 14, 2021 - 20:13 UTC
This incident affected: Realtime Network (Storage and Playback Service, Presence Service).