Channel Groups services are reporting errors in the Mumbai PoP

Incident Report for PubNub

Postmortem

Problem Description, Impact, and Resolution

At about 02:45 UTC on August 10, 2022, we observed the Channel Groups service reporting errors in our Mumbai PoP, preventing users from making channel group related API calls there. We contacted our third-party vendor where the affected node resides, and the vendor applied a fix resolving the issue at 03:20 UTC. This issue occurred because of an error in our vendor's configuration that went undetected.

Mitigation Steps and Recommended Future Preventative Measures

To prevent a similar issue from occurring in the future we have confirmed that our vendor has applied a fix which addressed the root cause.

Posted Aug 12, 2022 - 16:09 UTC

Resolved

This incident has been resolved.

We apologize for the impact this may have had on your service. Please reach out to us by contacting PubNub Support (support@pubnub.com) if you'd like to discuss the impact on your service.

Posted Aug 10, 2022 - 03:58 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Aug 10, 2022 - 03:21 UTC

Investigating

At about 02:45 UTC on Aug 10 (Aug 09, 19:45 PST), the Channel Groups service began to report errors in the Mumbai PoP. PubNub Technical Staff is investigating, and more information will be posted as it becomes available.

If you are experiencing issues that you believe to be related to this incident, please report the details to PubNub Support (support@pubnub.com).

Posted Aug 10, 2022 - 03:02 UTC

This incident affected: Realtime Network (Stream Controller Service) and Points of Presence (Southern Asia Points of Presence).