At 18:50 UTC on 2021-08-31, we observed some users experiencing errors logging in or using the PubNub Admin Dashboard. An infrastructure provider we use to host the Dashboard had a networking outage resulting in that service being inoperable for several hours. Upon understanding the impact we rerouted traffic and the Dashboard was operational for all customers at 22:06 UTC.
Mitigation Steps and Recommended Future Preventative Measures
We were able to eventually diagnose the issue and route around it. This was made more complicated by a lack of alerting to help track down the exact issue. We will be adding alerting to understand when a similar issue occurs in the future and more quickly reroute the traffic.
Posted Sep 04, 2021 - 03:59 UTC
Resolved
This incident has been resolved.
Posted Aug 31, 2021 - 23:10 UTC
Monitoring
Our upstream vendor has implemented a fix and we are monitoring recovery.
Posted Aug 31, 2021 - 23:01 UTC
Update
We are continuing to experience increased latency and network degradation. We have identified the source of the issue and are working with our upstream vendor to remediate it.
Posted Aug 31, 2021 - 21:57 UTC
Update
Our service provider continues to work to resolve the issue but currently, there is no change in status. We will continue to monitor the status updates from our service provider and relay them here.
Posted Aug 31, 2021 - 20:33 UTC
Identified
Around 18:50 UTC, the PubNub Admin Dashboard went into a degraded state due to a service provider issue in the US West region. We alerted our service provider and we are awaiting resolution. We apologize for any inconvenience this may cause.
Posted Aug 31, 2021 - 20:00 UTC
This incident affected: Website and Portals (Administration Portal).