Vault and KV Store is not accessible for US West PoP
Incident Report for PubNub
Postmortem

Problem Description, Impact, and Resolution

The PubNub Vault and KV Store services were not accessible in US West PoP for 2.5 hours due to an unreported global change made by our storage vendor. Access was blocked for customers running Functions in this region and for the PubNub Admin Portal UI.

Mitigation Steps

While a solution was being investigated by that vendor, we reconfigured our balancers to take publishes sent to California and have them trigger Functions in Virginia.  This restored access and functionality for executing Functions that required Vault or KV Store access.

Recommended Future Preventative Measures

We will add monitoring to detect Vault and KV Store access errors to enable our system to failover to a separate region and resolve a similar situation more quickly in the future.

Posted Nov 10, 2020 - 22:44 UTC

Resolved
We have been monitoring for 2 hours without any further issues. This incident is resolved. If you continue to experience any further issues related to this incident, please contact Pubnub Support: https://support.pubnub.com
Posted Nov 06, 2020 - 23:51 UTC
Monitoring
All services continue to be fully operational. We have fixed the underlying issue and rerouted traffic back which has brought latency back to normal levels.
Posted Nov 06, 2020 - 21:19 UTC
Update
The Vault access from the PN Admin Portal has been been restored.
Posted Nov 06, 2020 - 18:48 UTC
Identified
We have identified the issue and were able to reroute traffic around it. Access to the KV Store and the Vault within running Functions have been restored, although latency might be higher. Customers still cannot manage Vault resources via PN Admin Portal.
Posted Nov 06, 2020 - 18:24 UTC
Investigating
Customers in our US West PoP are experiencing errors when setting and retrieving Vault secrets and KV Store within the portal or within a Function. Functions that do not use Vault are working normally.
Posted Nov 06, 2020 - 18:02 UTC
This incident affected: Points of Presence (North America Points of Presence) and Functions (Vault, Key Value store).