The PubNub Vault and KV Store services were not accessible in US West PoP for 2.5 hours due to an unreported global change made by our storage vendor. Access was blocked for customers running Functions in this region and for the PubNub Admin Portal UI.
While a solution was being investigated by that vendor, we reconfigured our balancers to take publishes sent to California and have them trigger Functions in Virginia. This restored access and functionality for executing Functions that required Vault or KV Store access.
We will add monitoring to detect Vault and KV Store access errors to enable our system to failover to a separate region and resolve a similar situation more quickly in the future.