This incident has been resolved. We haven't observed any more network connectivity problems with our upstream vendor.
Posted Feb 13, 2023 - 23:10 UTC
Update
We are continuing to monitor for further issues.
Posted Feb 13, 2023 - 22:19 UTC
Update
We are continuing to monitor for further issues.
Posted Feb 13, 2023 - 20:59 UTC
Update
We are continuing to monitor for further issues.
Posted Feb 13, 2023 - 19:41 UTC
Update
We are continuing to monitor for further issues.
Posted Feb 13, 2023 - 19:16 UTC
Update
We are continuing to monitor for further issues.
Posted Feb 13, 2023 - 18:51 UTC
Update
We are continuing to monitor for further issues.
Posted Feb 13, 2023 - 18:28 UTC
Update
We are continuing to monitor for further issues.
Posted Feb 13, 2023 - 18:05 UTC
Monitoring
We have been able to confirm that all "impacted" Serverless Cloud Routers will have at worst a few milliseconds of latency added to their routing. Having this impact is also highly improbable, but definitely possible.
At this time we are quickly moving to fix the final impacted Serverless Cloud Routers, but are reducing the severity of this incident as we have been able to confirm the scope is not dramatically hampering most traffic.
Posted Feb 13, 2023 - 17:37 UTC
Update
We have gotten an update from our upstream vendor and all new Cloud Routers on the Serverless offering will not be impacted here. Additionally, we have been given steps to mitigate all impacted Serverless Cloud Routers, and will be moving forward with this process shortly.
Posted Feb 13, 2023 - 17:33 UTC
Identified
Our upstream vendor appears to be having network connectivity problems which are impacting particular hosts in a particular region. It does not impact all hosts in the impacted region and we are currently awaiting resolution or mitigation with our upstream vendor.
Posted Feb 13, 2023 - 08:42 UTC
Investigating
Some user's Cloud Routers on the Serverless offering are failing to come online. Additionally, some existing deployments may have disrupted or intermittent network connectivity. We are investigating with our upstream partner to determine the source of the outage. We will post additional updates as soon as possible.