We observed a networking issue with one of our cloud infrastructure providers at our edge that resulted in some customers having difficulty connecting to our API from 12:00 to 13:20 UTC.
Mitigation Steps and Recommended Future Preventative Measures
We will improve our alerting around our synthetic tests to detect this type of issue when it occurs and reduce the time that customers would be impacted. We are also enabling some features provided by our cloud infrastructure provider which will improve resiliency in the face of availability zone failures.
Posted Oct 20, 2020 - 20:06 UTC
Resolved
This incident is resolved. There were networking issues that contributed to the errors and latency and we are still working on the exact root cause.
Posted Oct 10, 2020 - 14:58 UTC
Monitoring
The system recovered on its own and we continue to investigate for root cause. We will remain in monitoring mode for the next 30 minutes.
Posted Oct 10, 2020 - 14:11 UTC
Investigating
Around 12:30 UTC, some customers began reporting errors and latency with the publish, subscribe and authorization services in the Mumbai region. Engineering has been investigating since that time and errors/latency are beginning to decrease.
Posted Oct 10, 2020 - 13:25 UTC
This incident affected: Realtime Network (Publish/Subscribe Service, Access Manager Service) and Points of Presence (Asia Pacific Points of Presence).