On July 25, 2022, a routine change was pushed to hosts in several data centers which triggered a bug in the Top of Rack switches. This bug impacted the way route withdrawals were handled for IP sharing in our Newark, London, Frankfurt, Singapore, and Mumbai data centers. Customers with IP sharing configured who had a failover event occur may have seen the IP address not fail over as expected.
Due to the infrequent nature of this failure scenario, this issue was identified on Aug 25, 2022 after customers notified Support of the issue. After some investigation by the NetOps team, the source of the issue was determined and a fix was rolled out to the Newark and Mumbai data centers. During the rollout to the London data center, an issue occurred that resulted in networking loss to the London Data Center. More information on that can be found in the status page for that incident.
Following this, a status page was issued for the remaining data centers as we rolled out the fix on August 26, 2022.
Going forward, we are reviewing our policies regarding public communication of maintenance. Further, we are in discussions about expanding internal testing to identify edge case failures sooner.