Failure to serve some kits in Australia

Incident Report for Fort Awesome

Postmortem

Cloudflare identified the root cause and have shared the following incident report with us:

Cloudflare was performing scheduled maintenance in our Sydney facility. Before Cloudflare can perform maintenance, engineers must remove the datacenter from production by disabling anycast. As part of disabling anycast, a series of flags are set which notify all products and services that the facility is going into maintenance and the datacenter should not receive traffic.

Due to a bug in the automation to disable anycast, the flags to notify our Tiered Cache and Argo products were not set accordingly. As a result, when Sydney was removed from service, any request that had Sydney as an upper tier would continue to send traffic to Sydney despite the fact that the facility was not supposed to receive traffic. This caused increases in origin connectivity failures for surrounding locations.

We’re confident that they will take some steps to mitigate this same problem in the future.

Posted Jun 22, 2021 - 14:54 UTC

Resolved

We heard back from Cloudflare that they did see some network connectivity problems with origin servers in Amazon Web Services ap-southeast-2 (this is where our origin server exists). They have mitigated the issue and we'll follow up with them on pro-active steps we could take to avoid this in the future.

Posted Jun 15, 2021 - 11:06 UTC

Monitoring

At approximately 4:00 PM Eastern Time some customers in Australia were experiencing service interruption for our Kits service. The most common error seen was a 522 (origin server timeout). After investigating, it appears that dynamic load balancing shifted origin servers which caused the timeouts to occur. We are following up with Cloudflare, our provider, to identify if this is correct and figure out ways to mitigate this in the future.

At this time, the problem has been corrected and we will continue to monitor the situation and follow up with Cloudflare.

Throw any questions you have to hello@fontawesome.com.

Posted Jun 14, 2021 - 20:58 UTC

This incident affected: kit.fontawesome.com.