Executive Summary
On the 26th of April 2022 at 12:53 AEST, Squiz monitoring systems detected a degradation of service affecting some Squiz Edge customers.

Initial investigation by Squiz teams indicated a large number of HTTP 500 errors were being served to end users by the Squiz Edge CDN. Detailed investigations by our Data Centre team determined that a very large number of valid requests were being sent to systems hosted on Squiz Edge infrastructure in a very short period of time, resulting in a degredation of service for content served by the Sydney Edge Node. Our Data Centre team took measures to restore service. The number of HTTP 500 errors served from the Sydney Edge node decreased to normal operational error rates by 13:36 AEST, and after a period of monitoring the incident was closed at 14:00 AEST.

Customer Impact
Impact was to Squiz Edge requests served by the Sydney Edge Node between 12:53 AEST and 14:00 AEST on the 26th of April 2022. During this time some end users may have received HTTP 500 error messages or network timeouts.

Root Cause
The root cause was an extremely high rate of requests for specific content hosted in the Squiz Cloud in a very short period of time. The Squiz Edge CDN can handle bursts of requests at this rate, however in this situation as it was business hours in Australia and New Zealand the Sydney Edge Node was also serving the majority of Squiz Edge traffic at the time. The burst of traffic caused a component of the Edge system in the Sydney Node to suffer performance degredation. This degredation resulted in approximately 7.5% of all requests to the Sydney Edge Node being served HTTP 500 error messages or network timeouts between 12:53 AEST and 14:00 AEST. Once the component was identified the DataCentre Team took remedial action to restore service for the component, which when completed resulted in the number of HTTP 500 errors being served by Edge to decrease to within normal operational rates.

Mitigation and Follow-up Actions
In response to this Incident our Data Centre Team have taken the following actions:

• Additional monitoring and alerting for certain Edge system components has been deployed;
• Additional monitoring and alerting for both traffic and error rates at Squiz Edge has been deployed.
  

If you require a PDF copy of this post incident report please contact your Squiz Service Experience Manager or Squiz Customer Care.

Mat Walker
Major Incident Manager

Squiz Customer Care