Executive Summary

On the 31st of May 2022 at 06:52 GMT, Squiz monitoring systems detected a degradation of service affecting customers hosted in our Sacramento and New York data centres. Investigation by the Squiz Data Centre team indicated a Distributed Denial of Service (DDoS) attack on a customer’s search website. Other customers hosted in Sacramento and New York data centres may have experienced packet loss and elevated response times resulting in intermittent degradation of service. 

Once the issue was identified, the Squiz Data Centre team took remedial action to contain the attack by blocking the DDoS attack via DNS Blackholing, rerouting the incoming network traffic on the impacted website, resulting in partial stability. Recovery was achieved with changes to Firewall and Web Application Firewall rules at ~ 11:58 GMT.

Customer Impact

For the duration of the incident the targeted customers search website was disrupted and other customers in the Sacramento and New York data centres experienced increased response times and sporadic unreachability of services.

Root Cause

An application layer Distributed Denial of Service (DDoS) attack was launched on one of our customer sites causing a total disruption of service, and further impacting other customers due to being hosted in the same environments. The attack used a payload looking like normal web traffic that was not initially detected and blocked by security measures.The initial observable symptoms looked like normal internal traffic fluctuations with the initial small amount of packet loss.

Containment and Recovery

The attack against the targeted customer was contained by DNS Blackholing all traffic directed at the target, restoring normal operations to the data centres. 

The targeted customer was recovered by  updates to the Firewall and Web Application Firewall rules to recognise the attacker’s behaviour and block further attacks using this tactic.

Mitigation and Follow-up Actions

In response to this Incident, the Squiz Data Centre team will undertake the following actions:

• Review current security measures to improve capability in preventing higher layer attacks.
• Validate the existing firewall rules across our data centres to ensure overall application and environment stability. 
• Improve visibility of traffic monitoring across our data centres.
• Validate and increase alerts for Firewalls to monitor load spikes.
• Migrate VM Gateways from stateful firewall to the border routers, ensuring enhanced security is in place.

‌

If you require a PDF copy of this post incident report please contact your Squiz Service Experience Manager or Squiz Customer Care.