Purpose of This Report

This is a summary and analysis of an issue that occurred with the delivery of an Acquia product or service. The purpose of this document is to share details about what happened and why, so there is a common understanding of what is required to prevent a future occurrence if at all possible. Any remaining issues or risks are identified, as are recommended or pending actions.

Executive Summary

On 21 June 2022  06:27 UTC Cloudflare experienced downtime causing interruption of services in multiple Acquia products. Sites using Acquia Edge or independent Cloudflare services were similarly impacted. This included cloud.acquia.com and some of the Support services relying on services using Cloudflare.

Event Summary

On 21 June 2022  between  06:27 -  08:00 UTC users attempting to access the Acquia Cloud interface services along with Acquia Edge services received unexpected HTTP 5xx errors. During this period, visitors to sites using Acquia Edge services returned this same error response.

Acquia Actions

• Jun 21, 06:28 UTC Acquia’s platform monitoring system detected multiple sites down. Acquia also received direct support requests from customers whose applications were impacted.
• Jun 21, 06:32 UTC Cloudflare declared an incident affecting multiple locations via their service status page and Acquia identified this as the cause of issues for Acquia services and applications.
• Jun 21, 07:20 UTC A fix was implemented by Cloudflare and they were monitoring the results
• Jun 21, 07:42 UTC All services were restored by Cloudflare. Impacted Acquia services began to be restored to normal service.
• Jun 21, 07:56 UTC The underlying cause of this service interruption was addressed. All affected Acquia Cloud interface services were restored. All services were operational at this time and Acquia was monitoring the performance.
• Jun 21, 08:54 UTC Acquia monitored performance and after confirming that no sites or Acquia services were still impacted, and then publicly noted the issue as resolved.

Identified Root Cause

Acquia services were impacted due to maintenance being conducted by Cloudflare. This outage was caused by a change to BGP communities that was part of a long-running project to increase resilience in Cloudflare’s busiest locations.

Cloudlare provided a full postmortem of this outage via the following publication - https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/

This publication also includes all corrective actions taken and future follow up action to prevent recurrence of this issue.