Infrastructure Issue Affecting Customer Sites
Incident Report for Pantheon Operations
Postmortem

On March 24, 2021 at 8:02 AM PDT, an invalid PHP configuration was deployed to the platform. After observing increased error rates, the deployment was stopped before full rollout and reverted, with remediation efforts beginning immediately for impacted sites. This invalid configuration caused elevated error rates for about 9% of sites on the platform. All sites were fully remediated by 3:39 PM PDT.

Pantheon is implementing improvements to its internal processes, infrastructure, and tooling to prevent similar incidents from happening again in the future. Additionally, improvements to our ability to more quickly detect and recover from such incidents have been identified and are being worked on.

Posted Mar 26, 2021 - 10:14 PDT

Resolved
This incident has been resolved.
Posted Mar 24, 2021 - 15:30 PDT
Update
We are continuing to monitor for any further issues.
Posted Mar 24, 2021 - 14:51 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 24, 2021 - 13:53 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2021 - 12:54 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2021 - 11:52 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2021 - 10:52 PDT
Update
We are continuing to work on a fix for this issue.
Posted Mar 24, 2021 - 09:51 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Mar 24, 2021 - 09:11 PDT
Investigating
We are addressing an infrastructure failure that is affecting a small portion of customer sites.
Posted Mar 24, 2021 - 08:59 PDT
This incident affected: Customer Sites.