Infrastructure Issue Affecting Customer Sites
Incident Report for Pantheon Operations
Postmortem

On Monday, January 18th starting at approximately 11:26 AM PST, Pantheon experienced malicious load-generating traffic that caused resource exhaustion on a sufficient portion of Pantheon’s redundant infrastructure to have some uptime impact on 27% of customer sites. Pantheon engineers began mitigations shortly after the malicious traffic was first observed, and all sites were operating normally by 12:34 PM. 

Pantheon is improving its internal processes to detect and resolve such issues faster in the future. Tasks to mitigate such incidents have also been identified and will be prioritized.

Posted Jan 21, 2021 - 09:32 PST

Resolved
This incident has been resolved.
Posted Jan 18, 2021 - 13:57 PST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 18, 2021 - 13:21 PST
Update
Almost all customer sites are back up and we are continuing to work on a fix for this issue.
Posted Jan 18, 2021 - 13:07 PST
Identified
The issue has been identified and a fix is being implemented.
Posted Jan 18, 2021 - 12:23 PST
Investigating
We are addressing an infrastructure failure that is affecting a small portion of customer sites.
Posted Jan 18, 2021 - 11:46 PST
This incident affected: Customer Sites.