Degraded Workflows
Incident Report for Pantheon Operations
Postmortem

At 16:45 UTC 03 June, an issue with internal certificate provisioning occurred causing workflow failures and dashboard unavailability. Engineers identified the issue at 17:00 and began working on a fix. At 18:20 a fix was applied and normal workflow and dashboard functionality was restored. Engineers monitored availability and the issue was resolved at 20:24 UTC.

We have identified some improvements that will help us detect and prevent similar issues in the future.

Posted Jun 06, 2022 - 14:22 PDT

Resolved
This incident has been resolved.
Posted Jun 03, 2022 - 13:24 PDT
Update
We are continuing to monitor for any further issues.
Posted Jun 03, 2022 - 12:51 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 03, 2022 - 11:23 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Jun 03, 2022 - 11:03 PDT
Update
We are continuing to investigate this issue.
Posted Jun 03, 2022 - 10:29 PDT
Update
We are continuing to investigate this issue. For urgent issues, please contact support via helpdesk@pantheon.io.
Posted Jun 03, 2022 - 10:20 PDT
Update
We are continuing to investigate this issue.
Posted Jun 03, 2022 - 10:16 PDT
Investigating
Our monitoring has detected elevated response times for some workflows, which may slow backups and clone operations.
Posted Jun 03, 2022 - 10:08 PDT
This incident affected: Dashboard, Workflow Operations, Terminus Operations, and Autopilot.