Infrastructure affecting Dashboard Availability
Incident Report for Pantheon Operations
Postmortem

On April 20, 2021 at 8:46 am PDT, we observed an increase in failed Dashboard logins and Terminus authentication errors. These issues were quickly traced to our user authentication service, which had experienced performance degradation at the database level. We coordinated with our upstream authentication service provider and restored service by 12:50 pm.

We have identified improvements to increase the resilience of the authentication service, and we have deployed a fallback authentication path which could be utilized by most customers in the event of a future issue with our primary service.

We understand that access to the Dashboard and Terminus are essential for our customers to develop on the Pantheon platform, and we apologize for the disruption.

Posted Apr 26, 2021 - 12:57 PDT

Resolved
This incident has been resolved.
Posted Apr 20, 2021 - 15:21 PDT
Update
We are continuing to monitor for any further issues.
Posted Apr 20, 2021 - 14:16 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 20, 2021 - 13:31 PDT
Update
We are continuing to work on a fix for this issue.
For urgent issues please contact support via email at helpdesk@pantheon.io.
Posted Apr 20, 2021 - 13:10 PDT
Update
We are continuing to work on a fix for this issue.
For urgent issues please contact support via email at helpdesk@pantheon.io.
Posted Apr 20, 2021 - 12:03 PDT
Update
We are continuing to work on a fix for this issue.
For urgent issues please contact support via email at helpdesk@pantheon.io.
Posted Apr 20, 2021 - 10:45 PDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 20, 2021 - 10:10 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Apr 20, 2021 - 09:08 PDT
Update
We are continuing to investigate this issue.
Posted Apr 20, 2021 - 09:05 PDT
Investigating
We are investigating an incident impacting logging into our dashboard. We are continuing to investigate and will update this incident with more details as they are available.
Posted Apr 20, 2021 - 09:00 PDT
This incident affected: Dashboard, Support Tickets, and Terminus Operations.