SA IdP Dashboard Data is not Updating
Incident Report for SecureAuth Service
Postmortem

We apologize for any inconvenience experienced during this incident, and we treat such disruptions seriously. Below is a summary of the issue, a root cause, and what we are doing to improve service going forward.

Incident Description

A TLS certificate that is used on an endpoint for SA IdP dashboard metric data collection expired on May 9. This meant HTTPS connections to this endpoint failed and caused SA IdP dashboard metric data to become outdated, degrading service for part of the administrative console. After some customers reported an issue with outdated metrics, the SecureAuth team investigated and fixed the issue by renewing the TLS certificate. The new certificate was in place by 6:10 PM on May 12. Data was refreshed around 1:30 AM on May 13, allowing the normal metric refresh period to run its course.

Root Cause

The DevOps team recently replaced several expiring certificates, and this one was missed due to a monitoring gap.

Corrective Actions

The DevOps team is following up by ensuring the TLS certificate monitoring gap is addressed and by reviewing additional monitoring requirements for the endpoint in question.

Note: All times/dates are UTC.

Posted May 16, 2023 - 15:39 PDT

Resolved
Dashboards are again populating with data - if you continue to have issues, please contact support@secureauth.com
Posted May 12, 2023 - 19:15 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted May 12, 2023 - 11:28 PDT
Identified
The issue was an expired certificate within the backend services. The certificate has been updated and the job to update the dashboards will run later this afternoon to update the dashboards. Dashboards should be updated at roughly 1900 PST.
Posted May 12, 2023 - 11:28 PDT
Investigating
The Dashboard data for the SA IdP is not updating for login information.

This is not impacting any SA IdP services or logins, only the reporting mechanism.
Posted May 12, 2023 - 10:32 PDT
This incident affected: SecureAuth Titan Services (Dashboard Service).