Description: Flexera One - SaaS Manager was Unavailable in North America

Timeframe: April 7th, 06:23 AM to April 7th, 10:18 AM PDT

Incident Summary

On Thursday, April 7th, we received multiple reports of customers receiving an error stating “There are currently no Managed Apps available” while trying to access Managed Apps in SaaS Manager. This was declared to be a major incident, and technical teams were promptly engaged to assist with the investigation. Technical staff checked the recent deployments and ruled out any recent changes that could have contributed to the outage.

After further investigation, technical staff observed multiple pods were crashing and returning errors. As a result, some of the services in the SaaS Manager cluster were unable to communicate with vault pods and request new vault tokens. The Vault pods came back online after they were rebooted at 7:34 AM PDT, however, the Managed-Apps page was still inaccessible.

Technical staff found that one of the service pods was not getting a response back from the Security Token Manager Service. The Security Token Manager Service pods were rebuilt at 8:17 AM PDT. Services were restored temporarily but the issue recurred shortly after.

Upon further investigation, technical staff found that logs for one of the Web Services were filled with Connection reset errors to the storage service. At 9:37 AM PDT, the impacted service was rebooted, following which services started coming back online.

After additional monitoring, technical staff declared the incident to be resolved at 10:18 AM PDT.

Root Cause

• The failure was caused by the Security Token Manager Services becoming unresponsive causing a downstream impact on other services

Corrective Action

• We are currently working on replacing the Legacy Security Token Manager Service with a new Secrets Manager Service – In Progress