Flexera One – EU – SaaS Manager is Currently Unavailable
Incident Report for Flexera System Status Dashboard
Postmortem

Description: Flexera One – SaaS Manager – EU – Error Accessing Managed SaaS Applications

Timeframe: October 31st, 7:04 AM to October 31st, 11:52 AM PDT

Incident Summary:

On October 31st, at 7:04 AM PDT, the technical staff was notified that the Managed SaaS Applications were inaccessible in the EU. This was declared to be a major incident, and additional SMEs were engaged to assist with the investigation.

Upon further investigation, at 10:12 AM PDT, technical staff found that requests to one of the network interface target IP addresses were unsuccessful. At 10:29 AM PDT, the IP address was de-registered from the API gateway's target group, following which the subsequent requests were successful.

At 11:52 AM PDT, the IP address was re-registered to the target group. Staff monitored the environment for the next hour and did not observe any further errors in the logs, and the application continued to operate normally. After additional health checks, the incident was declared resolved.

Root Cause:

The root cause has been identified to be the misconfiguration of one of the subnets in the network which may lead to dropped requests

Corrective Action:

  1. The impacted IP was de-registered and re-registered to remediate the issue
  2. We have engaged our network team and additional SMEs to review the existing network configuration and reconfigure it as needed
Posted Dec 19, 2022 - 15:30 PST

Resolved
This incident has been resolved and transitioned to a problem management investigation. We have been monitoring the environment since yesterday and have not observed any further issues. We are also working with the vendor to investigate the root cause to prevent future reoccurrence.
Posted Nov 01, 2022 - 09:20 PDT
Monitoring
At 11:52 AM PDT, the IP was re-registered to the target group. Health checks have confirmed that we are no longer experiencing the issue, and the environment has been stable. We will continue to monitor the environment for the next few hours to prevent any reoccurrence. We have also opened a case with the vendor, and a call will be scheduled to investigate the root cause.
Posted Oct 31, 2022 - 12:46 PDT
Update
After further analysis, technical staff found that requests to one of the network interfaces IP addresses were unsuccessful. The IP was de-registered from the API gateway's target group, following which the subsequent requests have been successful. A case has been opened with the Vendor and logs have been provided to them for further assistance.
Posted Oct 31, 2022 - 11:50 PDT
Update
We are continuing to isolate the problems to identify the root cause
Posted Oct 31, 2022 - 11:07 PDT
Update
The investigations are still ongoing. Further analysis has shown that calls to one of the API endpoints are failing intermittently. We are investigating further.
Posted Oct 31, 2022 - 09:40 PDT
Update
Technical teams found that the resource limit on one of the nodes was exceeded. We attempted to restart the pods on the impacted node but we are still seeing intermittent issues. Teams are investigating further.
Posted Oct 31, 2022 - 08:48 PDT
Investigating
Incident Description: We are currently experiencing issues with the SaaS Manager application in the EU region. Customers may receive an error while trying to access Managed SaaS Applications.

Error “An unexpected error occurred while loading managed applications”

Priority: 1

Restoration activity: Technical teams have been engaged and are investigating
Posted Oct 31, 2022 - 07:34 PDT
This incident affected: Flexera One - IT Asset Management - Europe (IT Asset Management - EU SaaS Manager).