Flexera One – IT Visibility – NA – GoodData Dashboards are Inaccessible
Incident Report for Flexera System Status Dashboard
Postmortem

Description: Flexera One – IT Visibility – NA – GoodData Dashboards were Inaccessible

Timeframe: November 29th, 6:55 PM to December 1st, 1:41 AM PST

Incident Summary

On December 1st, at 12:31 AM PST, technical staff received reports of an issue impacting the IT Visibility GoodData Dashboards. Customers using IT Visibility in the NA region may have received an error while trying to access the ITV GoodData Dashboards. API and ServiceNow Exports were unaffected by this issue.

Investigation revealed that on November 29th, a new feature was introduced in NAM to enable customers to directly upload and access mashup data via ITV GoodData Dashboards. To support this integration, a new role was created with support from the vendor and was later deployed to production.

Technical staff later found that some of the work was still pending on the vendor’s side, and the change was prematurely deployed to production after a miscommunication with the vendor.

On December 1st, at 1:41 AM PST, technical staff reverted the change to restore the service. In addition, health checks and monitoring also confirmed that the GoodData dashboards were accessible again.

Root Cause

• Following a scheduled change, a new feature was introduced in the environment; however, the new access role had not fully propagated to all workspaces
• There was still outstanding work on the vendor’s side and an incomplete data file was sent to Flexera for deployment
• During the authentication process, our system was unable to identify the new role successfully, resulting in an error

Corrective Action

  1. We have successfully reverted the change to promptly restore the service
  2. The vendor will provide clear confirmation of when the new feature is safe to deploy to production
  3. Technical staff will continue their work to enhance automated alerting to ensure these issues are reported and resolved in a timely manner
Posted Dec 13, 2022 - 08:22 PST

Resolved
The change has been reverted successfully. GoodData Dashboards are accessible now and health checks have also confirmed that the service has returned to its stable state. This incident has been resolved.
Posted Dec 01, 2022 - 01:43 PST
Identified
We have identified the issue to be related to a scheduled release. The team is currently working on reverting the change.
Posted Dec 01, 2022 - 01:30 PST
Investigating
Incident Description:
Customers using IT Visibility may receive an error message while trying to access GoodData Dashboards. This is impacting NA customers only.

Priority: 2

Restoration activity:
Technical teams have been engaged and are currently investigating.
Posted Dec 01, 2022 - 01:14 PST
This incident affected: Flexera One - IT Visibility - North America (IT Visibility US).