Invest LI Monitoring Dashboard missing data on some columns
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 09:06 Thursday 07 October 2021 UTC to 18:25 Monday 11 October 2021 UTC, some users were seeing no data in our monitoring dashboards.

Scope of Impact

During the incident window, customers who were using the Invest platform for the first time or who we're using a new browser would have been impacted by no data under the columns "Flight Progress" ; "Flight Budget Delivery" ; "Impressions" and "Revenue" in the Invest monitoring grid. Delivery was not affected by this issue.

Timeline (UTC)

2021-10-07 09:06 UTC: First internal report of the Flight Process and Flight Budget Delivery reporting columns in Invest monitoring grid were not displaying
2021-10-07 09:52 UTC: Issue is escalated to Outage and Engineering starts work on the fix
2021-10-07 11:08 UTC: Issue is identified and mitigated in our test environment
2021-10-11 18:25 UTC: The fix is released to clients

Cause Analysis

The incident was caused by a change in a default value used by our monitoring grid. The introduced new default value was not a valid option for the grid, therefore, causing these columns to not show data. The default is only used in cases where users have not previously set a value, which is why only new users or devices would have been affected.

Resolution Steps

Our engineers resolved this issue swiftly by implementing a fix in the code which checked whether or not the time interval values are correct and if not replace with the correct default value.

Next Steps

In order to prevent this from happening the engineering department is working on a code that will check local browser storage for similar invalid values on our components. And in effect minimizing the probability of this happening again in the future.

Posted Oct 13, 2021 - 15:17 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Oct 08, 2021 - 14:12 UTC
Identified

We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.

Posted Oct 07, 2021 - 11:34 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Invest Monitoring Workflows
  • Impact(s):
    • Multiple columns on the Invest Monitoring Dashboard are missing data. Delivery is not impacted
  • Severity: Minor Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Oct 07, 2021 - 10:05 UTC