Incident Summary
From approximately 09:06 Thursday 07 October 2021 UTC to 18:25 Monday 11 October 2021 UTC, some users were seeing no data in our monitoring dashboards.
Scope of Impact
During the incident window, customers who were using the Invest platform for the first time or who we're using a new browser would have been impacted by no data under the columns "Flight Progress" ; "Flight Budget Delivery" ; "Impressions" and "Revenue" in the Invest monitoring grid. Delivery was not affected by this issue.
Timeline (UTC)
2021-10-07 09:06 UTC: First internal report of the Flight Process and Flight Budget Delivery reporting columns in Invest monitoring grid were not displaying
2021-10-07 09:52 UTC: Issue is escalated to Outage and Engineering starts work on the fix
2021-10-07 11:08 UTC: Issue is identified and mitigated in our test environment
2021-10-11 18:25 UTC: The fix is released to clients
Cause Analysis
The incident was caused by a change in a default value used by our monitoring grid. The introduced new default value was not a valid option for the grid, therefore, causing these columns to not show data. The default is only used in cases where users have not previously set a value, which is why only new users or devices would have been affected.
Resolution Steps
Our engineers resolved this issue swiftly by implementing a fix in the code which checked whether or not the time interval values are correct and if not replace with the correct default value.
Next Steps
In order to prevent this from happening the engineering department is working on a code that will check local browser storage for similar invalid values on our components. And in effect minimizing the probability of this happening again in the future.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.