Incident Summary
From approximately 17:05 UTC on Thursday, May 12, 2022 to 13:18 UTC on Friday, May 13, 2022 UTC, Video Analytics reporting experienced unexpected delays due to a planned reduction in server capacity.
Incident Impact
Nature of Impact: Video Analytics report users experienced approximately a one-hour delay in data delivery above the contracted SLA of 6 hours.
Timeframe: The delay was identified at 03:44 on Friday, May 13, and was resolved as of 13:18 the same day.
Customers Impacted: Invest users
Scope: Global
Magnitude: All customers
Timeline (UTC)
2022-05-13 03:44: Xandr teams identified a delay in processing of reporting jobs and escalated the problem.
2022-05-13 03:44-10:33: Xandr teams investigated job delays and isolated the cause to the reduction in server capacity.
2022-05-13 10:33: Xandr teams restored server capacity to remediate job reporting delays.
2022-05-13 13:18: Incident resolved.
Cause Analysis
The incident occurred due to unexpected performance degradation caused by a planned reduction in server capacity.
Resolution Steps
Xandr teams resolved the issue by restoring server capacity.
Next Steps
Incorporate additional monitoring to proactively ensure server capacity at all times.
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.