Budget Delays
Incident Report for Xandr
Postmortem

Incident Summary

From 07:05 AM to 09:56 AM UTC, Friday, February 5th, 2021, line item budgets experienced instability in our primary data center. This issue was resolved by switching to a secondary data center.

Scope of Impact

During the incident window, some line items could have under and/or over delivered, as their budgets would not have updated in a timely manner.

Timeline (UTC)

2021-02-05 07:05 UTC: Incident Identified: NYM2 data center became unresponsive
2021-02-05 09:05 UTC: Budgeting team received alerts
2021-02-05 09:56 UTC: Incident Resolved: Budgeting task moved from NYM2 data center to LAX1 data center

Cause Analysis

The root cause of the incident was caused by an issue in one of the Budget Pacing processes in the NYM2 data center that required patching.

Resolution Steps

Our engineers resolved the issue by moving the bulk of the processing from one datacenter to the other.

Next Steps

  • Review monitoring mechanisms for NYM2 data center
Posted Feb 11, 2021 - 17:12 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Feb 05, 2021 - 22:01 UTC
Monitoring

We have identified and patched the following issue:

  • Component(s): Bidding, Reporting
  • Impact(s):
    • Updates not reflected for some objects, especially buy-side
    • Some data incomplete or incorrect until reprocessed (please repull data as necessary)
    • Some objects may spend under or over budgets
  • Severity: Minor Outage
  • Datacenter(s): NYM2

We are monitoring our systems closely, and will provide an update as soon as the issue has been fully resolved.

Posted Feb 05, 2021 - 10:40 UTC