Drop in Ad Serving
Incident Report for Xandr
Postmortem

Incident Summary
From approximately 00:00 to 01:18 UTC on Friday January 29, 2021, an increase in throttled auctions causing a drop in Ad Serving.

Scope of Impact
During the incident window, users may have experienced a noticeable increase in processed auctions, followed by a noticeable decrease in processed auctions.

Timeline (in UTC)
2021-01-29 00:00: Incident Started: Abnormal auction throttling behavior
2021-01-29 00:05: Internal incident alerts
2021-01-29 00:11: Incident escalated to engineering
2021-01-29 00:36: Preemptive mitigation steps taken
2021-01-29 00:46: Root cause identified
2021-01-29 00:52: Incident Mitigated: Auction metrics back to normal
2021-01-29 01:02: Incident starts again, confirming root cause
2021-01-29 01:18: Incident Resolved

Cause Analysis
The root cause of this incident was due to downtime experienced by our internal budget pacing controller, which caused traffic to behave abnormally. Internal intelligent auction throttling reacted to the change by evaluating more, than less, auctions than usual.

Resolution Steps
Our engineers resolved the immediate issue by using previous intelligent throttling data to normalize the flow of traffic.

Next Steps
Improve detection for changes to auction traffic

Posted Feb 16, 2021 - 22:14 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jan 30, 2021 - 03:38 UTC
Monitoring

We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.

Posted Jan 30, 2021 - 01:50 UTC
Identified

We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.

Posted Jan 30, 2021 - 01:30 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Ad Serving
  • Impact(s):
    • Decrease in requests sent to Bidder customers
    • Drop in delivery on external supply
  • Severity: Major Outage
  • Datacenter(s): Global

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jan 30, 2021 - 00:53 UTC