Ad server monetization was affected for clients using AST between 13-17:30 UTC on April 28, 2021
Incident Report for Xandr
Postmortem

Incident Summary
From approximately 13:00 to 17:30 UTC on Wednesday, April 28, 2021 ad server monetization was impacted for clients using AST.

Scope of Impact
During the incident, ad server clients using AST likely saw a decrease in seller revenue and CPM's.

Timeline (UTC)
2021-04-28 13:00: Incident Started: Clients noticed a drop in impressions
2021-04-28 13:40: The Engineering team was alerted
2021-04-28 17:30: Incident Resolved: Engineering rolled back an AST release
2021-04-29 14:30: Retroactive incident ticket was opened

Cause Analysis
The root cause of the incident was due to an issue with an AST release where a flag was removed within the AST configuration causing cookies to not send in the header.

Resolution Steps
The issue was resolved by rolling back the AST release to a previous version.

Next Step(s)
• Implement alpha testing to identify future issues with AST releases
• If testing goes well, we hope to release the new AST version early next week

Posted Apr 30, 2021 - 18:55 UTC

Resolved
This incident has been resolved.
Posted Apr 29, 2021 - 16:09 UTC
Monitoring

We have identified and patched the following issue:

  • Component(s): Ad Serving using AST
  • Impact(s):
    • Reduction in seller revenue
  • Severity: Major Outage
  • Datacenter(s): Global

We are monitoring our systems closely, and will provide an update as soon as the issue has been fully resolved.

Posted Apr 29, 2021 - 15:21 UTC