Adserving down in AMS1, FRA1, and LAX1

Incident Report for Xandr

Postmortem

Incident Summary
From approximately 15:21 - 17:00 UTC, on Thursday, June 24, 2021 FRA1, AMS1 and LAX1 had partial impbus downtime

Scope of Impact
During the incident window, ad serving was partially disrupted across FRA1, AMS1 and LAX1 datacenters

Timeline (UTC)
2021-06-24 15:21: Incident Started: Impbus started failing FRA1, AMS1 and LAX1
2021-06-24 15:31: Issue identified, and offending tags disabled
2021-06-24 16:21: Prebid bidder was disabled and impbus started to recover
2021-06-24 16:24: Hotfix kicked off
2021-06-24 17:00: Incident Resolved: Impbus recovered to normal levels.
2021-06-24 22:42: Prebid bidder re-enabled.

Cause Analysis
An error condition was not handled gracefully in accept bid calls that were from some of the video impressions which assumed that creative was always set and valid.

Resolution Steps
Our engineers resolved the issue by pushing a hot fix and restarting the impacted instances of impression bus.

Next Steps
Improve detection, monitoring, and alerts for changes in impression bus traffic.

Posted Jun 29, 2021 - 15:38 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Jun 24, 2021 - 17:08 UTC

Investigating

We are currently investigating the following issue:

Component(s): Bidding, Ad Serving
Impact(s):
- Increase in defaults and PSAs for Console customers
- Decrease in requests sent to Bidder customers
- Some objects may spend under budgets
- Drop in delivery on external supply
Severity: Major Outage
Datacenter(s): FRA1, AMS1, LAX1, SIN3

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Jun 24, 2021 - 15:44 UTC