Bid Rate Drop in data centers
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 16:52 UTC on April 8th to 23:21 UTC on April 8th, 2021, a maintenance job caused a drop in the amount of traffic our data centers could handle.

Scope of Impact

Invest customers would have seen fewer impressions which would have impacted budgets.

Timeline (in UTC)

  • 2021-04-08 16:52: Maintenance job completed.
  • 2021-04-08 21:22: IM ticket created.
  • 2021-04-08 23:21: Incident resolved and IM ticket closed.

Cause Analysis

The incident was caused by maintenance work. A drop in the limit of traffic our data centers could accommodate resulted in data centers timing out on some bid requests.

Resolution Steps

Our engineers resolved the issue by restoring the amount of traffic our data centers can handle to the original amount.

Next Steps

The maintenance procedure has been amended to prevent any further drops due to routine maintenance.

Posted Apr 22, 2021 - 18:45 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Apr 08, 2021 - 22:16 UTC
Identified

We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.

Posted Apr 08, 2021 - 21:43 UTC
Investigating

We are currently investigating the following issue:

  • Component(s): Bidding, Ad Serving
  • Impact(s):
    • Increase in defaults and PSAs for Console customers
  • Severity: Major Outage
  • Datacenter(s): NYM2, LAX1

We will provide an update as soon as more information is available. Thank you for your patience.

Posted Apr 08, 2021 - 21:27 UTC