Incident Summary
On Sunday, November 28th, 2021, ad serving in NYM2 was temporarily limited due to an issue with OpenRTB Macro processing in win notifications due to an unexpected macro present in the bid response.
Scope of Impact
During the incident window ad serving was limited in the NYM2 datacenter.
Timeline (UTC)
2021-11-28 15:37:00: Incident Started
2021-11-28 15:56:00: Alert that the impression bus was 70% Percent in NYM2
2021-11-28 16:15:00: Incident created
2021-11-28 16:27:00: Engineering team begins works on a fix
2021-11-28 17:00:00: Problem bidder de-activated
2021-11-28 17:09:00: Fix is released
2021-11-28 17:30:00: Ad serving returns to normal
2021-11-28 17:37:00: Incident resolved
2021-11-29 11:30:00: Bidder is re-activated after resolving the issue
Cause Analysis
A bidder began to send win notifications (nurls) with an erroneous macro in bid responses. This revealed an issue with OpenRTB Macro processing causing impression bus crashes, reducing ad serving.
Resolution Steps
We altered how we process OpenRTB macros in bid response to prevent the problem in the future. The offending bidder was temporarily disabled until they resolved the issue on their end.
Next Steps
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.