Incident Summary
From approximately 00:00 to 01:18 UTC on Friday January 29, 2021, an increase in throttled auctions causing a drop in Ad Serving.
Scope of Impact
During the incident window, users may have experienced a noticeable increase in processed auctions, followed by a noticeable decrease in processed auctions.
Timeline (in UTC)
2021-01-29 00:00: Incident Started: Abnormal auction throttling behavior
2021-01-29 00:05: Internal incident alerts
2021-01-29 00:11: Incident escalated to engineering
2021-01-29 00:36: Preemptive mitigation steps taken
2021-01-29 00:46: Root cause identified
2021-01-29 00:52: Incident Mitigated: Auction metrics back to normal
2021-01-29 01:02: Incident starts again, confirming root cause
2021-01-29 01:18: Incident Resolved
Cause Analysis
The root cause of this incident was due to downtime experienced by our internal budget pacing controller, which caused traffic to behave abnormally. Internal intelligent auction throttling reacted to the change by evaluating more, than less, auctions than usual.
Resolution Steps
Our engineers resolved the immediate issue by using previous intelligent throttling data to normalize the flow of traffic.
Next Steps
Improve detection for changes to auction traffic
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the cause of the issue, and our engineers are actively working towards a resolution. We will provide an update as soon as possible. Thank you for your patience.
We are currently investigating the following issue:
We will provide an update as soon as more information is available. Thank you for your patience.