Incident Summary
From approximately 21:37 UTC on Thursday, Jan 12 to 10:21 UTC on Friday, Jan 13, 2023. A reduction in bids and drop in delivery from external bidders was observed during the impact window

Incident Impact
• Nature of Impact: High timeout rates across all external bidders and a 20% drop in auction transactions.
• Timeframe: ~12.73 Hours. 21:37 UTC on Thursday, Jan 12 to 10:21 UTC on Friday, Jan 13, 2023
• Customers Impacted: External bidders
• Scope: Limited to AMS3
• Magnitude: Only external bidders were affected

Timeline (UTC)
• 2023-01-12 18:00: Incident Started. Our engineering team received the alerts and engaged the vendors to investigate the issue
• 2023-01-12 14:18: IM Ticket Created
• 2023-01-12 14:28: Escalated to Engineer. Shifted the outbound specific traffic to a different service provider to avoid latency-prone paths.
• 2023-01-13 21:28: Incident Resolved

Cause Analysis
A reduction in bids and drop in delivery from external bidders was observed due to a hardware issue that was observed on the service provider’s end.

Resolution Steps
• Shifted the outbound specific traffic to a different service provider to avoid latency-prone paths.

Follow-Up Items
• Engineering team followed-up with the vendors to put both circuits back on redundant devices to help avert similar occurrences in future that would potentially result in causing an IM.