Incident Summary:

Between June 25 (Friday) 15:00 UTC and June 28 05:00 UTC, clients experienced issues accessing the Deals dashboard in Monetize UI (stuck in "loading"), drop in volume of bid requests from some sellers, incorrect data in the deal metrics reports. This was due to a change deployed to one of the data processing jobs in New York data center.

Scope of Impact:

During the incident window clients would have experienced the following impacts:

• Unable to access the deals (with buyers) in Monetize UI in the deals dashboard
• Incorrect data in the Deal Metrics reports
• Drop in the volume of bid requests from some sellers

Timeline (UTC):

2021-06-25 15:00 : Change to data processing job was deployed
2021-06-28 02:00 : IM Ticket created
2021-06-28 05:00 : Patches deployed, data synced by Engineers
2021-06-28 05:00 : Incident resolved

Cause Analysis:

The changed deployed to the data processing job was not compatible with another application downstream (which consumed the output of the job) causing these issues.

Resolution Steps:

Our engineers resolved the issues by upgrading the job to a version which was compatible with all downstream consumers, deployed a patch to fix the deals UI issue and synced data with data from LA data center which was not impacted by this issue to fix the data in deal metrics reports.

Next Steps:

Teams to validate and upgrade compatibility testing runbook, and to add more robust monitoring to detect similar problems in the future.