Incident Summary

From approximately 14:00 UTC on Monday, Nov 28 to 16:33 UTC on Monday, Nov 28, 2022. Xandr Digital Platform UIs (Invest, Monetize and Curate) pages started to take longer to load. A tracking pixel implemented on the platform began to block browser requests until the previous request was finished.

Incident Impact

• Nature of Impact: All web-based functionality was slow to load/intermittently unavailable.
• Timeframe: ~2.55 Hours. 14:00 UTC on Monday, Nov 28 to 16:33 UTC on Monday, Nov 28, 2022
• Scope: Global
• Magnitude: All customers were impacted.

Timeline (UTC)

• 2022-11-28 14:00: Incident Started. Our Engineering team received the alert from our automated monitoring tool.
• 2022-11-28 14:16: IM Ticket Created.
• 2022-11-28 16:20: Incident Resolved. Vendor resolved the issue on their end.
• 2022-11-28 16:33: IM Ticket Closed

Cause Analysis

This was caused by an asset being loaded from a third party vendor that blocked other browser requests. Rather than operating asynchronously, the tracking pixel was loaded via a CSS directive which caused the browser to wait until the request was completed before loading any other assets or requests.

Resolution Steps

• Third party vendor resolved the issue on their end.

Follow-Up Items

• Remove dependency on third party tracker code from all repos
• An nginx substitution pattern was written to scrub out the pixel import statement so our CSS file wouldn’t attempt to load the tracking pixel.