Incident Summary:

On Tuesday, February 2nd at 05:56 UTC, customers reported having video creatives that were stuck in an inactive state. Inactive creatives can't serve and therefore limit potential spend. The creatives were stuck in an inactive state because their hosted media assets were not marked as "ready" for serving. The creatives media assets were not marked as "ready" due to an internal bug in our application that blocked asset processing indefinitely.

Scope of Impact:

During the incident window, customers would not have been able to "activate" video creatives.

Timeline (UTC):

2021-02-01 21:26 : Incident started
2021-02-02 07:56 : IM Ticket created
2021-02-02 08:03 : Escalated to Engineer
2021-02-02 08:49 : Incident resolved
2021-02-04 15:30 : Client communication on resolved status posted

Cause Analysis:

A number of video files failed to transcode and caused the queue to stop being processed which left many assets waiting. It was due to the high volume of file processing leading to one of them getting stuck sometimes.

Resolution Steps:

Our engineers resolved the issue by manually re-triggering post-processing on the media assets of the stuck creatives and arrived at a complete resolution by restarting an internal server which appeared stuck.

Next Steps:

The issue was due to a high volume of file processing which lead to one of them getting stuck for a random reason (sometimes) and others had to wait. Will work on improving how the system works to prevent recurrence.