Incident Summary

From approximately 17:00 UTC on Thursday, Jan 12 to 20:30 UTC on Thursday, Jan 12, 2023, a 6 Hour SLA breach caused a delay in Key Value Auction and Standard Feed’s.

Incident Impact

Nature of Impact: Stale data reporting

Timeframe: ~3.5 Hours. 17:00 UTC on Thursday, Jan 12 to 20:30 UTC on Thursday, Jan 12, 2023

Customers Impacted: Global

Scope: Global

Magnitude: All customers were impacted

Timeline (UTC)

2023-01-12 17:00: Incident started.

2023-01-12 17:37: IM Ticket Created

2023-01-12 17:45: Escalated to Engineer

2023-01-12 20:30: Incident Resolved

Cause Analysis

The exact root cause of this issue is still unknown however, the potential reason could have been due to a 3 hour timeout duration that maxed out during a client side copy operation, coupled with a slight delay in reporting the failure by our internal alerting tool.

Resolution Steps

The job resumed its operation and the timeouts were resolved on its own during the subsequent runs.

Follow Up Items

• Inspect relevant alerts to ensure the alerts are set with better threshold parameters and add detailed introspection to trace the origin of the issue.
• Defined a process to explicitly state the conditions which would allow us to supersede the failed job.
• Reduced the 3 hours timeout duration on the client side copy function and improved the logging process to avert similar occurrences in future that would potentially result in causing an IM.