Some Recently Created Buy-Side Objects Not Appearing in UI
Incident Report for Xandr
Postmortem

Incident Summary

From approximately 16:39 UTC on Tuesday, October 12th, 2021 to 15:46 UTC on Thursday, October 14th, 2021, a database change failure caused some clients to see out-of-date data in the Buyer Monitoring Workflow screen. Additionally, some clients have seen newly created insertion orders and line items fail to show up in overview screens and changes to these objects not go into effect.

Scope of Impact

During the incident window some clients may have seen out-of-date data in the Buyer Monitoring Workflow screen. Additionally, some clients have seen newly created insertion orders and line items fail to show up in overview screens and changes to these objects not go into effect.

Timeline (in UTC)

2021-10-12 16:39: Incident Started: New database column added
2021-10-12 16:52: Escalated to engineering
2021-10-12 18:27: Database alteration rolled back
2021-10-12 21:06: Database cluster in LAX recovered
2021-10-12 21:47: Incident ticket created
2021-10-14 15:46: Incident resolved

Cause Analysis

The incident was caused by an inadvertent database change causing data replication failures. As a result, database replica's need to be replaced, resulting in downtime for downstream dependencies.

Resolution Steps

Our engineers resolved the issue by migrated the impacted database to a different host and re-syncing downstream dependencies.

Next Steps

  • Review and improve standard operating procedures.
  • Engage further with third party partners to add features that support graceful application changes.
Posted Nov 19, 2021 - 18:17 UTC

Resolved

The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.

Posted Oct 12, 2021 - 22:38 UTC
Identified

We have identified the following issue:

  • Component(s): Buy-side pages
  • Impact(s):
    • Some buyside objects not appearing in Grid Screen. Please use search to navigate to impacted objects
  • Severity: Minor Outage
  • Datacenter(s): Global

Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.

Posted Oct 12, 2021 - 21:52 UTC