Incident Summary
From approximately 16:39 UTC on Tuesday, October 12th, 2021 to 15:46 UTC on Thursday, October 14th, 2021, a database change failure caused some clients to see out-of-date data in the Buyer Monitoring Workflow screen. Additionally, some clients have seen newly created insertion orders and line items fail to show up in overview screens and changes to these objects not go into effect.
Scope of Impact
During the incident window some clients may have seen out-of-date data in the Buyer Monitoring Workflow screen. Additionally, some clients have seen newly created insertion orders and line items fail to show up in overview screens and changes to these objects not go into effect.
Timeline (in UTC)
2021-10-12 16:39: Incident Started: New database column added
2021-10-12 16:52: Escalated to engineering
2021-10-12 18:27: Database alteration rolled back
2021-10-12 21:06: Database cluster in LAX recovered
2021-10-12 21:47: Incident ticket created
2021-10-14 15:46: Incident resolved
Cause Analysis
The incident was caused by an inadvertent database change causing data replication failures. As a result, database replica's need to be replaced, resulting in downtime for downstream dependencies.
Resolution Steps
Our engineers resolved the issue by migrated the impacted database to a different host and re-syncing downstream dependencies.
Next Steps
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have identified the following issue:
Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.