Incident Summary
From approximately 07:27 UTC on Thursday, April 8 to 17:40 UTC on Friday, April 9, 2021, data in the buyer monitoring workflow UI was stale as as a result of database maintenance.
Scope of Impact
During the incident window, users would have seen stale object (insertion order and line item) metadata and some stale reporting data in the buyer monitoring workflow UI.
Timeline (UTC)
2021-04-09 07:27: Incident Started and internal ticket was raised
2021-04-09 07:38: Incident escalated to Engineers
2021-04-09 07:40: Engineer identified a quick workaround by reverting database changes
2021-04-09 07:43: Client facing impact was resolved and communications were sent out
2021-04-09 12:14: Issue was re-escalated to Engineers
2021-04-09 16:23: Engineers identified that the cache-invalidator was not running due to the application user's permissions issue
2021-04-09 17:40: Incident Resolved: The application user was granted necessary permissions
2021-04-12 07:44: Issue was confirmed to be resolved and incident ticket was closed
Cause Analysis
The root cause of the incident was due to database maintenance, which inadvertently introduced a misspelled schema name. This misspelling enabled a set of data operations on the misspelled schema, impacting data syncing.
Resolution Steps
Our engineers resolved the issue by dropping and recreating all the necessary application users, which restarted database validation of the full schema-name.
Next Step(s)
• Implement more robust alerting and monitoring for the impacted application
The incident has been fully resolved. We apologize for the inconvenience this issue may have caused, and thank you for your continued support.
We have patched the issue and are monitoring our systems closely. We will provide an update as soon as the issue has been fully resolved.
We have identified the following issue:
Our engineers are actively working towards a resolution, and we will provide an update as soon as possible. Thank you for your patience.