On November 9, 2021, asynchronous scoring delays affected a small subset of customers for 3.25 hours. (94.24%, or 49.16 million, of scoring messages were processed in less than a second on November 9.) This was caused by temporary database locks due to inefficient queries that processed unexpectedly large database operations (joins and index creations).
Immediate resolution was achieved by increasing the number of database sync workers and AWS Relational Database Service instances to work through the scoring backlog. Ongoing work is continuing, as we analyze and improve database queries in the affected areas.