During a scheduled database operation, customers were unable to save appointment, billing, and chart information.
| Date/Time (All PST) | Activity |
|---|---|
| 2022-05-07 10:20 | Scheduled index creation operation started. |
| 2022-05-07 10:20 | Operations team monitored index creation activity. |
| 2022-05-07 10:54 | Support team started to receive tickets about inability to save appointment, billing, and chart information. |
| 2022-05-07 11:32 | Operation is completed on the primary database server and is initiated on the read replicas. This blocks all other updates on the replicas and causes replication lag to grow. |
| 2022-05-07 11:38 | Status page created to reflect identified status. |
| 2022-05-07 11:54 | Status page updated to reflect progression of index creation. |
| 2022-05-07 12:21 | Status page updated to reflect the near completion of the index creation. |
| 2022-05-07 13:03 | Status page updated to monitoring. |
| 2022-05-07 13:50 | Status page updated to resolved. |
The scheduled index creation that was ran caused our databases to block (and queue) any other transactions from occurring until the index was fully created.
Customers were unable to save appointment, billing, and chart information.
Moving forward, all database scheduled operations will use a different schema change tool and this will ensure tables can be altered without blocking read or write operations. Additionally, scheduled operations will have maintenance pages in place during the scheduled operations windows as well as be scheduled outside business hours or when traffic volume is typically low.