The system was unable to process an update to the Quest compendium because of a 500 error from the Quest side. There is a problem on our side, however, where if anything goes wrong during downloading or parsing, any subsequent codes won’t be there because they are removed.
All times are EST.
| Date/Time | Activity |
|---|---|
| 2022-08-30 08:32 | Started receiving initial reports of issues with Quest test orders. Investigation started. |
| 2022-08-30 09:00 | Support team reached out to Quest to see if there were any issues on the Quest side. |
| 2022-08-30 09:30 | Informed by Quest that they did not have any issues on the Quest side. |
| 2022-08-30 10:22 | Status Page created. |
| 2022-08-30 10:28 | Support team pulled a ticket from a previous occurrence from March 2019. |
| 2022-08-30 10:32 | Problematic scheduled job identified. |
| 2022-08-30 10:34 | Specific error from scheduled job identified. |
| 2022-08-30 12:36 | Restarted scheduled job. |
| 2022-08-30 12:46 | Status Page updated to identified |
| 2022-08-30 12:56 | Verified that issue is resolved |
| 2022-08-30 13:00 | Status Page updated to monitoring |
| 2022-08-30 14:17 | Status Page resolved. |
The management command deletes all the rows from the table so if something goes wrong nothing is left.
Impact was widespread to all customers utilizing Quest for test orders.
This requires development effort from the engineering team.
Management command ran again to fix the problem.