A defect in our job scheduling library caused heavy database load and led to delays propagating status updates between our backend storage and other Ambra components. The engineering team updated the storage code to improve the caching performance plus upgraded to a newer version of the third-party job scheduling library which resolved a related defect. Once the issue was resolved it took additional time for the backlog of notifications to be processed.
Posted May 25, 2023 - 18:13 EDT
Resolved
The storage changes have been implemented across all storage nodes at this time, and study notifications are now in real time.
Posted May 23, 2023 - 12:41 EDT
Monitoring
The engineering team has begun rolling out the storage change in batches and will be monitoring post-deployment behavior as the change is implemented.
Posted May 23, 2023 - 12:14 EDT
Update
At this time, our storage backlog has decreased. The engineering team is working on deploying a storage change that will further improve performance. This change will be rolled out in stages. Further updates will continue to be posted as available.
Posted May 23, 2023 - 11:47 EDT
Identified
At this time, our engineering team has identified a backlog in our storage in processing study notifications. The team is working towards a resolution and we will provide further updates as they are received.
Posted May 23, 2023 - 11:16 EDT
Investigating
We have received reports of issues on the Ambra platform. Engineering teams are currently investigating. Additional information will be provided as soon as it is available.
Posted May 23, 2023 - 10:49 EDT
This incident affected: Web Services, Image Processing, and Image Viewing.