On April 9, 2024, Inkling restarted the central API services after adding additional capacity. Following the restart, the services that process central queue items failed to connect to the queue properly. This caused items in the queue to have their processing delayed. This was mostly seen in a delay publishing Inkdocs.

To resolve the issue, Inkling engineers reconnected the services to the queue. By 10:30 AM PT on April 10, 2024, the services processed the backlog and resumed normal service. No data was lost in this process, only delayed. Changes to the queueing system were made to ensure this does not reoccur.

All previously delayed project publications have been processed, and there are no longer any delays expected for new tasks.

At 11:58 AM PT on April 9, 2024, a scaling event reduced the number of Inkling's main API instances in response to an unusual reduction in traffic. When traffic increased, the scaling increased the number of instances but those took some time to come online. During this time, the remaining instances were overwhelmed, initially leading to some intermittent timeouts and ultimately to 8 minutes of platform downtime between 12:30 and 12:38 PM PT.

To mitigate the immediate problem, Engineering added capacity and restarted the core internal services. This restored the platform. The scaling logic has been updated to ensure sufficient capacity at all times.