At approximately 22:00 UTC on 11/08/2023, we became aware of an issue preventing a small number of Linodes using specific plan types (Dedicated, Premium, and GPU) from completing shutdown/reboot jobs. The shutdown jobs themselves were determined to be running successfully with proper cleanup, but were hanging and not being marked completed, ultimately preventing new boot jobs from being issued. The problem was found to be related to a small part of a larger host code change that was deployed earlier that day. All changes had been fully tested and this behavior was not present before the release. To mitigate the issue, this specific feature was reverted on the entire fleet by 01:38 11/09/2023. The status page was then updated to monitoring and eventually resolved after no additional issues were observed.
This change will not be reintroduced until the issue has been fully recreated and our testing methodology has been adjusted to account for this in the future.