Service Issue - Linode LISH Console Fremont (US-West)
Incident
Report for Linode
Postmortem
On November 9th, 2022 at 18:23 UTC, our operations team was alerted to a hardware issue with one of the physical hypervisors located in our Fremont, CA (US-West) data center. At 18:30 UTC, our admin elected to perform a drive transplant to return the impacted services to a functional state in a timely manner. This drive transplant was completed at 19:00 UTC, and after some routine software upgrades, the hypervisor successfully booted all Linodes at 19:26 UTC.
This issue extended to infrastructure that Lish, Weblish, and Glish services are dependent on. As a result these services were unavailable during the maintenance event. We are investigating solutions to improve the redundancy of these services.
Resolved
The disks were successfully transplanted into a new host machine and the service for the LISH and GLISH console has additionally successfully booted. We haven't observed any additional issues with the LISH or GLISH Console, and will now consider this incident resolved. If you continue to experience issues, please open a Support ticket for assistance.
Monitoring
At this time we have been able to correct the issue affecting the GLISH and LISH consoles in Fremont (US-West). We will be monitoring this to ensure that the service remains stable. If you are still experiencing issues please open a Support ticket, call us at 855-454-6633 (+1-609-380-7100 Intl.), or send an email to support@linode.com.
Identified
Our team has identified the issue affecting the LISH and GLISH Console. The hardware that hosts the service experienced an unrecoverable error and the disks are being moved to another host. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
Investigating
Our team is investigating a connectivity issue affecting the LISH and GLISH Consoles in the Fremont Datacenter. During this time, users will experience issues when using these services. We will share additional updates as we have more information.
This incident affected: Linode Manager and API and Regions (US-West (Fremont)).