Connectivity Issue - Singapore
Incident
Report for Linode
Postmortem
On Sep 14, 2022, a cabinet in the Singapore data center experienced a power issue that temporarily prevented ~85% of the Linodes from reaching the network. At 18:02 UTC, a power event occurred that impacted both PDUs in a rack. By 18:16 UTC, non-redundant power was restored and the impacted Linodes started to regain network access.
From that point on, our team performed ongoing maintenance to completely replace the impacted PDUs to ensure this did not happen again, and that full redundancy was restored at 01:00 UTC on Sep 15, 2022.
We are actively investigating the source of the power issue and will be discussing this specific incident with the PDU vendors to fully determine the root cause and ensure this will not occur again.
Resolved
We haven’t observed any additional connectivity issues in our Singapore data center, and will now consider this incident resolved. If you continue to experience problems, please open a Support ticket for assistance.
Monitoring
The emergency maintenance in our Singapore data center has completed. We will be monitoring this to ensure that it remains stable. If you are still experiencing issues, please open a Support ticket for assistance.
Update
The maintenance is ongoing and we will keep this page updated as we have more information.
Update
We have scheduled an emergency maintenance window to resolve this issue.
While there is currently no customer impact and no impact expected, you may experience packet loss during this maintenance window.
While there is currently no customer impact and no impact expected, you may experience packet loss during this maintenance window.
Identified
Our team has identified the issue affecting connectivity in our Singapore data center. It appears that the issue is related to infrastructure power intake, but we are continuing to review the situation to confirm this. We are working quickly to implement a fix, and we will provide an update as soon as the solution is in place.
Investigating
Our team is investigating a connectivity issue in our Singapore data center. During this time, users may experience connection timeouts and errors for all services deployed in this data center. We will share additional updates as we have more information.
This incident affected: Regions (AP-South (Singapore)).