Incident Detected - Azure West US 2 Region Connectivity Issues - December 14, 2022 20:16 UTC
Incident Report for Workspot
Resolved
Azure has resolved the incident.
Their last update:
They have identified that multiple top-of-rack (ToR) network devices, connecting a single rack of servers, experienced an inadvertent power failure. A top-of-rack (ToR) network device, connecting a single rack of servers, experienced a fault.

They have restored the power to the top-of-rack (ToR) network devices to mitigate the issue. A traffic routing normally, there should be no further connectivity issues expected.

If you continue to notice connectivity issues or VM rebooting unexpectedly in West US2, please open a support ticket through the Workspot Control Support Portal.
Posted Dec 15, 2022 - 15:26 UTC
Monitoring
This incident is now mitigated. Between 20:16 UTC on December 14, 2022 and 00:39 UTC on December 15, 2022, Virtual Machines in West US 2 experienced connectivity issues. Affected Virtual Machines may have restarted unexpectedly during this incident.

Root cause: A network device experienced a fault, resulting in network connectivity loss to downstream Virtual Machines.

Mitigation: Service healing was triggered to automatically redeploy affected Virtual Machines to healthy infrastructure.

We are continuing to monitor the situation and will close the incident as resolved if no further issues are reported by our customers or Azure.
Posted Dec 15, 2022 - 00:42 UTC
Update
We are still waiting on an update from Azure.
Last update was:
Azure have identified that multiple top-of-rack (ToR) network devices, connecting a single rack of servers, experienced an inadvertent power failure. The Azure resources connected to these ToRs were shut down. Customers may have been unable to connect to their resources and Virtual Machines (VMs) that were deployed onto the affected nodes may have experienced a restart. They are actively working to return the ToRs to a normal state. An update will be provided in 60 minutes or as events warrant.
Posted Dec 15, 2022 - 00:02 UTC
Update
Azure have identified that multiple top-of-rack (ToR) network devices, connecting a single rack of servers, experienced an inadvertent power failure. The Azure resources connected to these ToRs were shut down. Customers may have been unable to connect to their resources and Virtual Machines (VMs) that were deployed onto the affected nodes may have experienced a restart. They are actively working to return the ToRs to a normal state. An update will be provided in 60 minutes or as events warrant.
Posted Dec 14, 2022 - 21:58 UTC
Update
Azure is continuing to work to reroute traffic around the affected network device. We are still seeing issues connecting to some VMs in West US 2 region of Azure.
We will update this incident every 30 minutes or as warranted.
Posted Dec 14, 2022 - 21:45 UTC
Identified
Azure reports a network device has experienced a fault, resulting in network connectivity loss to downstream resources. The unhealthy network device is being isolated from the network and traffic is being rerouted to healthy infrastructure.
We will update this incident every 30 minutes or as warranted.
Posted Dec 14, 2022 - 21:04 UTC
Investigating
Azure has reported that starting at 19:11 UTC on 14 Dec 2022, Virtual Machines in West US 2 may be experiencing connectivity issues. These Virtual Machines may have also restarted unexpectedly since the start of this incident.
A network device has experienced a fault, resulting in network connectivity loss to downstream resources. The unhealthy network device is being isolated from the network and traffic is being rerouted to healthy infrastructure.
We will update this incident every 30 minutes or as warranted.
Posted Dec 14, 2022 - 20:28 UTC
This incident affected: Workspot Control.