The R Servers in South East Asia region have now automatically recovered thanks to Microsoft Azure repairing the data center. This work is still on-going however Displayr's Azure resources in the data center have recovered and there is little risk that they will revert.
In terms of what Displayr could do to avoid this type of incident in future: our typical disaster recovery option would to have been to move the R Servers to an alternative region or zone in Azure that is within the same country (Singapore). Unfortunately there are no other Azure data centers available that could be used as a disaster recovery option, and all zones within the data center were affected.
An incident affecting an entire data center is highly unusual. This is the first time this incident has occurred since Displayr launched the South East Asia region in February 22, 2022. We have determined the risk of a similar incident occurring in future as very low.
Posted Feb 08, 2023 - 21:46 UTC
Update
In regards to the South East Asia issue with R calculations, we are waiting for Azure to resolve the issue in the data center. All attempts to create separate R Servers in the same region have failed due to issues that impact the entire region. Please monitor the event "Datacenter Cooling Event - Southeast Asia" at https://azure.status.microsoft/en-gb/status
A workaround is available for customers that are affected in South East Asia and are not concerned with data sovereignty or data locality. The following steps will cause your data to leave South East Asia and be sent to the Australia East region. Only follow these steps if you accept the consequences of the data leaving your region.
1. Edit your Displayr document. 2. Wait for your document to load. 3. In the address bar of your browser, copy and paste the following line and press Enter
4. The R Server for your document will have been temporarily changed to the Australia East region R Server. Any R calculations will now work. 5. You may have to repeat these steps after 10 minutes of inactivity.
Posted Feb 08, 2023 - 11:41 UTC
Identified
The issue is occurring due to an incident in the Azure South East Asia Data Center. Azure is still working on restoring services and they currently do not have an ETA. Here is a copy of their message:
> We do not have an exact ETA at this time, we currently expect an extended period of time to fully restore all cooling capacity. The Azure service recovery process will commence at this time and is expected to progressively return over a number of hours
In the meantime, we are experimenting with a possible workaround.
Posted Feb 08, 2023 - 09:42 UTC
Investigating
R calculations in the South East Asia region are taking a long time and then returning an error such as "There is a problem with your Internet connection that prevented the calculation from completing."
We are investigating the cause of the errors.
Other regions are not affected.
Posted Feb 08, 2023 - 08:57 UTC
This incident affected: Q (R Servers) and Displayr (R Servers).