Datto Cloud Communications Failing resulting in New Restores Not being Mounted/New DCMA Devices May fail to Register/Cloud Continuity Status Page 500 error

Incident Report for Datto

Postmortem

On Jan 23, 2023, at 3 A.M. UTC, Multiple Datto Products experienced a service interruption which caused new DCMA Devices not to register, Cloud VMs not to Virtualize, and the CC4PC Status Page not to properly load.

The root cause for this service interruption was identified to be an upgrade that was performed to our AWS Cluster that caused a conflict with other internal services that rely on that cluster.

Our Engineering team deployed a fix to correct the problem on Jan 24, 2023, at 21:38PM UTC.

Multiple corrective action items are taking place to ensure an outage like this can be avoided again in the future which are but are not limited to:

Ensuring our AWS Clusters and corresponding systems are fully up to date and all systems that rely on them are compatible with these updates.
Improving our internal monitoring and alerting services to ensure we are aware of problems like this well in advance and can take corrective actions more rapidly.

Posted Feb 10, 2023 - 15:44 UTC

Resolved

This incident has been resolved.

Posted Jan 25, 2023 - 20:46 UTC

Monitoring

Our Engineering team has identified and released a fix that has restored functionality to the impacted systems. The teams are currently validating that functionality is restored to each impacted service and will continue monitoring the success of the fix.

Posted Jan 24, 2023 - 22:06 UTC

Update

We are continuing to investigate this issue.

Posted Jan 24, 2023 - 19:52 UTC

Update

Our Team has identified additional impacted features such as some DWA and DLA Agents failing to renew expired certificates which will result in a failed backup.

Our Support team has been supplied with a workaround to manually renew the expired DWA Agent certificates.

Posted Jan 24, 2023 - 16:01 UTC

Update

We are currently aware of a problem where New Restores cannot be mounted in the Datto Cloud + New DCMA Devices May fail to register

Our Engineering team has been engaged and is actively working to resolve the problem.

You can monitor the current status of this issue at https://status.datto.com/

Posted Jan 24, 2023 - 14:19 UTC

Investigating

We are currently investigating an issue with the datto offsite servers preventing new restores to be mounted

Posted Jan 24, 2023 - 12:42 UTC

This incident affected: Cloud Continuity (Agent Registration, Recovery), Datto Continuity for Microsoft Azure (Device Registration), and Datto BCDR (Backup, Recovery).