[North Central US] Email delivery delays

Incident Report for CodeTwo

Postmortem

What happened?

Between 17:55 UTC and 18:40 UTC on 05 Dec 2022, a subset of customers in North Central US may have experienced issues with emails not getting signatures or being slightly delayed. This did not impact email deliverability in any way.

What went wrong and how we responded

We observed a problem with Microsoft’s SQL database availability in North Central US, which affected automatic scaling. We immediately switched to secondary services in the region but that did not fully remediate the issue, so we manually rescaled the SQL database.

What are the next steps?

We are investigating the database availability problem with Microsoft to make sure we’re no longer affected by such incidents on their side. We’ll also deploy custom scaling policies for the SQL database to replace standard mechanisms and ensure similar issues don’t happen in the future.

Posted Dec 08, 2022 - 11:14 UTC

Resolved

This incident is now resolved. All emails have been delivered but a small percentage might have been bereft of signatures due to database availability issues (if this concerns your tenant, we will notify you via email with more details within 48 hours).

We will also provide a public RCA within 48 hours after we fully analyze the problem. The RCA will be provided as an update of this incident here.

Please accept our apologies for all inconveniences.

Posted Dec 05, 2022 - 18:30 UTC

Monitoring

A fix has been implemented and we are seeing signs of recovery. The problem should be mitigated soon. We are monitoring the results.

Posted Dec 05, 2022 - 17:58 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Dec 05, 2022 - 17:46 UTC

Investigating

We are currently investigating a problem with Azure CosmosDB availability in North Central US. Some users may experience delays in email delivery. We are working on identifying the problem and fixing it ASAP.

Posted Dec 05, 2022 - 17:33 UTC

This incident affected: North Central US (Mail flow, Signature adding (rules processing)).