[Europe] Signature management apps partially offline for a subset of users

Incident Report for CodeTwo

Postmortem

Here’s a root analysis provided by Microsoft:

The Microsoft Azure Team has investigated the issue reported and we have identified it was due to a dependency that the Azure Web App service takes on Azure Storage.

In this specific scenario, a storage cluster experienced a larger than normal network traffic volume. Engineers initially triaged this to be due to a possible increased traffic load. Thus, the first mitigation performed was to modify a configuration setting to allow for an additional volume increase. However, this change alleviated some of the pressure but did not solve the underlying problem. Additional troubleshooting determined that the majority of the remaining impact was due to a lock contention in the storage volume, primarily due to the cluster being overloaded from the initial traffic increase.

After mitigating the issue, engineers were able to solve the issue of the increased traffic volume by putting in place mechanisms that would distribute the volume for the specific scenario that resulted in this incident. Additionally, future improvements are planned to address the interaction between storage cluster dependencies and network volume fluctuations.

We apologize for any inconvenience.
Regards,
The Microsoft Azure Team

Posted Jan 14, 2022 - 07:08 UTC

Resolved

This incident is now resolved. The root cause will be provided later. The preliminary root cause provided by Microsoft is as follows:

Due to a sudden change in demand a single storage scale unit in the region reached an operational threshold, and automated throttling became active to ensure the health of existing resources on this scale unit. This throttling would have caused some requests to experience latency, or fail, and this would have had knock-on impact for other services with dependencies on this cluster.

All signatures were imprinted normally during the incident.

Posted Dec 17, 2021 - 12:03 UTC

Investigating

We are currently investigating a problem of degraded availability of signature management apps. Some users might be unable to edit or create signature rules. All other services are operational. Signatures are imprinted as normal. We're waiting for Microsoft to provide more info about the problem.

Posted Dec 17, 2021 - 10:09 UTC

This incident affected: North Europe (Signature management (app.codetwo.com)), West Europe (Signature management (app.codetwo.com)), UK South (Signature management (app.codetwo.com)), and Germany West Central (Signature management (app.codetwo.com)).