.Net 7 Rollout on Microsoft Azure

On Thursday, October 20 2022 Our partners at Microsoft started rolling out a pre-release version of the upcoming .NET version 7. This unfortunately resulted in a subset of customers not being able to deploy changes to Umbraco Cloud or deploy changes between environments.

The way that Umbraco Cloud works is tightly bound to the current SDK and runtime of .NET, and is susceptible to changes in the underlying framework.

Over the last couple of weeks, the roll-out has been scaled across Azure Data Centers, and more customers have gradually become affected.

We apologize for the inconvenience that this has caused our customers, and assure you that we have taken steps to address the specific issue and are working on initiatives to ensure that this type of issue will not happen again.

Root Cause Analysis

In preparation for the upcoming release of .NET 7 multiple preview versions are getting deployed to Microsoft Azure (our hosting provider).

We did not anticipate that this would cause a change in the default behavior on how we build and deploy changes internally on our Umbraco Cloud Projects. Additionally, the preview versions that were being pushed to our infrastructure, contained a bug that made the Umbraco Website unable to build on the current hardware.

The issue started occurring on Thursday, October 20, 2022, at that point, it was only a small subset of customers were affected by the rollout. Nevertheless, we did reach out to our partners and Microsoft to gain an understanding of the issue at hand.

On Wednesday 2nd of November, we saw a sharp uptick in the number of tickets coming in, and we took immediate action. When provisioning a new Website, we explicitly set the current .NET runtime, however, we discovered that if a version of the .NET runtime was installed automatically via Microsoft Azure roll-out then it would not enforce our settings. This caused the underlying infrastructure to utilize the latest available version of .NET - in this case, the pre-release of .NET 7.

The default behavior for our infrastructure is that it is running on the latest version of .NET installed. To hotfix the issue we communicated that applying a global.json to the repository, containing the targeted SDK would get the build process up and running again. The default .NET behavior is documented here (https://learn.microsoft.com/en-us/dotnet/core/tools/global-json)

‌

On Friday the 5th we rolled out a hotfix to a core component in Umbraco Cloud that would allow us to patch all ASP.NET Core-based projects (Umbraco 9 and 10). The component is also installed on non- ASP.NET Core projects (Umbraco 7 + 8), and though the component was tested for Umbraco 9 and 10, the release introduced a regression for Umbraco 8 sites.

Our support department was quickly discovering the malfunctioning component and reached out to the Platform Team, which halted the rolling out of component upgrades, then proceeded to patch the component, and re-released the hotfix for the component.

Monday the 7th, we created and tested an internal application, to target and migrate the affected projects.

Tuesday the 8th, we patched all affected websites on Umbraco Cloud.

Actions based on the root cause analysis

When creating a new Umbraco Cloud Website, we are creating a new bundle that contains the default files for a new Umbraco Cloud Website. This includes file structures, project files, readme, and configuration files. We’ve amended the bundle to include the global.json file that includes the targeted .NET runtime, and all new Umbraco Cloud projects are created with the new global.json

Additionally, we are creating internal procedures and guardrails for testing the component that was released onto Cloud On Friday the 5th. This includes a series of automated tests that will cover the scenario where the component is running on different project types.