Unable to create new tenants.
Incident Report for uniFLOW Online
Postmortem

User Impact

The creation of a new tenant would fail with an internal (undefined) error. Additionally, it was reported late in the incident investigation that changing a user to a privileged role which required MFA to be enabled also failed.

Scope of Impact

This impacted the AU, SG, US, UK, EU deployments.

Incident Start Date and Time

1 Mar 2022, 05:30 UTC

Incident End Date and Time

1 Mar 2022, 07:30 UTC

Root Cause

During both Tenant Creation and Privilege User rights elevation uniFLOW Online requires MFA to be enabled. In the case of the tenant creation, it is the creation of the Root Tenant Admin account.

The user permissions for this action are managed by Microsoft’s Roles Based Access. It was identified that the role in use could no longer set the MFA options due to an unknown Microsoft change. The role used had the required permission and capability since MFA was introduced last year with uniFLOW Online 2021.2.

Actions Performed

Once identified our team was able to rectify the configuration to use a new permission role with the required permission. This was tested and reviewed in accordance with our ‘Least Privilege Model’.

Next Steps

We apologize for the impact to affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. The use case will be reviewed with the Quality Assurance and Operations teams. Testing scenarios will be investigated which will highlight such changes or failures with the Microsoft MFA process in the future.

Posted Mar 06, 2022 - 05:21 UTC

Resolved
Hello Everyone,
We are pleased to say the situation has been resolved and Tenant creation is now working again.

The team will now look into the cause and review the information collected during this incident. From this we will build a Post Mortem that should be published within 5 business days of this event.

Thanks for your patience,
uniFLOW Online Operations Team.
Posted Mar 01, 2022 - 07:44 UTC
Monitoring
Hello Everyone,

Sorry for the delay, we are working on this issues with high priority. We have identified the problem and busy deploying a mitigation right now. This status of this ticket will be moved to 'Monitoring' so we can check and validate the changes.

uniFLOW Online Operations.
Posted Mar 01, 2022 - 07:19 UTC
Investigating
Incident details:
Identified: 1 Mar 2022, 05:30 UTC

Description:
It is currently not possible to create a new tenant reported to us and confirmed via our logging. An error is shown during the tenant creation process. There is no impact to existing customer tenants or any uniFLOW Online functionality.

Incident Scope:
Confirmed impacted deployments are, AU, EU, SG, US & UK.

Next Update:
30 minuets

Kind regards,
The uniFLOW Online team
Posted Mar 01, 2022 - 06:20 UTC
This incident affected: EU Deployment (Other services), US Deployment (Other services), SG Deployment (Other services), AU Deployment (Other services), and UK Deployment (Other services).