User Impact

During the incident users would experience the uniFLOW SmartClient disconnecting from the cloud and forced into emergency mode. This would still allow printing but at reduced usability and functionality.

Scope of Impact

This issue was isolated to our AU (Australian) Deployment

Incident Start Date and Time

Jun 20, 2022, 1:30 AM UTC

Incident End Date and Time

Jul 07, 2022, 8:30 AM UTC (Mitigation controls in place and no further issues detected. The status page remained in a monitoring state until the 12th of July).

Root Cause

This issue originally presented as a resource issue with the Azure IoT Hub. However, the connections and consumed resources did not match expected usage patterns. We could validate this by comparing to several of our other larger Azure deployments.

Considering a possible issue in the IoT stack we redeployed the IoT Hub into a new Azure Data Centre. We quickly saw the issue reappear and raised a support case with Microsoft Support Engineers to look deeper into the IoT metrics.

It was found the SmartClient was suddenly making multiple connections to the IoT Hub. This multiplying effect meant that we saw a huge rise in connection in comparison to the number of SmartClients and the expected usage pattern.

During deeper analysis, the root cause was found to be due to the uniFLOW SmartClient being used in environments with a unique network configuration where more than one proxy may be used. When the second proxy connection is established a second connection is established which causes the first connection to disconnect. This, in turn, causes a retry to take place from the first connection after which the two connections continuously disconnect and reconnect resulting in the resource exhaustion.

IoT Hub resource allocation has been optimized based on this finding in order to ensure a stable uniFLOW Online environment. Our development team has improved the uniFLOW SmartClient to better handle proxied environments already providing a fix which is currently going through our Quality Control process. Once this fix has been tested and verified it will be included in the 2022.3 release of uniFLOW Online.

Next Steps

We apologize for the impact to affected customers. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):

• Make available the uniFLOW SmartClient so tenant admins can package and redeploy to all users. This is planned for 2022.3, release is scheduled to take place in September.
• We are further optimizing our product to reduce unnecessary resource utilization and network usage which will also be available with 2022.3.
• Additional metrics, monitoring and alerting has been implemented. This is key to managing the growth as the AU deployment grows and we are rolling out the updated uniFLOW SmartClient.

Customer Actions

To resolve these issues completely it is important that the uniFLOW SmartClient is updated post the 2022.3 release scheduled for September. We will communicate this via the uniFLOW Online Status Page and via the Admin Notification Widget along with instructions.