Slow Performance within LMS365
Incident Report for ZENSAI
Postmortem

We have identified the root cause of the issue with degraded performance of the LMS365. It was related to the Incident on the Microsoft Azure Front Door.

More information you can find below.

Incident: MO427682, impacted service: Microsoft 365 suite, impacted feature: Portal, current status is: Service restored
——-

Published Time: 9/7/2022 10:36:35 PM

Title: Some users may be unable to leverage services or features that utilize Azure Front Door

User Impact: Users may be unable to leverage services or features that utilize Azure Front Door.

More info: Examples included, but weren't limited to:

  • Microsoft Intune users may have been unable to set up remote help sessions
  • Microsoft Endpoint Manager users may have been unable to access said service.
  • Anonymous users joining meetings through the browser meeting join experience may have intermittently failed. Repeated attempts to join meetings may have intermittently succeeded.

Final status: We’ve completed the operation to optimize traffic flow and restore the affected Azure Front Door APIs, and we’ve monitored telemetry to confirm that service is restored and impact is remediated.

Scope of impact: This issue could have potentially affected any user if they were leveraging a service or feature that goes through Azure Front Door.

Start time: Wednesday, September 7, 2022, at 4:10 PM UTC

End time: Wednesday, September 7, 2022, at 7:55 PM UTC

Root cause: Portions of infrastructure responsible for connections to Azure Front Door APIs were operating in a suboptimal state, resulting in impact.

Next steps:

  • We’re continuing our investigation into the infrastructure responsible for connections to Azure Front Door APIs to ascertain methods to reinforce the environment against similar future impact.

This is the final update for this event.

Published Time: 9/7/2022 9:32:32 PM

Title: Some users may be unable to leverage services or features that utilize Azure Front Door

User Impact: Users may be unable to leverage services or features that utilize Azure Front Door.

More info: Examples include, but aren't limited to:

  • Microsoft Intune users may be unable to set up remote help sessions
  • Microsoft Endpoint Manager users may not be able to access said service.
  • Anonymous users joining meetings through the browser meeting join experience may intermittently fail. Repeated attempts to join meetings may intermittently succeed.

Current status: While the service environment is showing partial recovery, we’ve received reports from affected customers that impact is ongoing. We’re performing additional optimizations on the affected infrastructure to restore the affected Azure Front Door APIs and remediate the remaining impact.

Scope of impact: This issue could potentially affect any user if they're leveraging a service or feature that goes through Azure Front Door.

Root cause: Portions of infrastructure responsible for connections to Azure Front Door APIs are operating in a suboptimal state.

Next update by: Wednesday, September 7, 2022, at 9:30 PM UTC

Published Time: 9/7/2022 8:14:30 PM

Title: Some users may be unable to leverage services or features that utilize Azure Front Door

User Impact: Users may be unable to leverage services or features that utilize Azure Front Door.

More info: Examples include, but aren't limited to:

  • Microsoft Intune users may be unable to set up remote help sessions
  • Microsoft Endpoint Manager users may not be able to access said service.
  • Anonymous users joining meetings through the browser meeting join experience may intermittently fail. Repeated attempts to join meetings may intermittently succeed.

Current status: Our monitoring alert notified us of an issue where certain Microsoft 365 services and features that leverage Azure Front Door aren't working as intended. Our investigation indicates that portions of infrastructure responsible for connections to Azure Front Door APIs are operating in a suboptimal state. We've implemented optimizations on the affected infrastructure and redirected traffic to remediate impact. We’re monitoring service health to validate recovery.

Scope of impact: This issue could potentially affect any user if they're leveraging a service or feature that goes through Azure Front Door.

Root cause: Portions of infrastructure responsible for connections to Azure Front Door APIs are operating in a suboptimal state.

Posted Sep 08, 2022 - 12:10 CEST

Resolved
We can confirm that the issue with "Slow Performance within LMS365" is now resolved. If you notice any issue going forward, please reach out to us.
Posted Sep 08, 2022 - 11:11 CEST
Monitoring
We are currently aware of degraded performance that multiple users are experiencing across the LMS365 platform. We are monitoring this issue with the intentions of resolving it as soon as possible. If you notice any other areas are being affected, do not hesitate to reach out and inform us. Thank you for your patience as we strive to deliver the best experience we can!
Posted Sep 07, 2022 - 21:54 CEST
This incident affected: LMS365 North Europe, LMS365 United States, LMS365 Canada, and LMS365 Germany.