Autodesk Build | Autodesk Docs | BIM 360 Docs | BIM 360 Field Management | BIM 360 Insight | BIM 360 Design Collaboration | BIM 360 Model Coordination | Autodesk BIM Collaborate | Autodesk Takeoff | ACC AutoSpecs | ACC Admin Console
Incident Report for Autodesk Health Dashboard
Postmortem

Summary

On May 9, 2023, between 6:03 AM PDT and 8:46 AM PDT, Autodesk experienced a service disruption that impacted North American Data Center customers using BIM 360 HQ, Autodesk Build, Autodesk Docs, BIM 360 Docs, BIM 360 Field Management, BIM 360 Insight, BIM 360 Design Collaboration, BIM 360 Model Coordination, Autodesk BIM Collaborate, Autodesk Takeoff, ACC AutoSpecs, and ACC Admin Console.

Impacted Services

  • BIM 360 and Autodesk Construction Cloud (ACC) customers experienced degraded
    performance and the inability to view features within either product.
  • Some BIM 360 and Autodesk Construction Cloud (ACC) customers were unable to login.

Root Cause

  • A recent code change introduced a caching update that inadvertently caused the
    clearing of cache for a wider range of transactions than initially required, which resulted
    in a decrease in service performance. However, upon the routine collection of our app
    usage data, which typically causes spikes in memory and CPU usage, the combined
    volume of traffic was greater than our caching engine could handle. This resulted in
    degraded service and performance for BIM 360 and Autodesk Construction Cloud (ACC)
    customers.

Autodesk Actions

Autodesk has completed a post-incident analysis of the event and identified actions to be
taken. These include the following:

  • We have rolled back the recent code change which contained the code related to this
    performance regression.
  • A new code change will be rolled out to only target the transactions requiring cache
    clearing updates.
  • We are working to implement improvements to the monitoring and alerting for high
    memory and CPU usage to proactively identify and mitigate performance issues in the
    future.
  • We are exploring new ways to collect routine app usage data without creating spikes in
    memory and CPU usage that could negatively affect service and performance for
    customers.

Thank you for your patience and understanding.

Posted May 16, 2023 - 07:49 PDT

Resolved
Current status: Operational. Service is now working as expected and all features are operating normally.
Posted May 09, 2023 - 08:45 PDT
Monitoring
We are working to resolve an issue where customers may be unable to access services in our North American Data Center, services are now in recovery. Next update in 30 minutes.
Posted May 09, 2023 - 08:32 PDT
Update
We are working to resolve an issue where customers may be unable to access services in our North American Data Center.
Next update in 30 minutes.
Posted May 09, 2023 - 08:20 PDT
Update
We are working to resolve an issue where customers may be unable to access services in our North American Data Center.
Next update in 30 minutes.
Posted May 09, 2023 - 08:18 PDT
Update
We are working to resolve an issue where customers may be unable to access services in our North American Data Center.
Next update in 30 minutes.
Posted May 09, 2023 - 08:15 PDT
Identified
Current status: Service Degradation. We are investigating issues where some customers may be unable to access Autodesk services listed. We'll provide additional information here as it becomes available.
Posted May 09, 2023 - 07:45 PDT
Investigating
Current status: Service Degradation. We are investigating issues where some customers may be unable to access Autodesk services listed. We'll provide additional information here as it becomes available.
Posted May 09, 2023 - 07:25 PDT
This incident affected: ACC Admin Console, ACC AutoSpecs, Autodesk BIM Collaborate, Autodesk Build, Autodesk Docs, Autodesk Takeoff, BIM 360 Design Collaboration, BIM 360 Docs, BIM 360 Field Management, BIM 360 Insight, and BIM 360 Model Coordination.