Summary
On May 9, 2023, between 6:03 AM PDT and 8:46 AM PDT, Autodesk experienced a service disruption that impacted North American Data Center customers using BIM 360 HQ, Autodesk Build, Autodesk Docs, BIM 360 Docs, BIM 360 Field Management, BIM 360 Insight, BIM 360 Design Collaboration, BIM 360 Model Coordination, Autodesk BIM Collaborate, Autodesk Takeoff, ACC AutoSpecs, and ACC Admin Console.
Impacted Services
- BIM 360 and Autodesk Construction Cloud (ACC) customers experienced degraded
performance and the inability to view features within either product.
- Some BIM 360 and Autodesk Construction Cloud (ACC) customers were unable to login.
Root Cause
- A recent code change introduced a caching update that inadvertently caused the
clearing of cache for a wider range of transactions than initially required, which resulted
in a decrease in service performance. However, upon the routine collection of our app
usage data, which typically causes spikes in memory and CPU usage, the combined
volume of traffic was greater than our caching engine could handle. This resulted in
degraded service and performance for BIM 360 and Autodesk Construction Cloud (ACC)
customers.
Autodesk Actions
Autodesk has completed a post-incident analysis of the event and identified actions to be
taken. These include the following:
- We have rolled back the recent code change which contained the code related to this
performance regression.
- A new code change will be rolled out to only target the transactions requiring cache
clearing updates.
- We are working to implement improvements to the monitoring and alerting for high
memory and CPU usage to proactively identify and mitigate performance issues in the
future.
- We are exploring new ways to collect routine app usage data without creating spikes in
memory and CPU usage that could negatively affect service and performance for
customers.
Thank you for your patience and understanding.