For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Critical] Issues with Accessing Content, Logins, Public API, Uploads, and Downloads
Incident Report for Box
Postmortem

Last updated 21 April 2021 18:11 PDT

****

We recently addressed issues affecting view, upload, or download files. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 2:01 AM PDT and 4:22 AM PDT on April 1, 2021, some users may have experienced difficulties while working in Box. During this time, impacted users may have experienced latency while trying to view, upload, or download files. The issue occurred as a result of failure to effectively handle a crash of a caching host. We were able to resolve the issue by removing the impacted hosts from rotation.

In addition, we are making changes to the way we check the health and remediate the problems with caching hosts to prevent similar issues from occurring in the future.

Analysis

The incident started when one of our caching servers failed due to a hardware failure.  Auto-remediation did not happen in a timely manner as expected, which unfortunately triggered a latent bug in our caching proxy servers causing memory exhaustion.  This required the proxy servers to be manually restarted to remediate.

Corrective Actions

The following corrective actions have been completed or are planned:

  • Change the way degraded hosts are detected
  • Prevent caching services from failure when there is a problem with the cache hosts
  • Bring back failed caching services automatically if they are degraded or failed

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team

Posted Apr 01, 2021 - 20:23 PDT

Resolved
We have taken steps to resolve this issue and are considering the issue to be fully resolved. If you are seeing any new issues, please let us know at https://support.box.com.
Posted Apr 01, 2021 - 07:02 PDT
Update
We have observed a full recovery and we will continue to monitor this issue.
Posted Apr 01, 2021 - 05:43 PDT
Update
We continue to see higher rates of recovery and stabilization across all Box services. We will continue to monitor the recovery process.
Posted Apr 01, 2021 - 05:07 PDT
Monitoring
We are seeing initial signs of recovery. Users should notice less of an impact at this time, as we continue to monitor and remediate this issue.
Posted Apr 01, 2021 - 04:52 PDT
Identified
We are currently remediating an ongoing issue affecting latency in Box.com, Box Drive, Box Notes, Box Sync and Box API. Logging in and accessing content may also be impacted. We will provide more information as soon as it is available.
Posted Apr 01, 2021 - 04:07 PDT
Update
We are currently investigating an ongoing issue affecting latency in Box.com, Box Drive, Box Notes, Box Sync and Box API. Logging in and accessing content may also be impacted. We will provide more information as soon as it is available.
Posted Apr 01, 2021 - 03:05 PDT
Update
We are currently investigating an ongoing issue affecting the Box API including uploads and downloads. Logging in and accessing content may also be impacted. We will provide more information as soon as it is available.
Posted Apr 01, 2021 - 02:34 PDT
Investigating
We are currently investigating this issue.
Posted Apr 01, 2021 - 02:33 PDT
This incident affected: Box Platform / API (Content API, Uploads/Downloads), Box Web Application (Login/SSO, Uploads/Downloads, Collaboration, Search), Mobile Applications (Login/SSO, Uploads/Downloads), Box Notes (Web Application), Desktop Applications (Box Sync, Box Drive, Box Edit / Tools), and Box Website.