[CRITICAL] Issue with Box Services

We recently addressed issues affecting Box activity, including Box Metadata, file uploads and downloads, Box Notes, and the public API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

On July 7th, 2022 between 6:02 PM PDT and 7:10 PM PDT, some users may have experienced difficulties while working in Box. During this time, users may have experienced higher than normal latency, timeouts and/or errors when trying to access Box via one of our applications or our public API.

‌

Analysis

This issue occurred due to instability in one of our HBase clusters. To improve the capacity of the Box Metadata service, our engineering teams deployed an expanded data storage cluster on the afternoon of July 7th. This cluster was tested prior to its deployment, and performed as expected in production immediately after deployment. Approximately 3 hours after the deployment of this cluster, a single server within the cluster exhibited dramatically increased latency caused by a defective memory module. This resulted in users experiencing degradation of uploads, downloads, Box Notes, and other related services. Our automated server failure detection methods did not detect this server as failed, and as a result, traffic continued to be directed to this server. Requests to this server developed a backlog, which affected latency of Box Metadata generally, which resulted in timeout errors and slow performance in other Box services. We were able to resolve the issue by diverting traffic to a healthy passive cluster. In addition, we are working on process and tooling improvements to prevent similar issues from occurring in the future.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• The defective server was removed from service and repairs effected.
• Revised datastore cluster deployment, onboarding and validation processes to prevent this issue from recurring in the future and to improve detection of problematic deployments.
• Improving resilience of Box services to high latencies when accessing datastores to minimize the impact of any similar issues in the future.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team