We recently addressed issues affecting Box activity, including Box Metadata, file uploads and downloads, Box Notes, and the public API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 9:44 pm PST on December 4, 2022 and 12:32 am PST on December 5, 2022 and between 4:00 pm PST and 5:53 pm PST on December 5, 2022, some users may have experienced difficulties while working in Box. During this time, some users may have experienced higher than normal latency and/or timeouts when trying to access content in Box via one of our applications or our Public API. The issue occurred due to a sudden surge in traffic to a localized area of our metadata cluster, resulting in increased request queue latencies and degradation of Downloads, Box Notes, Metadata and other related services. The issue was resolved after the traffic to the servers hosting the data was decreased. We have since then made observability improvements to identify similar traffic surges faster and are working on process and tooling improvements to prevent similar issues from occurring in the future.

‌

Analysis

This issue occurred due to a sudden surge in traffic to a localized area of our metadata cluster. Although Box is designed to be capable of handling short surges in traffic, this issue was caused by a sustained increase in traffic directed to a single server. As a result, requests to this server developed a backlog, which affected latency of Box Metadata generally; this resulted in users experiencing degradation of uploads, downloads, Box Notes, and other related services. The issue was resolved after the traffic to server hosting the data was decreased. We are working on improvements to how data is sharded and cached and also process and tooling improvements to prevent similar issues from occurring in the future.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Added observability improvements to identify similar traffic surges more rapidly
• Added additional caching to proactively limit localized surges in traffic to individual servers
• Added the ability to quickly react and throttle surges in traffic to individual servers
• Improving resilience of Box services to high latencies when accessing data stores to minimize the impact of any similar issues in the future

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team