We recently addressed issues affecting content ingress and egress. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 11:15 pm PT on April 25, 2023 and 8:40 am PT on April 26, 2023, some users may have experienced difficulties while working in Box. During this time, one of our cloud partners experienced an infrastructure malfunction that impacted one of their European regions. This impact caused the region to exhibit elevated error rates and increased request latency which caused upload and download requests in Box to backup. We were able to resolve this issue by updating storage policies in this European region to use a different but still compliant region. In addition we have taken on a number of near term and longer term remediations to to prevent similar issues from occurring in the future. 

Analysis
Starting at approximately 7:00 pm PT on April 25, 2023 one of our European partners experienced an infrastructure problem in one availability zone of their France region. At 11:15 pm PT this malfunction triggered a fire that motivated an evacuation of datacenter personnel and resulted in region wide degradation. At this time requests that utilized storage in the impacted region saw increased latency and elevated error rates. This resulted in elevated request rates due to retries and longer request times. We remediated the issue by updating the storage policies to reroute the requests to another region that was still compliant with residency requirements, and by updating the behavior of internal retry and rate limiting logic.

Corrective Actions
The following corrective actions have been completed or are planned:

• Update internal rate limiting logic to trigger automatic scale-out sooner
• Update monitoring for all storage locations
• Audit request timeouts
• Extend automated health checking and automated policy re-routing to cover additional degradation cases

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team