For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Major] Issue with Admin Events API
Incident Report for Box
Postmortem

We recently addressed issues affecting the Admin Events API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 4:05pm on February 23, 2021 and 1:43pm on Februrary 24, 2021, some users may have experienced difficulties while working in Box. During this time, customers may have experienced a delay in the events received from the Admin Events API. The issue occurred as a result of traffic unexpectedly colocating on a single node in one of our database clusters following routine server patching to improve performance and stability. We were able to resolve the issue by moving Admin Events API traffic to our redundant database cluster. In addition, we are working to improve our monitoring for this traffic colocation condition to prevent similar issues from occurring in the future. 

Analysis 
Box's Admin Events API retrieves data from an internal database cluster that is populated by one of Box's data pipelines.  As part of this issue, we identified that a condition can be triggered that causes all traffic to our internal database cluster to become bottlenecked on one server which results in reads to the database cluster not showing the latest events as they are queueing up in Box's data pipeline.  After we identified the condition causing the problem, our engineers were able to redistribute traffic to this database cluster and allow the backlog of queued up events to be processed successfully.

Corrective Actions
The following corrective actions have been completed or are planned:

  • Update monitoring and alerts to detect traffic colocation conditions on the Admin Events API database clusters

  • Audit and update configuration to prevent traffic colocation conditions on the Admin Events API database clusters

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team

Posted Mar 23, 2021 - 13:10 PDT

Resolved
We have taken steps to resolve this issue and are considering the issue to be fully resolved. If you are seeing any new issues, please let us know at https://support.box.com.
Posted Feb 24, 2021 - 14:30 PST
Monitoring
We have taken steps to to resolve this issue and are monitoring for any further impact. If you continue to see issues, please contact Box Support at https://support.box.com.
Posted Feb 24, 2021 - 13:52 PST
Investigating
We are currently investigating an issue affecting Admin Events API. Some users may experience delayed event notifications. We will provide more information as soon as it is available.
Posted Feb 24, 2021 - 13:17 PST
This incident affected: Box Platform / API (Content API).