We also maintain a list of Known Product Issues separate from this site here.
We recently addressed issues affecting the Admin Events API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
Between 4:05pm on February 23, 2021 and 1:43pm on Februrary 24, 2021, some users may have experienced difficulties while working in Box. During this time, customers may have experienced a delay in the events received from the Admin Events API. The issue occurred as a result of traffic unexpectedly colocating on a single node in one of our database clusters following routine server patching to improve performance and stability. We were able to resolve the issue by moving Admin Events API traffic to our redundant database cluster. In addition, we are working to improve our monitoring for this traffic colocation condition to prevent similar issues from occurring in the future.
Analysis
Box's Admin Events API retrieves data from an internal database cluster that is populated by one of Box's data pipelines. As part of this issue, we identified that a condition can be triggered that causes all traffic to our internal database cluster to become bottlenecked on one server which results in reads to the database cluster not showing the latest events as they are queueing up in Box's data pipeline. After we identified the condition causing the problem, our engineers were able to redistribute traffic to this database cluster and allow the backlog of queued up events to be processed successfully.
Corrective Actions
The following corrective actions have been completed or are planned:
Update monitoring and alerts to detect traffic colocation conditions on the Admin Events API database clusters
Audit and update configuration to prevent traffic colocation conditions on the Admin Events API database clusters
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team