For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Major] Issue with Admin Events API
Incident Report for Box
Postmortem

We recently addressed issues affecting Enterprise Events API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 5:35am and 10:13am PDT on March 23, 2021, some users may have experienced difficulties while working in Box. During this time, a delay in the events received from the Admin Events API. The issue occurred as a result of a misconfigured server joining one of our database clusters that supports our Admin Events API. We were able to resolve the issue by removing the misconfigured server from our database cluster. In addition, we are working to improve our automation and monitoring to prevent similar issues from occurring in the future. 

Analysis 

Box's Admin Events API retrieves data from an internal database cluster that is updated as events occur within the Box application.  As part of this issue, we identified a condition in which servers could join the database cluster with a misconfiguration that prevents them from operating at the level of performance they should be configured for.  After we identified the problem condition, our engineers were able to remove the misconfigured server and deploy a code change to prevent misconfiguration from recurring.

Corrective Actions

The following corrective actions have been completed or are planned:

  • Update configuration logic to ensure the proper configuration is in place before a server joins our Admin Events database cluster.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

 

Sincerely,

The Box Team

Posted Apr 21, 2021 - 18:05 PDT

Resolved
The Admin Events API metrics are now recovered and no additional impact has been observed. We are considering the issue to be fully resolved. If you are encountering any issues, please contact Box Support at https://support.box.com.
Posted Mar 23, 2021 - 17:23 PDT
Update
No additional impact has been observed at this time. We are continuing to monitor metrics to ensure the issue is fully resolved.
Posted Mar 23, 2021 - 13:10 PDT
Update
The Admin Events API metrics are now recovered and no additional impact has been observed. We are considering the issue to be fully resolved. If you are encountering any issues, please contact Box Support at https://support.box.com.
Posted Mar 23, 2021 - 07:30 PDT
Update
The Admin Events API metrics are now recovered and no additional impact has been observed. We are considering the issue to be fully resolved. If you are encountering any issues, please contact Box Support at https://support.box.com.
Posted Mar 23, 2021 - 07:15 PDT
Monitoring
Our team has completed steps to remediate this issue and is observing steady improvement in Admin Events API metrics. We will continue our observation until the issue is fully remediated and then begin our monitoring window for any additional impact.
Posted Mar 23, 2021 - 07:08 PDT
Identified
We have identified an issue affecting the Admin Events API endpoint and are actively taking steps to remediate at this time. Some users may experience delayed event notifications. We will provide more information as soon as it is available.
Posted Mar 23, 2021 - 06:52 PDT
This incident affected: Box Platform / API (Content API).