For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

Issue with Admin Events API
Incident Report for Box
Postmortem

We recently addressed issues affecting timeliness of the Events API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

On Thursday, July 8, 2021 between 1:15PM PT and 4:05PM PT, some users may have experienced difficulties while working in Box. During this time, notifications may have slowed down for some users, and some Drive users may have experienced delays. This issue occurred as a result of an unexpected surge in traffic because of an internal migration. We were able to resolve the issue by adding more capacity to the affected service. In addition, we have added alerts to prevent similar issues from occurring in the future. 

Analysis 

A migration involving one of Box's backend services caused a sudden uptick in the incoming traffic to our messaging pipeline. Due to a preset protective limit, some of that incoming traffic was restricted, causing certain events to slow down.  The engineers investigating the issue were able to add additional capacity to the messaging service, and the issue resolved soon thereafter. 

Corrective Actions

The following corrective actions have been completed or are planned:

  • Adding additional capacity to the messaging pipeline

  • Adding new alerts to page the on-call engineers when traffic to the messaging pipeline is close to hitting the preset limit

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.  

Sincerely,

The Box Team

Posted Aug 18, 2021 - 00:54 PDT

Resolved
The Admin Events API metrics are now recovered and no additional impact has been observed. We are considering the issue to be fully resolved. If you are encountering any issues, please contact Box Support at https://support.box.com.
Posted Jul 08, 2021 - 17:26 PDT
Monitoring
Our team has completed steps to remediate this issue and is observing steady improvement in Admin Events API metrics. We will continue our observation until the issue is fully remediated and then begin our monitoring window for any additional impact.
Posted Jul 08, 2021 - 17:00 PDT
Update
We are continuing to investigate this issue.
Posted Jul 08, 2021 - 16:52 PDT
Update
We are continuing to investigate this issue.
Posted Jul 08, 2021 - 15:32 PDT
Investigating
We have identified an issue affecting the Admin Events API endpoint and are actively taking steps to remediate at this time. Some users may experience delayed event notifications. We will provide more information as soon as it is available.
Posted Jul 08, 2021 - 15:31 PDT
This incident affected: Box Platform / API (Content API).