For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Critical] Issue with Box Embed Uploads
Incident Report for Box
Postmortem

***UPDATED: Oct 18, 2021 @ 12:41 PM PT

We recently addressed issues affecting uploads via Box Embed Widget and a subset of accelerated uploads. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 1:00PM PDT on September 23, 2021 and 10:23PM PDT on September 27, 2021, some users may have experienced difficulties while working in Box. During this time, customers uploading via Box Embed Widget would see an error notification and multiple uploads of the same file. For customers uploading via accelerated nodes with the IP Allowlist feature enabled, users experienced errors in the upload. The issue occurred as a result of a recent code change in our upload service intended to clean up some deprecated pathways. We were able to resolve the issue by reverting the change and restoring to a previous known version. In addition, we are adding several new tests and alerts as part of the deployment process to prevent similar issues from occurring in the future.

Analysis 

As part of regular updates, an upload service deployment was done in production. This change incorrectly overwrote a header that is used by downstream services to determine the original context of the upload. As such, in the Embed Widget case, the downstream service failed to recognize that CORS headers need to be set in the response. This caused errors only on the client side, which were not visible to the service. Corrective actions are being taken to ensure both the client and server side is aware of this change going forward in a timely manner.

In the IP Allowlisting and accelerated nodes case, the upload was not recognized to be from the original source through accelerated node and thus failed authorization check.

We were able to resolve the issue by reverting the change and restoring to a previous known version. After the revert of the deployment, the issue was mitigated.

Corrective Actions

The following corrective actions have been completed or are planned:

  • Adding additional testing and alerting during the development for html5 uploads

  • Adding additional testing and alerting during the development for IP Allowlist and accelerator combination

  • Adding a sanity environment before full deployment of upload-proxy service

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

 Sincerely,

The Box Team

***********************************************************************************************************************

Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we will update as more information is confirmed.

Between 1:00PM PDT on September 23, 2021 and 10:23PM PDT on September 27, 2021, some users may have experienced difficulties while working in Box. During this time, some customers experienced issues uploading files to Box via the Box Embed Widget and for a subset of accelerated uploads. The issue occurred as a result of a recent code change in our upload service to clean up some deprecated pathways. We were able to resolve the issue by reverting the change and restoring to a previous known version. In addition, we are adding several new tests and alerts as part of the deployment process to prevent similar issues from occurring in the future.

We are conducting a full engineering postmortem and our overview is subject to change with further analysis and findings. We will publish the results as soon as we have concluded our investigation.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team

Posted Oct 12, 2021 - 09:31 PDT

Resolved
The Box Embed upload error has been fixed and no additional impact has been observed. We are considering the issue to be fully resolved. If you are encountering any issues, please contact Box Support at https://support.box.com.
Posted Sep 27, 2021 - 22:37 PDT
Monitoring
We are seeing a recovery of uploads using Box Embed. We will continue to monitor the situation and provide an update for resolution soon.
Posted Sep 27, 2021 - 20:32 PDT
Update
We have pushed a fix for the issue to all users. Some users are expected to see recovery with uploads in the Box Embed. This change will take full effect later tonight once it completes the process. Update later tonight when the change takes full effect.
Posted Sep 27, 2021 - 16:22 PDT
Identified
The issue has been identified. We are now testing a fix to push for all users. Next update to status site, in an hour or when change has been implemented to all users.
Posted Sep 27, 2021 - 13:46 PDT
Investigating
We are currently investigating an issue with Box Embed widget starting Sept 24th 6 AM PDT, uploads to Box via the embed will result in an error: "Something went wrong with the upload. Please Try Again." This will affect any Box Embed widget including customers using Netsuite and Salesforce integrations. Refreshing the page containing the asset will show that the files did upload successfully and that the error appears to be a false positive.
Posted Sep 27, 2021 - 12:53 PDT
This incident affected: Partners / Integrations (Box for Salesforce, NetSuite) and Box Platform / API (Box Embed Widget).