We were able to stabilize uploads, and we'll keep this as an area of focus. Resolving.
Posted Aug 20, 2020 - 21:02 PDT
Update
We're starting to see some progress. Workers keeping up with incoming work. Still monitoring
Posted Aug 20, 2020 - 13:22 PDT
Monitoring
We've sped up one hot path by about 75%, and provisioned larger workers. Still seeing delays.
Posted Aug 20, 2020 - 12:53 PDT
Identified
We've found code that has terrible performance for packages with many versions. Multiple background jobs containing this code consume all the worker slots denying other jobs from being processed. (Edit: Jobs that bring your uploads to your dashboard and indexes.)
Posted Aug 20, 2020 - 11:10 PDT
Update
Issue is quite difficult to reproduce. We have implemented additional instrumentation to capture the issue when it happens in production. We will keep this issue open over the next period of high traffic.
Posted Aug 20, 2020 - 00:18 PDT
Investigating
We are investigating upload delays due to locked up background job workers.