We fixed the main performance issue and now the system is stable. The custom fields feature has been enabled again. We're still going to monitor our system to detect possible regressions.
Posted May 18, 2020 - 14:14 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted May 18, 2020 - 12:33 UTC
Update
We found the culprit of this issue. Yesterday we deployed a change with our custom fields system with a non performant endpoint, this degraded our puma's trying to serve this requests for about 60 seconds. This was causing other requests to be delayed and eventually timedout.
We partially disabled custom fields feature in order to keep other parts of the app working.
We're fixing the performance regression and we'll enable full custom fields feature once we get a decent performance on affected endpoint.
We'll keep updating this incidence with further steps.