On Wednesday the 12th of May, we applied an infrastructure change to provision resources for a new service. This change unintentionally caused Document Check processing to fail for all checks created in EU, US and CA regions. As a result of this between 14:22 UTC and 15:16 UTC all Flash checks were being completed as Failed and all Boost checks were sent to Manual processing.
The aforementioned change in our infrastructure removed database access for a core component responsible for processing Document Checks. The database access was protected under a feature flag, but later we discovered the flag was removed and this database was being queried for all checks created at Onfido.
The incoming traffic in our pre-production environment was not enough to raise alerts that would allow us to detect the issue. This particular issue would only be visible in pre-production environment through our end-to-end tests, which are not automatically triggered for infrastructure changes.
Immediately:
In addition, we will: