We would like to provide additional detail surrounding the downtime which occurred on 09/06/2022
What happened?
At 08:02am PST, the US2 server environment experienced downtime which resulted in QLess applications becoming temporarily inaccessible on the server.
Duration: 7 minutes.
Cause
The root cause of the outage was due to a race condition that was triggered on one of our backend servers.
This race condition is a known issue, and can be triggered by heavy transactional load or other rare circumstances.
Remediation
Upon receiving a monitoring alert notification QLess engineers restarted the service.
Prevention
QLess has initiated a major review of the associated database which we believe is liable for triggering the race conditions. The work is fragmented into two stages. Currently the team is working on the details of the refactoring work. We will notify you on the progress of each refactoring stage.