Trading Engine Outage
Incident Report for BitMEX
Postmortem

Trading was suspended and the platform was put into maintenance mode between 00:24 UTC and 00:55 UTC on 13 May 2021 as a result of an isolated position audit failure in the trading engine. Trading resumed at 01:00 UTC once all audits had been passed, and after 5 minutes of cancel-only mode. Funds were entirely safe at all times.

BitMEX has an automated audit process which checks, among many other things, that the value of all positions on the platform sum to zero. This feature is one of many we have in place to ensure the integrity of the platform and safeguard user funds. For this check to be valid, the same mark price must be used to value every position.

With a recent change to our system architecture, positions are calculated in a distributed fashion. If the audit detects a discrepancy, it automatically halts trading to safeguard user funds and maintain the security and integrity of the platform. Much care has been taken to ensure correct synchronisation between processes, however a synchronisation bug occured. We have now implemented a change to address this, while upholding the continued safety and security of funds on the platform.

The exchange unsuspended at 00:44 UTC upon passing a stage of our audit, and stops and liquidations were processed whilst the platform was still in maintenance mode.

At 05:01:29 UTC we updated the funding rates for all Perpetual Swaps to exclude the affected time period.

We are compensating all users who were liquidated during the downtime, and expect to have completed this within the next 2 hours.

We apologise for the inconvenience this has caused. Please contact Support should you have any further queries.

Posted May 13, 2021 - 09:44 UTC

Resolved
We can confirm trading resumed at 01:00 UTC. A full review of the incident will be posted soon. We know how much this impacts our customers, and we'd like to apologise to all those affected. If you have any further questions, please contact Support.
Posted May 13, 2021 - 01:19 UTC
Monitoring
We’re bringing the BitMEX platform back online, initially via cancel only mode until 01:00 UTC, where no new orders can be placed. All funds are safe.
Posted May 13, 2021 - 00:56 UTC
Investigating
The Trading Engine is currently offline. We're working to bring it back online as soon as possible, we apologise for any inconvenience this may be causing. All funds are safe.

We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact Support
Posted May 13, 2021 - 00:31 UTC
This incident affected: BitMEX.com (Trading Engine).