Overview:

At around 5:41PM PDT, a change to the WebSocket server code was released which caused connections to abruptly close. The release was immediately rolled back. Operations returned to normal for most customers no later than 6:02PM PDT.

What Happened:

A change to WebSocket processing caused WebSocket connections to abruptly close. This triggered a thundering herd like effect where clients attempt to reopen a new web socket connection and continue to place additional request load on the application servers driving up throughput. This additional pressure degraded performance for all clients.

What we have implemented and will do in the future:

The root cause has been addressed and we are also exploring back-off mechanisms to prevent the thundering herd like effect.

As always, if you require further detail, please do not hesitate to email us at support@onfleet.com.