Error rates are back to normal (< 0.3%) and the system is fully recovered. Login, receiving and sending messages from the app, and all other functions are restored.
Since these incidents began to occur periodically, we've been working to identify the root cause so we can fix it. For this particular issue, the cause has been extraordinarily difficult to isolate, as it does not occur during standard load tests of the system and appears to relate to specific usage patterns combined with accumulated customer and user data. We have our whole backend engineering team engaged with the effort, and restoring consistent, long-term system performance is our highest priority.
Posted Jul 07, 2021 - 22:24 UTC
Update
System performance is improved but not yet fully recovered. We're currently throttling requests from the mobile apps in an effort to give the system an opportunity to recover. Drivers may continue to see occasional delays or errors when sending messages and refreshing data in their apps.
Overall error rates (including errors resulting from throttling mentioned above) are currently around 5-8%.
Posted Jul 07, 2021 - 21:53 UTC
Update
We've reenabled app logins. Performance remains degraded and we're continuing to investigate.
Posted Jul 07, 2021 - 21:10 UTC
Update
In an effort to help speed recovery, we've temporarily disabled app logins. Performance is still degraded overall but response times are trending downward.
Posted Jul 07, 2021 - 20:47 UTC
Investigating
We're currently investigating degraded system performance. Drivers may experience delays sending messages and performing workflow actions.
Posted Jul 07, 2021 - 20:17 UTC
This incident affected: Eleos Platform (API, Mobile Apps).