API & analytics unavailable
Incident Report for Onfleet
Postmortem

On 2022-09-01 03:06 UTC, we were notified that some users were experiencing issues accessing certain features through the dashboard and API. A database update earlier in the day resulted in some documents being modified incorrectly, which impacted feature access for the associated users. We performed a fix at 2022-09-01 04:43 UTC that resolved the issue for most users. We were notified again at 2022-09-01 14:08 UTC that some users continued to experience issues with the API. We determined that this was due to unexpected behavior with our caching system and performed a full cache refresh at 2022-09-01 15:50 UTC. This fully resolved the issue for all affected users.

We are enacting procedural changes to prevent similar database errors in the future and to resolve related incidents quicker. We are also implementing changes to our caching system to fix the issue experienced and detect such issues quicker.

Posted Sep 15, 2022 - 14:06 PDT

Resolved
This incident has been resolved.
Posted Aug 31, 2022 - 22:14 PDT
Monitoring
We have applied a fix and are watching to make sure access to the API and analytics has been restored.
Posted Aug 31, 2022 - 21:37 PDT
Identified
We discovered an issue after a recent database update. This issue affects some customers and prevents their use of our API and analytics, as well as other functionality. We are working on a fix currently.
Posted Aug 31, 2022 - 20:56 PDT
This incident affected: API and Analytics.