Services have returned to normal and users should expect full availability of dealer admin.
The issue was identified to be database performance related. Late Sunday evening we moved to a new database and the maintenance was completed without issue; the system was tested and verified. However, as normal traffic volume arrived Monday morning, the new database began showing signs of a latent performance problem that the previous database had masked.
To restore services, we removed some offending database queries and the team is now making permanent changes to ensure these queries can be scaled and performant.
Posted Sep 20, 2021 - 17:19 UTC
Monitoring
The cause of performance issues in dealer admin has been identified and a temporary fix has been deployed to improve availability. In the short term, users will not see order totals on their admin dashboard page. We are also maintaining a temporary throttle on requests to improve reliability. Users may experience occasional throttling notifications.
Posted Sep 20, 2021 - 16:19 UTC
Update
We are continuing to investigate an issue impacting performance of Roadster dealer admin. Users will experience inconsistent access, timeouts or application errors until a solution is identified. Our team is working on providing a fix as soon as possible and will provide updates hourly until resolved.
Posted Sep 20, 2021 - 15:07 UTC
Investigating
We have received reports of users having issues accessing dealer admin. This issue is being investigated.
Posted Sep 20, 2021 - 14:07 UTC
This incident affected: Americas Datacenter (Dealer Admin).