An issue with one of our internal messaging services caused a huge sudden load on the underlying system, eventually leading to network issues. The worker node running these services crashed and had to be restarted. As several services became unreliable, customers experienced parts of TOPdesk that were slow to respond, and login problems as the authentication service didn't work as expected.
These problems started at 09:00 CEST and escalated to a point where they affected customers at 9:30 CEST. At 10:00 CEST the worker node in question was restarted and most services were again working as expected, resolving the impact on customer environments. Meanwhile, our Operations team kept troubleshooting the messaging service. At 11:00 CEST full functionality was restored.
The root cause of this problem is a rare bug in our messaging system which proves hard to troubleshoot. A replacement for the messaging system was already scheduled and is in the process of being deployed, which will prevent this issue from occurring again.
Posted Jun 09, 2021 - 13:34 CEST
Monitoring
One of the workers responsible for running several services crashed and had to be restarted. As several services became unreliable, customers experienced parts of TOPdesk that were slow to respond, and login problems as the authentication service didn't work as expected. These problems occurred between 09:00 CEST and 10:00 CEST.
At 10:00 CEST the restart was complete and most services were again working as expected.
Our hosting team is still investigating the root cause of the crashed worker node. Once the root cause has been found we'll update our status page.
Posted Jun 09, 2021 - 10:38 CEST
Investigating
We are currently experiencing problems on the NL3 hosting location. As a result your TOPdesk environment may be slow or unavailable.
We are aware of the problem and are working on a solution.
Our apologies for the inconvenience. We aim to update this status page every 30 minutes until the issue has been resolved.
E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates.