Unplanned Outage
Incident Report for Anaplan
Postmortem

On April 22 at 05:22 UTC, Anaplan engineers were alerted to a network issue by our automated monitoring systems and began investigating immediately.

The investigation determined that our Amsterdam data center was experiencing packet loss on the network connection to our Washington data center. Once we identified the backbone connection between Amsterdam and Washington was at fault, Anaplan engineers switched the network connection to our failover link at 08:00 UTC, allowing traffic to route through a different network circuit.

The issue with the network connection was caused by maintenance being performed by our external providers. While we had earlier been informed of configuration work scheduled by our data center provider, it was classified as "no impact" maintenance, and we have since followed up with our data center and network providers to ensure they investigate the issues that occurred.

As additional preventive measures, Anaplan is improving its automated alerting to more quickly identify network degradation. We are also improving the tooling used to debug network issues during an outage such as this one. The aim is to more promptly distinguish a network issue from an application issue, which will lead to quicker resolution. A final change we are investigating is the potential for an automated switchover to the failover network circuit when connection issues arise.

We understand the impact to your experience when these events occur, and we are working diligently to improve our systems and processes to mitigate further recurrence of these issues. We appreciate your patience, and as always thank you for being a valued Anaplan customer.

Posted May 05, 2021 - 02:37 UTC

Resolved
We have confirmed with customers that the Anaplan platform is now accessible. We apologize for any inconvenience that this issue may have caused. We appreciate your patience while we document a comprehensive root cause analysis. If you have any further questions, concerns, or continue to receive an error, please don't hesitate to contact us at support@anaplan.com.
Posted Apr 22, 2021 - 08:31 UTC
Monitoring
Our engineers have restored service to the Anaplan platform. You should now be able to resume normal activities. We will continue to monitor the situation to ensure no additional issues arise and contact our customers to validate and confirm the platform is functional and operational. If you have any questions, concerns, or continue to receive errors, please don't hesitate to contact us at support@anaplan.com. Once again, we are extremely appreciative of your patience and understanding.

Furthermore, we would like to apologize for any inconvenience caused by the lack of status emails. We have uncovered the root cause of the notification failure and emails should now resume as expected
Posted Apr 22, 2021 - 08:16 UTC
Update
Thank you for your patience as we continue to investigate reports of Anaplan accessibility issues. If you continue to experience issues please don’t hesitate to contact us at support@anaplan.com.
Posted Apr 22, 2021 - 07:48 UTC
Update
Thank you for your patience as we continue to investigate reports of Anaplan accessibility issues. If you continue to experience issues please don’t hesitate to contact us at support@anaplan.com.
Posted Apr 22, 2021 - 07:47 UTC
Update
We are continuing to investigate this issue.
Posted Apr 22, 2021 - 07:16 UTC
Investigating
We are continuing to investigate the reported issues regarding Anaplan accessibility. We will provide regular updates on this page as we diligently work to resolve this issue(s). Please contact Anaplan Customer Care at support@anaplan.com if you continue to experience issues. Thank you for bearing with us.
Posted Apr 22, 2021 - 06:56 UTC
Identified
We have isolated the components impacted by the reported service interruption and have all engineers actively working to restore service to the Anaplan platform. A quick resolution is our top priority and we are committed to further investigating the root cause of this issue. We apologize for any inconvenience that this may have caused and appreciate your continued patience while we work toward a resolution. Thank you for being a valued Anaplan customer.
Posted Apr 22, 2021 - 06:38 UTC
Investigating
We are receiving reports of our customers experiencing issues accessing the Anaplan platform. Our engineers are aware and actively investigating. We will provide regular updates on this page until the issues are resolved. We apologize for the inconvenience this may be causing and appreciate your patience.
Posted Apr 22, 2021 - 06:17 UTC