Bots are currently not answering

Incident Report for iAdvize (HA)

Postmortem

This incident occurred after a deployment made early in the morning. The release content is not the cause of the incident.

The problem lies in restarting the services affected by the deployment. This action requires a large amount of memory and CPU. It seems that the node used this morning was too limited in CPU preventing the bot services from starting properly.

Around 9:25am (CEST), we manually ejected this node in order to force a reboot on a node with enough CPU.
We will adapt the server configuration to prevent this problem in the future.

Posted Sep 28, 2022 - 14:11 CEST

Resolved

After a monitoring period, we confirm the end of the incident. A post mortem will be published following this closure.
We thank you for your patience and apologize for the inconvenience.

Posted Sep 28, 2022 - 14:09 CEST

Monitoring

The service has been restored and the bots are operational again. We are continuing our analysis and monitoring the activity.

Posted Sep 28, 2022 - 09:37 CEST

Update

We have decided to cut off all bot engagement until we find the source of the problem. Therefore, if you have prequalifying bots, the notification is not being displayed anymore. Your visitors cannot initiate new conversations with bot.
No impact on conversations with agents only.

Posted Sep 28, 2022 - 09:09 CEST

Investigating

Our bot service stopped working at around 8:10 am (UTC+2). As a consequence, the bots are currently not answering. We are trying to restart the service.

Posted Sep 28, 2022 - 08:39 CEST

This incident affected: Bot (Bot service (except IA features)).