Searches & Routing In Kustomer Delayed

Incident Report for Kustomer

Postmortem

Summary

Beginning around 13:00 PST, we noticed a spike in latency for searches and routing. After investigations, we found that this was due to AWS Lambda errors originating from the AWS service.

At around 13:21 PST Amazon reported that invoke times for the US-EAST-1 region had been resolved. We continued to monitor our data stream to ensure full recovery of systems.

At 13:36 PST we determined that the status had returned to normal and that data stream recovery had completed.

Impact/Alerts

There were delays to searching and routing upwards of 15 minutes.

Root Cause

This was caused by AWS Lambda invoke times and errors in the US-EAST-1 region which surfaced in long delays for searching and routing within Kustomer

Resolution

At 12:36 PST it was determined that Amazon had resolved their Lambda errors and systems had returned to normal

Lessons/Improvements

Considerations to improve upon our elastic search in the event of AWS outages.

Posted May 27, 2021 - 20:13 EDT

Resolved

Kustomer is seeing the platform recover as the Amazon Web Services return to normal. You may reference the Amazon Web Services status page for more information and updates on this incident. https://status.aws.amazon.com/

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.

Posted Mar 16, 2021 - 16:36 EDT

Monitoring

Kustomer is currently experiencing a delay with searches related to Amazon Web Services issue. We are working to resolve the issue as quickly as possible. During this time you may experience a delay with conversations loading in searches.

Kustomer reporting is seeing a decrease in errors as we monitor the situation. You may reference the Amazon Web Services status page for more information and updates on this incident. https://status.aws.amazon.com/

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.

Posted Mar 16, 2021 - 16:25 EDT

Investigating

Kustomer is currently experiencing a delay with searches related to Amazon Web Services issue. We are working to resolve the issue as quickly as possible. During this time you may experience a delay with conversations loading in searches.

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.

Posted Mar 16, 2021 - 16:13 EDT

This incident affected: Prod1 (US) (Events / Audit Log, Search).