Summary

A change made to improve scaling of Kustomer’s chat infrastructure had an unintentional side effect of preventing customers from sending attachments over chat for approximately 3 hours. While the change affected all pods, the issue principally affected customers in prod2 during morning business hours.

Root Cause

To address a critical need to address scaling up our chat infrastructure, a change was made that inadvertently caused rate limit calculations on chat attachments to false hit limits due to the increased number of instances. Ultimately, the configuration was updated to resolve this.

Timeline

Dec 18, 2024

9:00 PM EST - Chat infrastructure scaled up, affecting rate limit calculations for attachments.

Dec 19, 2024

1:09 AM EST - Clients in Prod2 report that attachments cannot be added to chats.

2:14 AM EST - Engineering was alerted to attachments not working.

3:26 AM EST - Deployment of increase to rate limits initiated, with functionality coming back online as this rolled out.

3:53 AM EST - Incident declared resolved

Lessons/Improvements

• Testing Enhancements & Alerting - While checkouts of the chat functionality were done as part of standard checkouts to scaling up changes, a gap was revealed to the lack of testing specifically using attachments at scale. Additionally alerting failed to capture this specific issue as it was only a subset of the total chats in progress.
• Rate limiting [Resolved] - Improvements to Rate Limiting logic have already been rolled out.