Kustomer Loading Error
Incident Report for Kustomer
Postmortem

Summary

On May 17th 2021 beginning at 11:29 AM EST until 11:37 AM EST users visiting the Kustomer platform were shown a “403 Forbidden / Access Denied” in their browser. After the configuration was reverted, users were intermittently unable to load parts of the Kustomer platform until 12:50 PM EST.

Impact / Alerts

Services impacted:

  • Kustomer Website as a whole

Root Cause

The root cause was due to a misconfigured security policy applied to the Kustomer Web platform. The misconfigured policy was initially applied at 11:27 AM EST. Health check notifications in test environments failed to reach the internal team resulting in the security policy being applied to production. The error was discovered on production at 11:29 AM EST through health check notifications. We immediately triggered the revert which completed at 11:37 AM EST. Full recovery was delayed due to our aggressive caching policies which cached the error response. At 12:43 PM EST, we issued invalidation for all the caches involved, leading to full recovery by 12:50 PM EST.

Resolution

Revert the misconfigured security policy and invalidate affected caches.

Lessons/Improvements

  • [COMPLETE] Fix health checks notification for test environments
  • [TODO] Optimize caching policies to exclude caching error responses
Posted May 18, 2021 - 15:12 EDT

Resolved
Kustomer is was experiencing loading errors. There was an update to our server which was preventing loading. This update has been reverted. During this downtime, you may have received a 403 error and were unable to load the customer platform.

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page
Posted May 17, 2021 - 11:47 EDT
Monitoring
Kustomer is was experiencing loading errors. There was an update to our server which was preventing loading. This is already being reverted. During this time you may have received a 403 error and were unable to load the customer platform.

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page
Posted May 17, 2021 - 11:45 EDT
Investigating
Kustomer is currently experiencing loading errors . We are working to resolve the issue as quickly as possible. During this time you may not be able to load the customer platform.

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.
Posted May 17, 2021 - 11:39 EDT
This incident affected: Prod1 (US) (Web Client) and Prod2 (EU) (Web Client).