Delayed data processing and message sending on US 01, US 03, US 04, US 06

Incident Report for Braze, Inc.

Resolved

Data processing times across all clusters have returned to within normal ranges for the vast majority of customers (>98%) and should be fully resolved for all customers in a few minutes.

Posted Sep 30, 2021 - 11:42 EDT

Update

Message sending has sent through the backlog and returned to normal sending speeds everywhere.

Posted Sep 30, 2021 - 11:26 EDT

Update

We are continuing to monitor for any further issues.

Posted Sep 30, 2021 - 11:14 EDT

Update

We are continuing to monitor for any further issues.

Posted Sep 30, 2021 - 11:11 EDT

Monitoring

We have implemented a fix for the issue and are scaling up additional server capacity to process the backlog. The largest delays are to SDK data processing, most message sending is operating without delays.

The issue was caused by one dependency in the server build chain relying on a LetsEncrypt root certificate which expired at 10am ET today (https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/).

Posted Sep 30, 2021 - 11:06 EDT

Identified

We have identified an issue causing delayed data processing and message sending on and are working toward resolution. This issue affects customers on US 01, US 03, US 04 and US 06.

Posted Sep 30, 2021 - 10:46 EDT

This incident affected: US 01 Cluster (Data Processing, Outbound Messaging), US 03 Cluster (Data Processing, Outbound Messaging), US 04 Cluster (Data Processing, Outbound Messaging), and US 06 Cluster (Data Processing, Outbound Messaging).