Overview

On 11/05/21 a large number of email notifications were being delayed.

Timeline

1/05/2021

13:00 UTC - Currencycloud detected that some customers' emails were taking longer to be delivered than normal.

13:15: UTC - Currencycloud identified that email delivery was being throttled by recipient email servers.

14:00 UTC - Currencycloud started work on increasing the number of email delivery workers available.

15:07 UTC - Additional email delivery workers made available to send external emails.

15:40 UTC - Confirmation that emails were no longer being throttled by recipient mail servers.

17:08 UTC - Currencycloud continued to monitor the customer emails to ensure no further issues.

12/05/2021

08:11 UTC - After an extended period of monitoring we no longer see email notifications being delayed, so this incident is considered Resolved.

Resolution

Addition of email workers to our email delivery provider, Sendgrid.

Root Cause Analysis

An increase in traffic caused some customers' recipient email servers to throttle the delivery of Sendgrid email workers. This impacted a small number of mail servers and emails were re delivered on an IP address not impacted by throttling.

Remediation Items

Additional Sendgrid email workers have been added to be used in rotation to avoid a a single worker to be throttled based on it’s IP address. - COMPLETED
Segregate Sendgrid email delivery workers for priority client emails and other communications.