Summary
Between Monday, Mar 13, 2023, 15:30 UTC to 18:25 UTC, we have suffered from a dysfunction of our outgoing emails for all Infinity Portal notifications in all regions.

The event was triggered by an account issue at our external email provider service. Once identified we have contacted the service provider and got the issue resolved.

All outgoing emails that were suspended during the incident got released and sent successfully automatically once the issue was resolved.

Incident Timeline

• Monday, Mar 13, 2023, 15:30 UTC – Our email provider suspends our account’s ability to send out emails due to a usage issue.
• Monday, Mar 13, 2023, 15:40 UTC – Complaints from users not receiving emails are starting to come in. Our On-call engineer is starting to investigate.
• Monday, Mar 13, 2023, 17:00 UTC – Root cause identified as an account issue with the email provider. Contacting their support asking for a fix.
• Monday, Mar 13, 2023, 18:20 UTC – Issue resolved, and customers reporting that they are getting the emails.

Root Cause Analysis

The Infinity Portal is using a single external provider to send out emails for many of the validation and notification flows of the portal. Due to some kind of mis-use of the service’s terms of use, they have tried to reach out to us but failed to have the updated Point-of-Contact, thus getting no response. As a result, after a week, they have suspended all outgoing emails.

Lessons Learned

We want to apologize for the impact this event caused for our customers. While we are proud of our long track record of fulfilling all features' SLAs of the Infinity Portal, we know how critical the email service is to our customers, and their businesses. We will do everything we can to learn from this event and use it to improve our email availability even further. We are going to fix the notification point-of-contact so future reach out will be answered, and align to the terms of use of the provider to ensure the service usage.