IF emails not being delivered

Hotjar Service Status

This is Hotjar's status page, where you can get updates on how our systems are doing. If there are interruptions to service, we will post a report here.

Report an issue

Incident Report for Hotjar

Postmortem

What happened?

Between 10:34 AM UTC on July 28th and 14:03 UTC on July 29th, email notifications containing attachments, primarily Incoming Feedback and Invoice emails, were not being sent.

We’ve implemented a fix for this and any queued or failed email notifications will gradually be sent out over the next 48 hours approximately.

‌

Why did this issue occur?

After upgrading the Celery version, the serialization protocol changed, causing some jobs to receive malformed events.

‌

What will we do to prevent this from happening in the future?

We didn't receive alerts for this issue, so we'll look at how we can improve alerting and logging for these types of issues. Also, we’ll be looking at how to detect serialization/deserialization issues with end to end tests for the background jobs.

Posted Jul 29, 2021 - 14:37 UTC

Resolved

The issue has been resolved. Thank you for hanging in there with us! We'll be posting more details here with a postmortem shortly.

Posted Jul 29, 2021 - 14:31 UTC

Update

The issue has been resolved. Thank you for hanging in there with us! We'll be posting more details here with a postmortem shortly.

Posted Jul 29, 2021 - 14:30 UTC

Monitoring

We've implemented a fix for the issue and we're now monitoring the situation. There will be a number of email notifications in a queued or failed state. We'll be reprocessing these emails as soon as possible.

Posted Jul 29, 2021 - 14:09 UTC

Identified

We've identified the cause as to why email notifications are not being sent for some of our tools - mainly Incoming Feedback and Invoice emails. The issue seems to impact emails that contain attachments. Our team is currently working on a fix for this issue.

Posted Jul 29, 2021 - 12:56 UTC

Investigating

We're currently investigating an issue with Incoming Feedback email messages not being sent. Our engineers are working on a fix ASAP and we'll be posting updates to the status page. Thanks for bearing with us!

Posted Jul 29, 2021 - 12:12 UTC