Unexpected extension to the maintenance window due to database CPU load

Hotjar Service Status

This is Hotjar's status page, where you can get updates on how our systems are doing. If there are interruptions to service, we will post a report here.

Report an issue

Incident Report for Hotjar

Postmortem

On Jun 17, 2021 at 5:00 UTC we started a scheduled database maintenance. As we restored services, our engineers noticed an extra load to our main database cluster which caused us to call an incident and extend the maintenance window.

What happened?

After we restored services due to the database maintenance, we identified an issue with an extra load on our main database cluster. Due to that, the Hotjar App and data ingestion were offline while we worked to stabilize the issue. Data tracking was offline from 5AM UTC up to 12:20 UTC.

Why did this issue occur?

After we tried to restore services after maintenance mode, we had some issues with the data processing on our main database.

What will we do to prevent this from happening in the future?

We added more performance-related tests to our database migrations procedure and cleaned up data for discontinued features.

Posted Jun 17, 2021 - 16:05 UTC

Resolved

The issue has been resolved. Thank you so much for your patience!

Posted Jun 17, 2021 - 15:27 UTC

Monitoring

Our engineers have fixed the issue. The Hotjar app is back up and data ingestion is online as well, but with degraded performance as of now. We're monitoring the situation to make sure everything is working smoothly again.

Posted Jun 17, 2021 - 13:00 UTC

Identified

We've identified the issue with the extra load on our database. The Hotjar App is back up and data ingestion is back online as well, but with degraded performance for now. Our engineers are working to resolve it!

Posted Jun 17, 2021 - 12:32 UTC

Update

The Hotjar App is back to maintenance mode as we keep investigating this issue. Again, sorry for all the trouble here!

Posted Jun 17, 2021 - 11:14 UTC

Update

The Hotjar app is currently back up but with degraded performance. Data Tracking has been offline since the beginning of this maintenance period and we're working to get this fixed as soon as possible. Thank you for your patience!

Posted Jun 17, 2021 - 10:11 UTC

Investigating

Due to an extra load on our main database cluster as we restored service, we needed to extend our maintenance period to longer than what was expected. We're working to fix this ASAP. We're sorry for the trouble here and thanks for hanging in there with us!

Posted Jun 17, 2021 - 09:03 UTC