Postmortem
On 20/10/2021 at 11:41am UTC our engineers released a version of our tracking script which set invalid cookie values to our customers’ domain. This caused some customer sites to become inaccessible or behave incorrectly for some of their users.
What happened?
The cookie values that were set are considered invalid and insecure by certain firewall and server configurations which inspect request headers and block requests. We released a mitigation and a fix at 3:42pm UTC which should remove these invalid cookies and restore functionality to those behind firewalls as long as the Hotjar client script is still running. If sites are still inaccessible, clearing cookies or deleting the specific cookies prefixed with _hjSession_ and _hjSessionUser_ should restore functionality.
What will we do to prevent this from happening in the future?
We held an internal post-mortem and have identified a number of improvements that we’re working to implement in our process in order to prevent issues like this from happening in the future. These relate to both post-change testing on our end, and improvements to the process that we follow when releasing similar configuration changes. We will update this post-mortem when these actions have taken place with more details of exactly what has changed.