[Piano ID] Occasional user logouts (EU region)

Incident Report for Piano Software

Resolved

This incident has been resolved.
Posted 10 months ago. Sep 02, 2024 - 09:37 UTC

Monitoring

On August 29, 2024 Piano introduced a new backend cache in the EU cluster in order to improve latency and throughput for API calls and reporting related to Piano ID custom fields.

This cache activation was closely monitored, however the change resulted into unintended invalidation of some user sessions during the memory auto cleanup process. The issue wasn’t recognised right away because the number of invalidated sessions was below the threshold for the related monitor to trigger an alert.

We are improving our monitoring and memory policies to prevent this from happening in the future. The incident timeframe was between August 29, 05:30 UTC and August 30 8:00 UTC, and it is now fully resolved.
Posted 10 months ago. Aug 30, 2024 - 12:59 UTC
This incident affected: Login and Registration (EU - id.piano.io).
← Current Status Powered by Atlassian Statuspage