[Piano Analytics] Delay in real-time data
Incident Report for Piano Software
Postmortem

Incident explanation:

  • Due to a specific race condition, a limited part of the processing (about 1%) has been reprocessed which lead some delay until correction

Actions performed:

  • Data processing has been reconfigured to properly handle the streaming traffic.

Actions to come:

  • No more action needed

Posted May 25, 2023 - 09:36 UTC

Resolved
The issue has been resolved and the delay in real-time has now been caught up.
All scheduled exports have been sent.
Posted May 25, 2023 - 09:36 UTC
Identified
We have fixed the issue. Real time data delay has been caught up. We are now working on sending scheduled exports.
Posted May 25, 2023 - 08:34 UTC
Investigating
We are currently experiencing a delay in real-time data processing on Piano Analytics (since 5:30 AM UTC). This represents about 1% of all customer traffic.

Scheduled exports will be sent once the data is ready.
Our technical teams are actively working to resolve the issue.

We apologise for any inconvenience caused.

Affected customers - All customers
Posted May 25, 2023 - 07:36 UTC
This incident affected: Data Processing (Piano Analytics Data Processing), Data Collection (Piano Analytics Data Collection), and Dashboards, Interfaces and Platform (Piano Analytics Exports).