Engineers have confirmed that all of the receipts that made it to our servers were replayed and reprocessed. We have another redundant layer of logs that we have also replayed as a backup. Number of purchases in our systems look consistent with the traffic from previous days. Additionally, SDKs will retry any receipt that did not make it to our severs, although we expect them to be minimal. Customer Lists are also catching up with the latest updates.
We are closing the incident as all systems are operational. We are very sorry for any inconvenience caused, and we will keep working to make sure our backup systems are even smoother for the future. If you have any specific concerns related to your app or customers, do not hesitate to reach out to RevenueCat support.
Posted Mar 29, 2023 - 18:06 UTC
Update
All read paths have been restored.
This incident will stay open until we confirm that all receipts have been replayed correctly. The vast majority of receipts have been replayed earlier, but we are doing final checks to make sure everything has been captured.
Posted Mar 29, 2023 - 14:16 UTC
Monitoring
Last seen data (subscriber last seen time, last seen country, last seen platform, etc) is being written again. Everything should now be receiving accurate last seen data with the exception of the subscriber info endpoint. Once we verify our last seen data infrastructure is reliable, we will enable this last read path.
Services are fully restored. With our new mitigation tool in place (details here: https://twitter.com/jeiting/status/1639393214837694464), there was minimal disruption for the customers and there was no data loss. SDK users were able to unlock purchases. However there were delays in Stripe and consumable purchases, as well as some webhooks. But purchase data was not lost. We are working on catching up on the last-seen data (which mostly affects charts). We will keep you posted on the progress. Meanwhile if you have any questions, please do not hesitate to reach out to RevenueCat Support.
Posted Mar 29, 2023 - 09:06 UTC
Update
Update: Most services have been restored, however some users may see stale last-seen data. Engineers are arranging to replay purchases into the primary database. We will update again when the system is fully restored.
Posted Mar 29, 2023 - 06:46 UTC
Update
Engineers have traced the issue to one of our non-critical storage layers and are looking at ways to bypass. New purchases are still succeeding. More updates as investigation proceeds.
Posted Mar 29, 2023 - 05:37 UTC
Update
We are continuing to investigate this issue.
Posted Mar 29, 2023 - 05:15 UTC
Investigating
Engineers are mitigating the issue by enabling our backup systems which is expected to allow purchases to proceed. Engineers anticipate no impact to new and existing purchases. Webhooks may be delayed. More updates as investigation proceeds.
Posted Mar 29, 2023 - 05:13 UTC
This incident affected: API Uptime, Scheduled Data Exports, and Event Dispatching.