Issue with several product

Incident Report for Tink

Postmortem

Introduction

On 2022-07-11 between 22:30 and 23:10 CEST, We experienced an issue where success rates for all products dropped significantly and all credentials (all users) stuck in status AUTHENTICATING.

Root Cause Analysis

An infrastructure issue caused several of our services to become unresponsive, causing all requests during the time period to be blocked internally.

Remediation

The issue with the infrastructure was solved by a rotation/restart and once done, the corresponding services had to be restarted as well causing the incident to take some time to resolve.

Follow-up Actions

Improve our observability surrounding our node management system.
Increase reliability of our node management system.

Posted Jul 14, 2022 - 10:10 CEST

Resolved

This issue has been resolved. We apologize for the inconvenience.

Posted Jul 11, 2022 - 23:46 CEST

Monitoring

The service is currently recovering and we are actively monitoring the situation.

Posted Jul 11, 2022 - 23:22 CEST

Identified

The issue has been identified and we are currently working on a solution.

Posted Jul 11, 2022 - 23:15 CEST

Investigating

We are currently investigating an issue that is affecting several Tink services.

See details for the full list of affected services.

Posted Jul 11, 2022 - 22:49 CEST

This incident affected: Payments, Account Check, Transactions, and Income Check.