Connectivity issue
Incident Report for Altapay
Postmortem

Impact:

5 - Very high

Description & Scope of incident

All outgoing traffic was not passing through after a network/route preventing transactions from being processed in the timeframe 14:40 - 15:20

Analysis of Incident

A planned network change was implemented in the network infrastructure. This caused all outgoing traffic to the internet and 3rd party connections to fail while all incoming traffic was working as expected.

Our monitoring system was not set up to trigger an alarm on this exact scenario which caused a delay in detecting the issue.

Remedial Action

As soon as the root cause was identified the network change was reverted.

Mitigation plan:

  • Update the monitoring to include additional checks to catch similar issues.
  • Improve and review Internal change management process with focus on implementing measures which can avoid similar issues being introduced.
Posted Jan 27, 2023 - 13:28 CET

Resolved
This incident has been resolved
Posted Jan 26, 2023 - 16:35 CET
Monitoring
A fix has been implemented and we are monitoring the situation
Posted Jan 26, 2023 - 15:23 CET
Identified
The issue has been identified and we are working on implementing a fix.
Posted Jan 26, 2023 - 15:14 CET
Investigating
We are currently investigating a connectivity issue where payments are rejected
Posted Jan 26, 2023 - 15:11 CET
This incident affected: Networking and Processing.