[SEV-3] Togetherpay - Intermittent Processing Issues
Incident Report for Togetherpay
Postmortem

ProPay identified the root cause of 9/28 processing incident. A specific service on a specific application node was responding slowly to transaction requests which led to transaction delays and timeouts.

Once the issue was identified, the server was removed from service, reviewed and refreshed. ProPay teams are pursuing several approaches to better prevent, detect and remediate similar situations.

Posted Oct 12, 2021 - 11:04 EDT

Resolved
This incident is resolved and service is operating normally.
Posted Sep 28, 2021 - 14:20 EDT
Monitoring
Between 1:00 p.m. - 2:00 p.m. eastern, we experienced elevated timeouts. ProPay identified an issue with a particular node in their system which has been removed. Processing has returned to normal.
Posted Sep 28, 2021 - 14:08 EDT
Investigating
Since 1:00 p.m. eastern, approximately 5% of Togetherpay transactions are timing out. We are in contact with ProPay to investigate.

As soon as additional information is known, we will let you know.
Posted Sep 28, 2021 - 13:31 EDT
This incident affected: Togetherpay API - Production (Transaction Processing, ProPay Production Environment).