SEV3 Duplicate Bank Transfers Issue

Incident Report for ProPay

Postmortem

This event was caused by a misunderstanding during a project of migration of scheduled jobs.

Jobs were incorrectly enabled in more than one place, causing them to be run simultaneously and resulting in some duplicate bank transfers.

Additional steps have been put in place to prevent duplicate triggers being configured for bank transfer jobs.

Posted Aug 31, 2021 - 09:17 MDT

Resolved

Rather than re-delivering Deposit Detail Reports, we will be sending specific transaction data to the handful of affected partners. Some affected partners may have received a second report containing no data, which can be ignored. For those where the extra deposits and withdrawals were reported, we will provide the specific transaction identifiers that were voided so they can be removed from partner systems.
We expect at this point that merchants should not see extra deposits or withdrawals correlating to the duplicates reported today.
Please contact your relationship manager or the Client Support team with any specific follow up questions or if you'd like to request an affected Deposit Detail Report be re-generated and delivered.
We have identified the source of the error related to a manual task as part of the release this morning and are working on several action items to prevent similar situations in the future.
We sincerely apologize for the disruption this morning.

Posted Aug 04, 2021 - 12:26 MDT

Monitoring

We have corrected the duplicate bank transfers and are working on distributing the details of the transfers that were voided so affected partners can update their systems to reflect the activity accurately. This only potentially impacted accounts where bank transfer jobs are scheduled between 4am-9am Mountain time.

Posted Aug 04, 2021 - 11:21 MDT

Update

We are implementing a change to prevent the duplicate bank transfers (debits and credits) from being actually submitted to NACHA for processing. We will also be working to re-deliver impacted Deposit Detail Reports to more accurately reflect the outcome.

Posted Aug 04, 2021 - 09:52 MDT

Investigating

We're investigating a situation where the daily bank transfer process for a few limited partners this morning was initiated twice, causing duplicate bank transfers. The funds have neither been sent nor pulled from accounts yet, so we are working to correct the data to prevent financial impact. We will provide additional updates as we have more details.

Posted Aug 04, 2021 - 08:45 MDT

This incident affected: Production Supporting Solutions (Bank Transfers).