BluePay Transaction Data Partially Unavailable

Incident Report for CardConnect

Resolved

This incident has been resolved.
Posted 4 months ago. Jan 27, 2025 - 11:29 EST

Update

Duplicate settlements have been reversed/fixed.

Omaha Batches which did not settle on 6/19 were submitted for funding on 8/5, merchants should see funds processing through their accounts.

Please continue to monitor account activity.

We will continue to provide updates as they become available, if there are questions or concerns please contact your support team.
Posted 9 months ago. Aug 07, 2024 - 18:18 EDT

Update

Duplicate settlements have been reversed/fixed.

Omaha Batches which did not settle on 6/19 have been reviewed and prepared for settlement. Funding should be expected within the next 48 hours.

Please continue to monitor account activity.

We will continue to provide updates as they become available.
Posted 9 months ago. Aug 07, 2024 - 12:15 EDT

Update

Duplicate settlements have been reversed/fixed.

Omaha Batches which did not settle on 6/19 are in the process of being manually closed once validated.

Please continue to monitor account activity.

We will continue to provide updates as they become available.
Posted 10 months ago. Jul 26, 2024 - 12:16 EDT

Update

Duplicate settlements have been reversed/fixed. We are currently preparing a detailed master impact list for all affected items.

We have identified a grouping of Omaha Batches which did not settle on 6/19, We are working to review the pending authorizations and will close these batches once validated.

Please continue to monitor account activity.

We will continue to provide updates as they become available.
Posted 10 months ago. Jul 10, 2024 - 12:43 EDT

Update

Duplicate settlements have been reversed/fixed. We are currently preparing a detailed master impact list for all affected items.

We have identified a grouping of Omaha Batches which did not settle on 6/19, We are working to review the pending authorizations and will close these batches once validated.

Please continue to monitor account activity.

We will continue to provide updates as they become available.
Posted 10 months ago. Jul 09, 2024 - 12:15 EDT

Update

Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process.

Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed.

Duplicated credit card settlements on the First Data South processing platform are in the process of being reversed.

Device ID errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue.

Account managers will have full impact lists available by request in the coming days.

We will continue to provide updates as they become available.
Posted 10 months ago. Jul 08, 2024 - 10:09 EDT

Update

Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process.

Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed.

Duplicated credit card settlements on the First Data South processing platform are in the process of being reversed.

Device ID errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue.

Account managers will have full impact lists available by request in the coming days.

We will continue to provide updates as they become available.
Posted 10 months ago. Jul 03, 2024 - 13:01 EDT

Update

Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process.

Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed.

Duplicated credit card settlements on the First Data South processing platform are being worked on.

Invalid batch sequence errors caused some transactions from 6/19 not to settle, resulting in missing funds. The team is actively working on this issue.

Account managers will have full impact lists available by request in the coming days.

We will continue to provide updates as they become available.
Posted 10 months ago. Jul 02, 2024 - 12:43 EDT

Update

Duplicated ACH settlements have been reversed. Secondary issues with ACH returns are now in process.

Duplicated credit card settlements on the First Data North and First Data Omaha processing platforms have been reversed.

Duplicated credit card settlements on the First Data South processing platform are being worked on.

Account managers will have full impact lists available by request in the coming days.

We will continue to provide updates as they become available.
Posted 10 months ago. Jul 02, 2024 - 09:55 EDT

Update

Duplicated settlement reversals for all ACH transactions have been submitted.

North and Omaha duplicated settlement reversals have also been submitted.

We are continuing work through recovery efforts including correcting transaction records.

A full impact list will be provided to account managers to be provided upon request in the coming days.

We will continue to provide updates as they become available.
Posted 11 months ago. Jul 01, 2024 - 13:57 EDT

Update

We are continuing to correct the remaining double settled transactions.

We will continue to provide updates as they become available.
Posted 11 months ago. Jul 01, 2024 - 11:32 EDT

Update

We are continuing to correct the remaining double settled transactions.

We will continue to provide updates as they become available.
Posted 11 months ago. Jun 27, 2024 - 11:02 EDT

Update

Double settled Fiserv ACH Third Party Sender transactions have been reversed. We continue to work on reversals for Fiserv ACH Direct to Bank and credit card transactions.

We will continue to provide updates as they become available.
Posted 11 months ago. Jun 26, 2024 - 10:12 EDT

Update

Duplicate settlements on the BluePay Gateway are currently in the process of being reversed. This may take up to 3 business days to be completed.

Please subscribe to status.bluepay.com for updates to the timeline as we have them.
Posted 11 months ago. Jun 25, 2024 - 13:51 EDT

Update

We are continuing to work on a fix for this issue.
Posted 11 months ago. Jun 25, 2024 - 08:45 EDT

Update

We are continuing to work on a fix for this issue.
Posted 11 months ago. Jun 24, 2024 - 17:52 EDT

Update

Reversals of double settled credit card transactions over $10,000 on First Data processing platforms are pending settlement. Transaction Type is CREDIT for reversal of a SALE or CAPTURE. Transaction Type is SALE for the reversal of a REFUND. The Comment on the transactions is "Reversal of double settlement.".

Reversal of double settled credit card transactions under $10,000 on First Data processing platforms is starting now. It is expected to be completed Monday 6/24/2024.
Posted 11 months ago. Jun 23, 2024 - 16:36 EDT

Update

Reversal of double settled credit card transactions over $10,000 on First Data processing platforms has begun. Reversal of smaller First Data credit card transactions, credit card transactions on other platforms and ACH transactions are still being planned.
Posted 11 months ago. Jun 23, 2024 - 14:31 EDT

Update

We are continuing to work on a fix for this issue.
Posted 11 months ago. Jun 23, 2024 - 14:30 EDT

Update

Transactions that previously received a "COULD NOT LOCATE ORIGINAL TRANSACTION" error can now be reprocessed. The original transactions have been restored.
Posted 11 months ago. Jun 21, 2024 - 16:11 EDT

Update

We are continuing to work on a fix for this issue.
Posted 11 months ago. Jun 21, 2024 - 12:32 EDT

Update

During this time please refrain from manually refunding transactions appearing as duplicates. This will be resolved once a fix is in place.
Posted 11 months ago. Jun 21, 2024 - 11:17 EDT

Update

We are continuing to work on a fix for this issue.
Posted 11 months ago. Jun 21, 2024 - 08:41 EDT

Update

The data synchronization issue has caused some transactions to be double settled. After the missing data is restored we will be identifying the double settled transactions and determining the best way to reverse the duplicates.
Posted 11 months ago. Jun 20, 2024 - 16:18 EDT

Update

An application to synchronize the data across the database servers has been created. We are starting the process to restore the missing data. A completion time is not known yet.
Posted 11 months ago. Jun 20, 2024 - 14:01 EDT

Identified

A database server added to the cluster is failing to synchronize with the other database servers. This is causing some transaction and settlement data near the time of the addition to not be available for reporting and for use as tokens. We are working on correcting the synchronization issue and will provide an update when there is an ETA for correction of the issue.

Transactions using a token affected by this issue will receive the message "COULD NOT LOCATE ORIGINAL TRANSACTION". They will need to be reattempted after the issue is resolved.
Posted 11 months ago. Jun 20, 2024 - 09:36 EDT

Investigating

Additional data has been found to be missing. We will be working to correct the database synchronization issue and restore all the missing data.
Posted 11 months ago. Jun 20, 2024 - 01:22 EDT

Monitoring

Connectivity issue has been corrected and data merged. Data from all transactions is now available.
Posted 11 months ago. Jun 19, 2024 - 12:07 EDT

Identified

A connectivity issue between BluePay database servers caused transaction data from transactions processed 6/18/2024 4:16pm-4:26pm EDT to not be merged to all database servers. Affected transactions are currently not available in the BluePay website and not available from reporting APIs. We are currently working to correct the issue. Once connectivity is restored between the database servers, the transaction data will be merged making it available in the BluePay website and reporting APIs. The transactions will then settle in the next regularly scheduled settlement.
Posted 11 months ago. Jun 19, 2024 - 11:52 EDT
This incident affected: BluePay Gateway and Processors (ACH Payment Services by Fiserv).