Postmortem: Incident – INC160057 – PaymentStream Direct AFT Not Loading – Priority High (P2)

 On Thursday, May 25, 2023, at 6 a.m. PT (9 a.m. ET)  credit unions and corporate clients reported that they could not able access PS-AFT (Automated Funds Transfer) in PaymentStream Direct (PSD).  During the incident, when trying to access the AFT application in PSD the application would spin or time out.

 The identified point of failure was a file received with a larger than usual number of AFT transactions (11.7k), with an additional 400+ pending transactions. These transaction types resulted in a large number of pending transactions with null values. The query in place was not able to handle the volume of these transactions and it caused the query to hang and block other queries in the database. This is a pre-existing issue that has never been realized until we hit the combined volume of pending and null transactions that occurred in the file. To resolve the incident the team optimized the query to exclude null value transactions from file uploading since they are not required as part of this job (only data entry transactions pending are required to be included in the query).

 Incident Date and Time:

Start: 2023 May 25 – 6:20 a.m. PT (9:20 a.m. ET)

End: 2023 May 25 – 7:45 a.m. PT (10:45 a.m. ET)

Incident System Duration: 76 minutes

 ACTIONS

 Pending

RITM336534 – Central 1 will review and correct other queries that may exhibit similar behavior under high volume (Q2).

RITM336531 – Central 1 to ensure that database query monitors are in place for all payment Databases (Q2).

 Complete

PRB011125 – Root cause analysis

 If you have any questions about this postmortem, please reach out to me.

 Elizabeth (Liz) Ackroyd

Manager Client Support Services (Payments)

eackroyd@central1.com | 905.301.1337