Alert! - INC150402 - Vancouver Data Center (VAHC) power outage causing network problems
Incident Report for Central 1
Postmortem

Postmortem: INC150402 - Vancouver Data Center (VAHC) power outage causing network problems - P2

Summary:
On Wednesday, November 9th, Central 1’s Vancouver data center vendor performed scheduled maintenance on one of the power sources. When the maintenance started at 10 p.m. PT (1 a.m. ET), due to a configuration error, a critical device lost power preventing external network traffic into the data center. The device was identified and powered into an alternate power source restoring services by 11:45 p.m. PT (2:45 a.m. ET).

Postmortem:
On Wednesday, November 9th, Central 1 experienced a network outage affecting our Treasury, Payments and Digital Banking products that rely on servers and databases in VAHC as a result of the loss of power from the maintenance mentioned above.

Impacted features:
• Payment features including Bill Payments, e-Transfers and Deposit Anywhere were unavailable in online banking.
• MemberDirect and Forge would have been available because Digital Banking platform has resiliency to fail traffic automatically to our Toronto Data Center, as long as the financial institution supports connectivity to both data centers.
• RSA services including Increased Authentication and 2 Step Verification (2SV) would have been unavailable due to a connection failure to AWS.
• Database reliant digital banking features including last login and mobile alerts would not have been available.

At 10:10 p.m. PT, Central 1 network teams engaged their vendor and internal support teams to review impact and assess a recovery plan.

Known cause for failure:
Central 1 was able identify the point of failure was a hardware failure in VAHC due to the planned power outage. Due to a configuration error, the network equipment that caused the outage did not have redundant power supply, causing loss of external connectivity (all hardware is expected to have dual power sources). Once the switch was physically plugged into alternate power source by someone inside the Vancouver data center, it was powered up and services started to immediately recover.

The device was powered back on by 11:45 p.m. PT (2:45 a.m. ET) resulting in a 1 hour and 45-minute service outage.
Central 1 delayed the failover to our Toronto Data Center as the known path to recovery would be completed sooner and with less risk.

Impact Assessment:
Affected Service(s): Payment, exchange, AWS and digital banking services requiring the MD database.

Affected FI’s: All Central 1 clients for Digital Banking, Payments and Treasury services.

Incident ticket opened: 2022-11-09 22:57 PT and resolved 2022-11-09 23:32 PT

Central 1 Actions:
RITM321773 – Network: Datacenter Resiliency Improvements
Due Date: Q1 2023
Enhancements to monitoring and cluster configurations including a 3rd party assessment RITM321774 – Payments: Application Resiliency Improvements
Due date: Q1 2023
• A review of the Payments Database architecture
RITM321775 – Afterhours Incident Management Improvements to improve staff engagement and reduce mean time to resolve ff hours incidents.
Due date: Q4 2022

Central 1 is working on many initiatives in our Operational Excellence program which will help ensure the stability of services and delivering the quality of product expected from Central 1. We apologies for these outages knowing the impact it has to your credit union’s reputation and the ability of your members to use your reliable services.

Jason Seale, PMP
Director, Client Support Services

Central 1 Client Support Services
1441 Creekside Drive, Vancouver, BC, Canada V6J 4S7
T 1 800 661 6813 ext. 5185 C 778 558 5627 Support 888 889 7878
jseale@central1.com www.central1.com

Posted Dec 01, 2022 - 16:28 PST

Resolved
Power in the Vancouver Data Center (VAHC) has been restored and all services have recovered. Our teams will continue to monitor connections. Please contact Support is you experience any further issues.

Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 0
Posted Nov 09, 2022 - 23:50 PST
Identified
Please note that Central 1 is currently experiencing a network outage in the Vancouver Data Center (VAHC) due to a power outage.

Teams are currently working on recovering the power. Multiple online banking services (MemberDirect and Forge 2) are affected and also, Secure Site and PaymentStream Direct applications. An update will be provided by or before 12:30 a.m. PT (3:30 a.m. ET).

Central 1 - DigitalBanking_Support@Central1.com - 1.888.889.7878, option 2
Posted Nov 09, 2022 - 23:27 PST
This incident affected: Payment Services, Digital Banking Services, and Incident Alerting.