Resolved -
Central 1 has successfully recovered all services that were impacted from today’s incident by completing an internal network change, which fully restored traffic between our Forge and OAuth systems.
The initial outage has been linked to maintenance of our Storage Area Network (SAN). The change was expected to be non-service disrupting and a full postmortem will be completed in the coming days to ensure we can improve our change management communications as well as mitigate impacts of these types of changes in the future. Central 1 will finish this change this evening with an onsite subject matter specialist to mitigate any further risk of unintended consequences. The remaining maintenance activities will be initiated at 11 pm PT and are expected to be completed within the hour. The remaining maintenance is on the standby SAN, which is not currently supporting production traffic, which greatly reduces any risk of further impact. IF we did not proceed this evening, it would reduce our high availability standby state with the SAN.
Impacts today starting today at 1:04 p.m. PT (4:04 p.m. ET):
e-Transfers recovered at 1:25 p.m. PT (4:25 p.m. ET)
Deposit Anywhere recovered at 1:25 p.m. PT (4:25 p.m. ET)
Bill Payments services recovered at 4:10 p.m. PT (7:10 p.m. ET)
Digital banking login partially recovered at 2:46 p.m. PT (5:46 p.m. ET)
Digital banking full recovery at 6:10 p.m. PT (9:10 p.m. ET)
We understand the significance of identifying the root cause of these issues to prevent any recurrence in the future. To that end, our team will be conducting a thorough investigation into the matter to perform a comprehensive root cause analysis in the coming days and will distribute the report upon completion.
Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878
May 5, 19:15 PDT
Update -
Central 1 teams are engaged and triaging. All Payment Services except Deposit Anywhere have recovered. Digital Banking Services may still be intermittent. Biometric login via Mobile App may be affected as well, manual login is intermittent. All other services within Online Banking are available.
Incident began at 1:04 P.M. PT / 4:04 P.M. ET.
We will provide an update by 7:00 p.m. PT / 10:00 p.m. ET.
May 5, 18:07 PDT
Update -
Central 1 teams are engaged and triaging. All Payment Services except Deposit Anywhere have recovered. Digital Banking Services may still be intermittent. Biometric login via Mobile App was unaffected, however manual login is intermittent. All other services within Online Banking are available.
Incident began at 1:04 P.M. PT / 4:04 P.M. ET.
We will provide an update by 5:45 p.m. PT / 8:45 p.m. ET.
May 5, 16:51 PDT
Update -
Central 1 teams are engaged and triaging. Most Payments services have recovered however Digital Services may still be intermittent.
Incident began at 1:04 P.M. PT / 4:04 P.M. ET.
We will provide an update by 4:45 p.m. PT / 7:45 p.m. ET.
May 5, 15:46 PDT
Update -
Central 1 teams are engaged and triaging. Some Payments services have recovered however Digital may be intermittent or down.
Incident began at 1:04 P.M. PT / 4:04 P.M. ET.
We will provide an update by 3:30 p.m. PT / 6:30 p.m. ET.
Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878
May 5, 14:37 PDT
Investigating -
Central 1 is currently investigating an incident with both Payments and Digital Services. Members and clients may not be able to access these services at the moment. Technical teams are engaged and troubleshooting. We will provide an update by 2:30 p.m. PT / 5:30 p.m. ET.
Central 1 - Support@central1.com - DigitalBanking_Support@central1.com - 1.888.889.7878
May 5, 13:38 PDT