Postmortem: Android Phones getting error when launching the app | INC153333 P2

On Wednesday, February 15th at approximately 12 a.m. PT (3 a.m. ET) Google completed a scheduled retirement of  the v2 log list used by their Chrome Clients. This “log list” contains a list that is leveraged to validate SSL certificate authenticity for all Android Apps.  When this v2 list version was retired that our apps were dependent on, the apps were no longer able to locate the list, and thus the apps would not load.  If a member already had the app open when Google made the change, then the app would not have made this check (as it only verifies on app opening) and they would not have been impacted. However, any Android App users who launched an un-cached app (i.e., a new app or had previously swiped their app closed on their app screen) would have seen a pop-up message saying: "Your connection is not secure. Certificate transparency failed" when trying to launch their app. Cached Android Apps continued to work, and Apple (iOS) Apps and desktop banking were unaffected.

This log list is referenced upon app opening to validate the authenticity of SSL certificates for all Android Apps. The announcement to update the v2 log list was originally made in November 2021, with an October 2022 date proposed, but this date was delayed knowing that many business customers of Google were not prepared. An alternate date of February 15th, 2023 was selected, which still caught many businesses off guard. Due to the impact and the high demand of businesses requesting to roll back the change (Turning down Google's v1, v2 CT log list publishing), Google reversed the change at 8:20 a.m. PT (11:20 a.m. ET) and this restored app service. The impact lasted exactly 8 hours and 20 minutes.

Our estimate is that this likely affected anywhere between 5- 20% of Android users. .The current breakdown of iOS vs Android Apps for C1 clients is 67% vs 33%.

 Central 1 is prioritizing the actions needed prior to the future and final retirement of the v2 log list and will communicate our plan once developed. No new date has been provided by Google yet, but they have assured business users that they will provide ample notice. We do know that all C1 client apps will need to be updated to prevent this from happening again and we are working towards this.

 Actions:

RITM329453 - Product Governance – 3rd Party Lifecycle Support

Due date: Q3 2023

• Review current document 3rd party libraries, software, license model and other dependencies used by products
• Review the communication updates for each identified dependency

 RITM329454 - Google v2 log list sunset and new App release planning

Due date: ASAP (no later than October 2023)

• Plan for this dependency retirement – building the App without it, and the rollout strategy for the App

We recently experienced a service disruption that caused inconvenience and frustration for some of our customers. We want to assure all of our customers that we are fully committed to improving our service delivery and taking the necessary steps to prevent similar disruptions in the future. We have conducted a thorough postmortem analysis of the incident and identified several areas where we can make improvements.

If you have any questions about this postmortem please contact me to discuss.

Jason R Seale

Director of Client Support Services

jseale@central1.com | 778.558.5627