Summary:
On February 2nd and 3rd there was a Teamsite 8 outage for 15 to 30 minutes each day. Service was restored by an application restart.
Postmortem:
Teamsite 8 is our content management system that MemberDirect clients use for making changes to their public websites, and both MemberDirect and Forge 2 clients leverage for client-controlled changes within the online banking experience.
On February 2nd between 1:30 to 2 p.m. PT (4:30 to 5 p.m. ET) Teamsite 8 was unavailable for credit union staff to use (INC152946). There was no direct impact to members from this incident. The application restart restored service within minutes however it does take approximately 2 additional hours for the preview capabilities (used for user acceptance testing on changes) to be restored. This is typical after service restart due to content syncing.
On February 3rd, between 1:30 to 1:45 p.m. (4:30 to 4:45 p.m. ET). Teamsite 8 became unavailable again for credit union staff to use to make their updates (INC152976). There was no direct impact to members, and there was the 2-hour recovery for the preview site as well.
Known point of failure:
Central 1 uses a backing store with Teamsite to store information not currently in main memory for optimizing available disc space on Teamsite. Central 1 did review our memory consumption increase due to recent changes going live this past year for our MemberDirect Business product (MDB). The MDB builds increased sharply from 20-30 a year to over 400 in 2022.
Platform performed a one-time cleanup to improve memory consumption (CHG132420) and added monitoring until we move to a more optimized Teamsite release architecture (which will completely remove this risk).
Impact Assessment:
Affected Service(s): Only Teamsite 8 content management system used by CUs to update their content – no direct member impact.
# Affected FI’s: All Teamsite 8 customers
# Affected End Users: 50 per Incident.
Impact windows: The application itself was unavailable for a total of 38 minutes across the 2 Incidents., although it took two more hours (each time) to recover the preview site used for testing changes before pushing them live.
Central 1 Actions:
CHG132420 - Clean up old EDITIONS for MDB Core (Completed)
PRB011067 – Teamsite Improvements
Due date: End of March 2023
PRB011068 – Teamsite Memory Management
Due date: End of March 2023
Build a process need to delete unneeded data from historic builds and unused content.
Central 1 is moving towards an improved architecture of our content management system in 2023. The improvements will mitigate the legacy management of an older Teamsite version (moving from Teamsite 8 to 20). We appreciate the credit unions patience during this transition and will communicate upcoming changes and improvements in release notes.
If you have any questions about this postmortem please do not hesitate to reach out to me.
Jason Seale
Director, Client Support Services
jseale@central1.com | 778.558.5627