The Talkeetna cluster is an HP-UX cluster of servers that hosts Banner Job Submission services, Banner print queues, and runs most Application Manager jobs. Users often will use SSH, SCP, or SFTP clients to connect to the hosts in order to place or retrieve files used by Banner and Application Manager. The functionality of these servers is currently being migrated to the Ellucian Cloud as part of the Banner Cloud migration project with a target go live of October 2022.
Around 8:30 AM, technicians became aware of login issues being reported by end users trying to access prod.alaska.edu. At 9AM it was identified that the cluster node that provides authentication services for end-user SSH and SFTP access quit responding to authentication requests. Technicians were able to restore authentication services at around 12:30 PM after the cluster restarted.
Service should be available 24/7 except for published maintenance windows.
At around 12 AM, Tazlina, one of the cluster nodes, lost a disk that was part of its root volume group vg00. This volume group was mirrored to another disk so no data was lost, however the logical volumes in the volume group were unmounted from the file system which caused the kerberos service that handles authentication for the cluster and many other services on Tazlina to fail as the data needed resided in a path that was no longer accessible. The volumes that went offline due to the failed disk included the root path /, /var, /opt, /stand, /home, /usr, /usr/users, /var/adm/crash, and /hyperion.
Most of the cluster packages and processes including job submission and app manager agents were running on Tetlin, another cluster node, so this failure did not impact monitored services and was undetected until end users reported issues authenticating. At 11:56 AM Tetlin tried to Transfer of Control (TOC) to Tazlina. Tazlina was unable to take control due to the missing volumes and so this triggered a restart of both Tetlin and Tazlina in order to preserve data integrity. Upon reboot, Tazlina was able to remount the logical volumes off of the mirrored physical volume of vg00. Cluster services were started on Tetlin as a single node cluster, and the kerberos service was restarted on Tazlina to enable end-user authentication for the cluster.
May 23, 2022: 10:50 AM: The failed disk was identified as the top right disk on the chassis
May 23, 2022: 3:30 PM: Technicians searched through the data center but were unable to locate a replacement disk. Currently working to identify support or purchase options.
May 23, 2022: 5:30 PM: Technicians completed removing the failed disk from all of the logical volumes and volume group.
June 3, 2022: Replace failed disk and reconfigure mirrors for logical volumes.
June 10, 2022: Implement fix for more resilient kerberos service.