ACS University of Alaska Network Outage
Physical Defect in ACS’ Fiber Results in Outage
Event Occurrence: April 24, 2024
The University of Alaska receives network services from ACS. The ISP provides the long haul WAN circuits that comprise the University of Alaska’s core network from/to Fairbanks, Anchorage, Juneau, Seattle, and Portland. The circuits from Fairbanks <-> Seattle and Anchorage <-> Portland comprise the University’s connection to Internet2, AWS, and some commodity internet sites.
On April 22nd, one of ACS’ fibers that supply Wide Area Network (WAN) Connectivity (to Internet2, Google, Microsoft and AWS) to the University of Alaska Fairbanks Main Campus began generating faults and errors resulting in lost and delayed packets, thereby impacting internet performance for the vast majority of users at UAF and UA Statewide Fairbanks Information Technology services.
Restore services as soon as possible.
One of the two ACS fibers supplying connectivity from the University of Alaska to Internet2 and the Lower 48 became severely impacted resulting in poor and lost performance. Due to the nature of the issue, it was not immediately clear as to what the exact root of the problem was. An emergency maintenance outage was published for Apr 24th at 1000. During this window the ACS rebooted some network equipment and replaced fiber jumpers, but it did not fix the underlying issue. The location and root cause of the fiber errors still eluded ACS’ ability to isolate and rectify. In order to stabilize services, this fiber was disabled, rerouting all traffic onto the single remaining fiber. This restored UA to proper functional status until fiber repairs could be accomplished. ACS has not yet corrected the issue on the impacted fiber.
This issue was caused by a hardware fault in ACS’ gear, there are not many countermeasures that we can take to prevent this issue from happening again. Short of provider diversification, which has historically been cost-prohibitive.