Monitoring systems alerted us of link failures between the Santa Clara and Fremont Site, similar to the 2023-02-12 Incident, however unlike the events of 2-12 this presented as a total loss of light across the B-Side span, and at the time of the failure no issue on the A-side was present. Once again the network team engaged the fiber provider and investigations began. At approximately 14:50 UTC, the fiber provider identified this as a fiber cut. At 16:05 UTC, repair work began to splice and restore the fiber, 18:50 UTC the fiber provider reported splicing complete and service to the circuit was restored.
Unfortunately while the B-side was down, we experienced a few very brief hits to the A-side fiber causing it to flap. These hits ranged from sub-second to under 30 seconds. We observed a total of 4 incidents, 06:20, 07:22, 08:28, 11:15 UTC. The fiber provider of the A-side (different provider than the B-side) was engaged to investigate. Internally, all physical segments, equipment and logs across this circuit were investigated, however, outside of the physical interface flaps no additional information was identified, and more than likely was an environmental factor outside our control.
Fiber cuts like the one on February 21st are not uncommon, and can be caused by environmental factors outside our control. In almost all cases these issues are identified, and repaired with no impact to the customer. Unfortunately what is uncommon is for the redundant path (or any fiber circuit) to take sub-second hits as described above.
While the proximity in time of these two events makes it hard to believe that they are not related in some way, the fact of the matter is that it’s a coincidence to experience two failures like this in such a short time span. The failures of the B-side fiber were completely different in nature.
However, as a result, we compelled them to move our circuit to a completely new path and pair.
At this time both circuits are considered stable and we do not expect a recurrence like this to occur in the future.