We have restored the routing configuration of the affected DNS instances and restored traffic to them at 09:00 UTC today.
Posted Apr 16, 2024 - 09:31 UTC
Monitoring
At 09:15 UTC today an unscheduled software update was rolled out on several of our Hosted DNS instances. This update applied an incorrect routing daemon configuration. Due to the incorrect configuration, DNS requests received by these DNS instances could no longer be answered. This affected 24 out of 119 K-root sites and 8 of 20 Authoritative DNS sites.
At 14:03 UTC the cause of the incident was identified, and anycast BGP announcements for the affected instances were withdrawn, redirecting traffic to other locations. This was completed at 14:15 UTC.
We are currently restoring the correct routing configurations on all affected servers, and will restore traffic to them once this has been completed. Furthermore, we will be applying measures to prevent automatic updates of the routing daemon software on all of our DNS servers.