Dear colleagues,
Yesterday at 09:15 UTC an unscheduled software update was rolled out on
several of our Hosted DNS instances. This update applied an incorrect
routing daemon configuration. Due to the incorrect configuration, DNS
requests received by these DNS instances could no longer be answered.
This affected 8 of our 20 Authoritative DNS sites.
At 14:03 UTC the cause of the incident was identified, and anycast BGP
announcements for the affected instances were withdrawn, redirecting
traffic to other locations. This was completed at 14:15 UTC.
We re-applied the correct routing configurations on all affected
servers, and restored BGP announcements at 09:00 UTC today. Furthermore,
we have applied measures to prevent automatic updates of the routing
daemon software on all of our DNS servers.
Kind regards,
Martin Pels
RIPE NCC