Authoritative DNS service degradation 2024-04-15
Dear colleagues, Yesterday at 09:15 UTC an unscheduled software update was rolled out on several of our Hosted DNS instances. This update applied an incorrect routing daemon configuration. Due to the incorrect configuration, DNS requests received by these DNS instances could no longer be answered. This affected 8 of our 20 Authoritative DNS sites. At 14:03 UTC the cause of the incident was identified, and anycast BGP announcements for the affected instances were withdrawn, redirecting traffic to other locations. This was completed at 14:15 UTC. We re-applied the correct routing configurations on all affected servers, and restored BGP announcements at 09:00 UTC today. Furthermore, we have applied measures to prevent automatic updates of the routing daemon software on all of our DNS servers. Kind regards, Martin Pels RIPE NCC
Hi Martin, Thanks for the information. One site (out of 2) of mine was affected while the other one was working fine (at least the graphs are saying so 😊). Regards, Gaurav Kansal
On 16 Apr 2024, at 15:14, mpels@ripe.net wrote:
Dear colleagues,
Yesterday at 09:15 UTC an unscheduled software update was rolled out on several of our Hosted DNS instances. This update applied an incorrect routing daemon configuration. Due to the incorrect configuration, DNS requests received by these DNS instances could no longer be answered. This affected 8 of our 20 Authoritative DNS sites.
At 14:03 UTC the cause of the incident was identified, and anycast BGP announcements for the affected instances were withdrawn, redirecting traffic to other locations. This was completed at 14:15 UTC.
We re-applied the correct routing configurations on all affected servers, and restored BGP announcements at 09:00 UTC today. Furthermore, we have applied measures to prevent automatic updates of the routing daemon software on all of our DNS servers.
Kind regards, Martin Pels RIPE NCC
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/dns-wg
participants (2)
-
Gaurav Kansal
-
Martin Pels (RIPE NCC)