RIPE NCC Authoritative and Secondary DNS services on Monday 14 December
Dear colleagues, Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day. This was due to an attack on one of the ccTLDs for which the NCC hosts a secondary DNS service. The attack traffic started around 08:00 UTC. RIPE NCC staff applied various countermeasures during the day. These mitigations were effective for some time. However, after implementing each of these mitigations, the traffic patterns were modified to evade them. Towards the end of the day, the volume of the attack traffic targeted at our servers had increased to such a level that it was overloading our incoming links and our mitigation measures were no longer sufficiently effective. At that time we were forced to contact our upstream peers to assist us with mitigation measures. Apart from the ccTLD service for the attacked domain, normal services were restored at around 18:30 UTC. The attack is ongoing, and we continue with mitigation measures in order to provide the best service possible under the circumstances. We note that attacks like this rely on spoofing source addresses in the attack packets. Therefore, Source Address Validation and BCP-38 should be used wherever possible to reduce the ability to abuse networks to transmit spoofed source packets. Kind regards, Romeo Zwart
Thanks for the information Romeo I wonder if perhaps you would consider doing a presentation at the next WG meeting on the issues you encountered and mitigation techniques you used. Thanks Brett -- Brett Carr Senior DNS Engineer Nominet UK
On 15 Dec 2015, at 12:35, Romeo Zwart <romeo.zwart@ripe.net> wrote:
Dear colleagues,
Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day.
This was due to an attack on one of the ccTLDs for which the NCC hosts a secondary DNS service. The attack traffic started around 08:00 UTC. RIPE NCC staff applied various countermeasures during the day. These mitigations were effective for some time. However, after implementing each of these mitigations, the traffic patterns were modified to evade them. Towards the end of the day, the volume of the attack traffic targeted at our servers had increased to such a level that it was overloading our incoming links and our mitigation measures were no longer sufficiently effective.
At that time we were forced to contact our upstream peers to assist us with mitigation measures. Apart from the ccTLD service for the attacked domain, normal services were restored at around 18:30 UTC.
The attack is ongoing, and we continue with mitigation measures in order to provide the best service possible under the circumstances.
We note that attacks like this rely on spoofing source addresses in the attack packets. Therefore, Source Address Validation and BCP-38 should be used wherever possible to reduce the ability to abuse networks to transmit spoofed source packets.
Kind regards, Romeo Zwart
On 15 Dec 2015, at 18:25, Brett Carr <brett.carr@nominet.uk> wrote:
Thanks for the information Romeo I wonder if perhaps you would consider doing a presentation at the next WG meeting on the issues you encountered and mitigation techniques you used.
Thanks
Brett
+1 Cheers, -- Nico
Hi Brett, On 15/12/15 18:25 , Brett Carr wrote:
Thanks for the information Romeo I wonder if perhaps you would consider doing a presentation at the next WG meeting on the issues you encountered and mitigation techniques you used.
We will consider it. As you will understand, and will have noticed in our communication about this, we are trying to balance providing operationally relevant information about the event with a desire to not aid in designing any future events. So the information we give will likely be unsatisfactory for many people in the technical audience we have here. However, we might be able to present more information in a somewhat generalised way that is still useful to the community. As said, we will consider it. Regards, Romeo
Thanks
Brett
-- Brett Carr Senior DNS Engineer Nominet UK
On 15 Dec 2015, at 12:35, Romeo Zwart <romeo.zwart@ripe.net> wrote:
Dear colleagues,
Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day.
This was due to an attack on one of the ccTLDs for which the NCC hosts a secondary DNS service. The attack traffic started around 08:00 UTC. RIPE NCC staff applied various countermeasures during the day. These mitigations were effective for some time. However, after implementing each of these mitigations, the traffic patterns were modified to evade them. Towards the end of the day, the volume of the attack traffic targeted at our servers had increased to such a level that it was overloading our incoming links and our mitigation measures were no longer sufficiently effective.
At that time we were forced to contact our upstream peers to assist us with mitigation measures. Apart from the ccTLD service for the attacked domain, normal services were restored at around 18:30 UTC.
The attack is ongoing, and we continue with mitigation measures in order to provide the best service possible under the circumstances.
We note that attacks like this rely on spoofing source addresses in the attack packets. Therefore, Source Address Validation and BCP-38 should be used wherever possible to reduce the ability to abuse networks to transmit spoofed source packets.
Kind regards, Romeo Zwart
Same question as for the root incident - would you be willing to share more information OTR with software implementors (such as well, me)? Pinky swear that I'm not the perpetrator. Best, Marek On 15 December 2015 at 18:48, Romeo Zwart <romeo.zwart@ripe.net> wrote:
Hi Brett,
On 15/12/15 18:25 , Brett Carr wrote:
Thanks for the information Romeo I wonder if perhaps you would consider doing a presentation at the next WG meeting on the issues you encountered and mitigation techniques you used.
We will consider it. As you will understand, and will have noticed in our communication about this, we are trying to balance providing operationally relevant information about the event with a desire to not aid in designing any future events. So the information we give will likely be unsatisfactory for many people in the technical audience we have here.
However, we might be able to present more information in a somewhat generalised way that is still useful to the community. As said, we will consider it.
Regards, Romeo
Thanks
Brett
-- Brett Carr Senior DNS Engineer Nominet UK
On 15 Dec 2015, at 12:35, Romeo Zwart <romeo.zwart@ripe.net> wrote:
Dear colleagues,
Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day.
This was due to an attack on one of the ccTLDs for which the NCC hosts a secondary DNS service. The attack traffic started around 08:00 UTC. RIPE NCC staff applied various countermeasures during the day. These mitigations were effective for some time. However, after implementing each of these mitigations, the traffic patterns were modified to evade them. Towards the end of the day, the volume of the attack traffic targeted at our servers had increased to such a level that it was overloading our incoming links and our mitigation measures were no longer sufficiently effective.
At that time we were forced to contact our upstream peers to assist us with mitigation measures. Apart from the ccTLD service for the attacked domain, normal services were restored at around 18:30 UTC.
The attack is ongoing, and we continue with mitigation measures in order to provide the best service possible under the circumstances.
We note that attacks like this rely on spoofing source addresses in the attack packets. Therefore, Source Address Validation and BCP-38 should be used wherever possible to reduce the ability to abuse networks to transmit spoofed source packets.
Kind regards, Romeo Zwart
Hi Romeo, Perhaps you can share more details in the member only part of the next DNS-OARC session? Jacques
-----Original Message----- From: dns-wg [mailto:dns-wg-bounces@ripe.net] On Behalf Of Romeo Zwart Sent: December-15-15 12:48 PM To: Brett Carr Cc: RIPE DNS Working Group Subject: Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December
Hi Brett,
On 15/12/15 18:25 , Brett Carr wrote:
Thanks for the information Romeo I wonder if perhaps you would consider doing a presentation at the next WG meeting on the issues you encountered and mitigation techniques you used.
We will consider it. As you will understand, and will have noticed in our communication about this, we are trying to balance providing operationally relevant information about the event with a desire to not aid in designing any future events. So the information we give will likely be unsatisfactory for many people in the technical audience we have here.
However, we might be able to present more information in a somewhat generalised way that is still useful to the community. As said, we will consider it.
Regards, Romeo
Thanks
Brett
-- Brett Carr Senior DNS Engineer Nominet UK
On 15 Dec 2015, at 12:35, Romeo Zwart <romeo.zwart@ripe.net> wrote:
Dear colleagues,
Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day.
This was due to an attack on one of the ccTLDs for which the NCC hosts a secondary DNS service. The attack traffic started around 08:00 UTC. RIPE NCC staff applied various countermeasures during the day. These mitigations were effective for some time. However, after implementing each of these mitigations, the traffic patterns were modified to evade them. Towards the end of the day, the volume of the attack traffic targeted at our servers had increased to such a level that it was overloading our incoming links and our mitigation measures were no longer sufficiently effective.
At that time we were forced to contact our upstream peers to assist us with mitigation measures. Apart from the ccTLD service for the attacked domain, normal services were restored at around 18:30 UTC.
The attack is ongoing, and we continue with mitigation measures in order to provide the best service possible under the circumstances.
We note that attacks like this rely on spoofing source addresses in the attack packets. Therefore, Source Address Validation and BCP-38 should be used wherever possible to reduce the ability to abuse networks to transmit spoofed source packets.
Kind regards, Romeo Zwart
Romeo Zwart writes:
Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day.
etc.
According a message from Stephane Bortzmeyer "The RIPE name server was retired on 16 december, for unknown reasons (as far as I know, the RIPE-NCC did not communicate on that)." Can you comment on that? Thanks, jaap
Hi Jaap, On 15/12/29 13:08 , Jaap Akkerhuis wrote:
Romeo Zwart writes:
Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day.
etc.
According a message from Stephane Bortzmeyer
"The RIPE name server was retired on 16 december, for unknown reasons (as far as I know, the RIPE-NCC did not communicate on that)."
Can you comment on that?
With this limited amount of information, that would be hard. Which zones are we talking about and what does 'retired' mean in this context? I haven't seen Stephane's message (yet). Was that a private message or sent to a mailing list? Can you forward the whole message or have Stephane provide more detail about his observations directly to me? Thanks, Romeo
Thanks,
jaap
Romeo Zwart writes:
Hi Jaap,
On 15/12/29 13:08 , Jaap Akkerhuis wrote:
Romeo Zwart writes:
Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day.
etc.
According a message from Stephane Bortzmeyer
"The RIPE name server was retired on 16 december, for unknown reasons (as far as I know, the RIPE-NCC did not communicate on that)."
Can you comment on that?
With this limited amount of information, that would be hard. Which zones are we talking about and what does 'retired' mean in this context?
I haven't seen Stephane's message (yet). Was that a private message or sent to a mailing list? Can you forward the whole message or have Stephane provide more detail about his observations directly to me?
It seems that I have indeed removed to much of the context. Stephane's message was on the centr security list which archives seem to be sealed (contrary to what I thought). It was refering to the attack on the .tr name servers about which you reported in <https://www.ripe.net/ripe/mail/archives/dns-wg/2015-December/003184.html> that it had impacted RIPE's DNS service. Apparently Stephan wanted to know why RIPE NCC dropped serving the .tr zone. (My guess, since de RIPE NCC dropped out of the root zone as well, it was done in coordination with the tr people). So I was just curious wat happened on RIPE's end. jaap
Hi Jaap,
On 29 dec. 2015, at 19:58, Jaap Akkerhuis <jaap@NLnetLabs.nl> wrote:
Romeo Zwart writes:
Hi Jaap,
On 15/12/29 13:08 , Jaap Akkerhuis wrote: Romeo Zwart writes:
Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services were functioning in a severely degraded state during parts of the day.
etc.
According a message from Stephane Bortzmeyer
"The RIPE name server was retired on 16 december, for unknown reasons (as far as I know, the RIPE-NCC did not communicate on that)."
Can you comment on that?
With this limited amount of information, that would be hard. Which zones are we talking about and what does 'retired' mean in this context?
I haven't seen Stephane's message (yet). Was that a private message or sent to a mailing list? Can you forward the whole message or have Stephane provide more detail about his observations directly to me?
It seems that I have indeed removed to much of the context.
Stephane's message was on the centr security list which archives seem to be sealed (contrary to what I thought). It was refering to the attack on the .tr name servers about which you reported in <https://www.ripe.net/ripe/mail/archives/dns-wg/2015-December/003184.html> that it had impacted RIPE's DNS service.
Ah ok, some context helps. :)
Apparently Stephan wanted to know why RIPE NCC dropped serving the .tr zone. (My guess, since de RIPE NCC dropped out of the root zone as well, it was done in coordination with the tr people).
Indeed it was.
So I was just curious wat happened on RIPE's end.
We can share some more detail next week. Kind regards, Romeo
jaap
Dear Jaap and colleagues, On 29 December you wrote to the list:
Stephane's message was on the centr security list which archives seem to be sealed (contrary to what I thought). It was refering to the attack on the .tr name servers about which you reported in <https://www.ripe.net/ripe/mail/archives/dns-wg/2015-December/003184.html> that it had impacted RIPE's DNS service. Apparently Stephan wanted to know why RIPE NCC dropped serving the .tr zone. (My guess, since de RIPE NCC dropped out of the root zone as well, it was done in coordination with the tr people).
So I was just curious wat happened on RIPE's end.
In the incident report you reference above, I did not mention the .TR zone explicitly, which apparently led to unnecessary confusion and an undesired atmosphere of secrecy around the incident. I did mention in the same message that, after applying various mitigation measures during the day, we turned to our upstreams to assist us with mitigation in the late afternoon of Monday 14th. In practice this meant we asked for upstream blackholing of the attack traffic, which effectively meant we were no longer serving the .TR zone. While the event was ongoing, we were of course communicating with the .TR staff frequently. On Tuesday morning, 15 December, the .TR staff informed us that they removed the RIPE NCC secondary server from the .TR zone altogether. I hope this clarifies matters sufficiently. If you have more questions please feel free to ask. I should add, however, that we do not intend to share more details about the attack itself, or the mitigation applied, on this list. An observation that we have made during the past months is that the impact of attacks upon our DNS infrastructure is increasing. This seems to be a more general trend that readers on this list are likely to be aware of, but this may not be the case for the community at large. For the RIPE NCC this means that we are investigating the options to increase the capacity and robustness of our DNS services further. Kind regards, Romeo Zwart
participants (6)
-
Brett Carr -
Jaap Akkerhuis -
Jacques Latour -
Marek Vavruša -
Nico CARTRON -
Romeo Zwart