All-Probe Traceroute + detect RFC1918 addresses
Hi Folks, Has anybody ever run a all-probe traceroute and then to detect any RFC1918 addresses in there? (though many probes will have locally some RFC1918) We got CAIDAs spoofer project, but that primarily afaik checks that by doing connections, not by checking ICMP returns. I just saw towards 213.244.71.2 : 11 Bundle-Ether42.br03.mrs01.pccwbtn.net (63.223.38.78) 29.068 ms 29.301 ms 29.129 ms 12 Bundle-Ether41.br03.mrs01.pccwbtn.net (63.223.38.74) 31.462 ms 31.410 ms 31.459 ms 13 10.74.42.10 (10.74.42.10) 77.574 ms 63.222.97.82 (63.222.97.82) 73.651 ms 63.222.97.90 (63.222.97.90) 73.514 ms 14 10.74.42.129 (10.74.42.129) 82.789 ms * 10.74.19.29 (10.74.19.29) 78.695 ms 15 * * 10.74.25.22 (10.74.25.22) 78.914 ms 16 * * 10.74.25.22 (10.74.25.22) 78.875 ms 17 * * * Which means the whole path till that IP was not doing any kind of RPF.... thus spoofing anything else would be possible too. At least one could kick PCCW in this case... but likely there are others. And as we are in 2021... a hall of shame might be appropriate... Of course, one should also do that for IPv6; though I expect outside the stray ULA address (thank you apple; though they are fixing that ULA issue with homepods apparently) very little of it, though "meten is weten" (measuring is knowing). Greets, Jeroen
On 2021/09/15 17:32 , Jeroen Massar via ripe-atlas wrote:
Has anybody ever run a all-probe traceroute and then to detect any RFC1918 addresses in there? (though many probes will have locally some RFC1918)
Atlas performs quite a few traceroutes on all probes. For example, https://atlas.ripe.net/measurements/5017/ which is a unicast destination. So anything close to the probes should show up.
Of course, one should also do that for IPv6; though I expect outside the stray ULA address (thank you apple; though they are fixing that ULA issue with homepods apparently) very little of it, though "meten is weten" (measuring is knowing).
My router at home gets a lot of error icmps with link local source, from quite a few hops away...
Hi, On Wed, Sep 15, 2021 at 06:58:50PM +0200, Philip Homburg wrote:
My router at home gets a lot of error icmps with link local source, from quite a few hops away...
*wake up* IPv4 "link local" (169.254.*) or IPv6 LLA (fe80::)? The latter would make me very curious what sort of intermediate devices are forwarding these (which is strictly forbidden). Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
On 2021/09/15 19:06 , Gert Doering wrote:
On Wed, Sep 15, 2021 at 06:58:50PM +0200, Philip Homburg wrote:
My router at home gets a lot of error icmps with link local source, from quite a few hops away...
*wake up* IPv4 "link local" (169.254.*) or IPv6 LLA (fe80::)?
IPv6 link local
The latter would make me very curious what sort of intermediate devices are forwarding these (which is strictly forbidden).
As far as I know, this a well known Juniper bug where their routers forward errors ICMPs without checking whether the source address is link local. I don't know what happens to other packets with link local source. Philip
Philip Homburg <philip.homburg@ripe.net> writes:
As far as I know, this a well known Juniper bug where their routers forward errors ICMPs without checking whether the source address is link local.
Thinking about this... Is there any reason except formalities why you shouldn't forward those packets? The won't generate a reply and can't generate an ICMP error, so their source address will never be used as a destination. Agreed, it would be better if they had a global source. But they don't. Maybe because there was none configured on the router/host the error message originates from? In any case, the source address is what it is and you can either forward the packet or drop it. Dropping it means the information is lost. Maybe breaking PMTU or whatever. I believe it's better to ignore the formalities here and forward those packets. It's certainly harmless. At least as harmless as forwarding any other ICMP error messages. Bjørn
On 20210915, at 21:59, Bjørn Mork <bjorn@mork.no> wrote:
Philip Homburg <philip.homburg@ripe.net> writes:
As far as I know, this a well known Juniper bug where their routers forward errors ICMPs without checking whether the source address is link local.
Thinking about this...
If you can't verify the source, which with LL you cannot as they are on every interface around the world, it is spoofable. Do you really want to receive a fe80::/10 at your recursive DNS service as a request (which could be valid, locally). fe80::/10 should never have a TTL other than 1... it is link-local. The whole point of the thread is to find networks that allow non-routed addresses, the standard BCP38 trick. Detecting RFC1918 in traceroutes might just be a cheap-ish way to identify these kind of networks (especially when outbound NAT happens, thus spoofing gets killed). Greets, Jeroen
Jeroen Massar <jeroen@massar.ch> writes:
If you can't verify the source, which with LL you cannot as they are on every interface around the world, it is spoofable.
Yes.
Do you really want to receive a fe80::/10 at your recursive DNS service as a request (which could be valid, locally).
If you allow fe80::/10 at your recursive DNS service then you will make sure this prefix cannot be spoofed. This is a policy for the next-hop router seen from the DNS server (if there is any - I'm not sure why I'd allow fe80::/10 on any link with a router). But the point is that this policy does not need to affect any other router on the Internet. And I would most certainly not depend on the configuration of other routers outside my control for anything security related. Not that different from the RPF you will do in your own network to protect agaist spoofing the other prefixes allowed at your recursive DNS service. It's nice if people do RPF elsewhere too, but not something I'd depend on for my DNS service. I still do not see any problems letting the DNS server receive "illegal" ICMP errors though.
fe80::/10 should never have a TTL other than 1... it is link-local.
Maybe so. But that's not enforced anywhere. Many common IP applications can use LL addresses with the appropriate zone identifier, but will not change the header to adapt to any LL restrictions. And the same goes for applications implemented in the kernel. if you ping ff02::1 on a link with some hosts, then you'll probably see a number of replies from LL sources with ttl=64.
The whole point of the thread is to find networks that allow non-routed addresses, the standard BCP38 trick. Detecting RFC1918 in traceroutes might just be a cheap-ish way to identify these kind of networks (especially when outbound NAT happens, thus spoofing gets killed).
Yes, I see the value in that test and it does sound like a reasonable approach. It was just the notion that there should be any problem forwarding ICMP errors with "illegal" source addresses that caught my attention. I don't think there is, regardless of the reason the address is defined as illegal. As long as the Juniper bug is limited to ICMP errors, then I am willing to accept it as a standard-stretching feature. And if they document the bug, then it is a feature by definition :-) Bjørn
On 2021/09/15 21:59 , Bjørn Mork wrote:
I believe it's better to ignore the formalities here and forward those packets. It's certainly harmless. At least as harmless as forwarding any other ICMP error messages.
In my opinion this is the wrong approach. If there is a good reason to violate the standard, then the thing to do is to bring up in the IETF. Philip
Philip Homburg <philip.homburg@ripe.net> writes:
On 2021/09/15 21:59 , Bjørn Mork wrote:
I believe it's better to ignore the formalities here and forward those packets. It's certainly harmless. At least as harmless as forwarding any other ICMP error messages.
In my opinion this is the wrong approach. If there is a good reason to violate the standard, then the thing to do is to bring up in the IETF.
You are of course correct. Let's hope someone else cares enough to do that :-) An additional argument for allowing the ICMP errors, and only those, is that you can't send back any dest unreachable with "Beyond scope of source address". Bjørn
On 9/15/21 11:32 AM, Jeroen Massar via ripe-atlas wrote:
Hi Folks,
Has anybody ever run a all-probe traceroute and then to detect any RFC1918 addresses in there? (though many probes will have locally some RFC1918)
Since probes are running measurements to many targets already, the full dataset will uncover a lot without having to run more measurements. A quick query: https://gist.github.com/sdstrowes/e9d4a3c7c03dd1aafa3198333cc39ffa Out of ~106M IPv4 traceroutes, this finds ~6M that contain 10.0.0.0/8 in an ICMP response more than 4 hops from the origin. That's not the smartest approach, but it's a good ballpark of what's in the data. It'd be reasonably easy to take that and whittle it down to a set of probes and/or probe ASNs that see this. With more work it'd be possible to identify ASNs on the forward path as a strong hint (asymmetric routing to one side) of where these pass through. S.
We got CAIDAs spoofer project, but that primarily afaik checks that by doing connections, not by checking ICMP returns.
I just saw towards 213.244.71.2 :
11 Bundle-Ether42.br03.mrs01.pccwbtn.net (63.223.38.78) 29.068 ms 29.301 ms 29.129 ms 12 Bundle-Ether41.br03.mrs01.pccwbtn.net (63.223.38.74) 31.462 ms 31.410 ms 31.459 ms 13 10.74.42.10 (10.74.42.10) 77.574 ms 63.222.97.82 (63.222.97.82) 73.651 ms 63.222.97.90 (63.222.97.90) 73.514 ms 14 10.74.42.129 (10.74.42.129) 82.789 ms * 10.74.19.29 (10.74.19.29) 78.695 ms 15 * * 10.74.25.22 (10.74.25.22) 78.914 ms 16 * * 10.74.25.22 (10.74.25.22) 78.875 ms 17 * * *
Which means the whole path till that IP was not doing any kind of RPF.... thus spoofing anything else would be possible too.
At least one could kick PCCW in this case... but likely there are others.
And as we are in 2021... a hall of shame might be appropriate...
Of course, one should also do that for IPv6; though I expect outside the stray ULA address (thank you apple; though they are fixing that ULA issue with homepods apparently) very little of it, though "meten is weten" (measuring is knowing).
Greets, Jeroen
On 20210915, at 19:25, Stephen Strowes <s@sdstrowes.co.uk> wrote:
On 9/15/21 11:32 AM, Jeroen Massar via ripe-atlas wrote:
Hi Folks,
Has anybody ever run a all-probe traceroute and then to detect any RFC1918 addresses in there? (though many probes will have locally some RFC1918)
Since probes are running measurements to many targets already, the full dataset will uncover a lot without having to run more measurements.
A quick query: https://gist.github.com/sdstrowes/e9d4a3c7c03dd1aafa3198333cc39ffa
Out of ~106M IPv4 traceroutes, this finds ~6M that contain 10.0.0.0/8 in an ICMP response more than 4 hops from the origin. That's not the smartest approach, but it's a good ballpark of what's in the data.
It'd be reasonably easy to take that and whittle it down to a set of probes and/or probe ASNs that see this. With more work it'd be possible to identify ASNs on the forward path as a strong hint (asymmetric routing to one side) of where these pass through.
Good one. Indeed, if one can go through the existing traceroute data, one would have the possibility to detect these. Any way we could automate this into a nice warning page along with the probability that an ASN is the cause of passing on RFC1918 (and thus likely not filtering at all)? Greets, Jeroen
Hi Jeroen,
We got CAIDAs spoofer project, but that primarily afaik checks that by doing connections, not by checking ICMP returns.
I just saw towards 213.244.71.2 : [...] Which means the whole path till that IP was not doing any kind of RPF.... thus spoofing anything else would be possible too.
Maybe I have a wrong imagination on how this spoofing testing is done, but... If you spoof the source address you simply can't get any return. Your traceroute proves, that your packet leaves the network, but it's not proof for, that BCP38 was not implemented. Because IF your packets were forwarded with spoofed source, you won't got any response, not even from the first hop. Obviously, because your false source IP would get the ICMP returns, not your probe. So, IMHO, it's more likely to me your traceroute is proof for a NAT and not for spoofing. Greetings, Max
On 20210916, at 16:28, Max Grobecker <max.grobecker@ml.grobecker.info> wrote:
Hi Jeroen,
We got CAIDAs spoofer project, but that primarily afaik checks that by doing connections, not by checking ICMP returns.
I just saw towards 213.244.71.2 : [...] Which means the whole path till that IP was not doing any kind of RPF.... thus spoofing anything else would be possible too.
Maybe I have a wrong imagination on how this spoofing testing is done, but... If you spoof the source address you simply can't get any return. Your traceroute proves, that your packet leaves the network, but it's not proof for, that BCP38 was not implemented.
A traceroute with a spoofed address indeed would not return anything, as you are not the source. This is about doing a traceroute, from whatever your host/probe is configured for, and thus also means that a NAT would change that source address. For instance a result (towards the above IP) could look like; ... 11 Bundle-Ether41.br03.mrs01.pccwbtn.net (63.223.38.74) 31.672 ms 31.720 ms Bundle-Ether42.br03.mrs01.pccwbtn.net (63.223.38.78) 29.302 ms 12 Bundle-Ether41.br03.mrs01.pccwbtn.net (63.223.38.74) 31.549 ms 31.094 ms 31.863 ms 13 63.222.97.82 (63.222.97.82) 76.494 ms 10.74.42.53 (10.74.42.53) 81.986 ms 63.222.97.90 (63.222.97.90) 77.033 ms 14 * * * 15 * * 10.74.19.109 (10.74.19.109) 87.029 ms 16 * * * which shows that hop 15 was able to respond with RFC1918, all the hops upto that point are thus not bothering to check if the source was allowed to send; I can tell you RFC1918 should never be visible there... Yes, the ISP that allowed this traceroute is very aware of this problem, at least they filter prefixes at the edge, thus hosts cannot spoof, but their transit/peering links they can spoof what they want. Which leads to them having big issues when a spoofed DNS amp DDoS comes in as they have little clue where it comes from, thus they understand they need to fix this..... but likely many places, while one can convert this to a dollar amount lost (when a DDoS happens), there are other more urgent things for them.
Because IF your packets were forwarded with spoofed source, you won't got any response, not even from the first hop.
Correct, but this is more about recognizing where filtering does not happen, as that indicates that other paths over that route can also spoof if their edges do not filter at source.
Obviously, because your false source IP would get the ICMP returns, not your probe.
So, IMHO, it's more likely to me your traceroute is proof for a NAT and not for spoofing.
If one sees RFC1918 in a traceroute (especially >5 hops away, thus just not the client->CPE hops), it indicates that every hop in the middle is not filtering at least RFC1918; more likely they are thus just doing any kind of reverse prefix filtering aka the largest part of BCP38. Greets, Jeroen
Jeroen Massar via ripe-atlas <ripe-atlas@ripe.net> writes:
If one sees RFC1918 in a traceroute (especially >5 hops away, thus just not the client->CPE hops), it indicates that every hop in the middle is not filtering at least RFC1918; more likely they are thus just doing any kind of reverse prefix filtering aka the largest part of BCP38.
To be difficult again.... But I don't think you can make that assumption unless you are able to detect that the ICMP errors cross some network border. Using RFC1918 on links in your own network is fine. And having them show up in traceroutes is a feature. To illustrate, this a traceroute from a standard mobile netowrk access to one of the DNS resolvers used from that access: bjorn@miraculix:~$ traceroute -e 130.67.15.198 traceroute to 130.67.15.198 (130.67.15.198), 30 hops max, 60 byte packets 1 77.16.1.8.tmi.telenormobil.no (77.16.1.8) 435.127 ms 644.941 ms 644.890 ms 2 ti0006c360-ae17-0.ti.telenor.net (146.172.18.85) <MPLS:L=18694,E=2,S=0,T=1/L=26,E=2,S=1,T=1> 653.783 ms 653.735 ms 653.686 ms 3 ti0300c360-ae4-0.ti.telenor.net (146.172.23.174) <MPLS:L=844,E=2,S=0,T=1/L=26,E=2,S=1,T=2> 653.638 ms 653.592 ms 653.543 ms 4 10.67.115.189 (10.67.115.189) 653.481 ms 653.432 ms 653.383 ms 5 * * * 6 ti0001a401-ae18-21.ti.telenor.net (213.142.76.153) 653.270 ms 206.020 ms 32.959 ms 7 ti0275c360-ae49-0.ti.telenor.net (146.172.22.98) <MPLS:L=8866,E=0,S=1,T=1> 33.781 ms 32.122 ms 35.930 ms 8 ti0275a400-ae1-0.ti.telenor.net (146.172.101.94) 37.812 ms 40.701 ms 40.666 ms 9 ns11.e.nsc.no (130.67.15.198) 40.628 ms 40.591 ms 40.555 ms So you can see the RFC1918 address on one of the links between the VPN with the PGW and the Internet. So what? It's still one ISP. And much more useful info than hop number 5. Bjørn
Hi, On Fri, Sep 17, 2021 at 03:23:24PM +0200, Bjørn Mork wrote:
Using RFC1918 on links in your own network is fine. And having them show up in traceroutes is a feature.
s/feature/sign of sloppy network design, and a RFC1918 violation/ Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Gert Doering <gert@space.net> writes:
On Fri, Sep 17, 2021 at 03:23:24PM +0200, Bjørn Mork wrote:
Using RFC1918 on links in your own network is fine. And having them show up in traceroutes is a feature.
s/feature/sign of sloppy network design, and a RFC1918 violation/
Sorry for being slow. but I need a direct reference to the paragraph this violates. Not that I'm against violating some parts of RFC1918 where required. The part disallowing leakage of RFC1918 into DNS is too impractical and unjustified. Bjørn
HBi, On Fri, Sep 17, 2021 at 03:43:29PM +0200, Bjørn Mork wrote:
Gert Doering <gert@space.net> writes:
On Fri, Sep 17, 2021 at 03:23:24PM +0200, Bjørn Mork wrote:
Using RFC1918 on links in your own network is fine. And having them show up in traceroutes is a feature.
s/feature/sign of sloppy network design, and a RFC1918 violation/
Sorry for being slow. but I need a direct reference to the paragraph this violates.
Section 5: It is strongly recommended that routers which connect enterprises to external networks are set up with appropriate packet and routing filters at both ends of the link in order to prevent packet and routing information leakage. I think that speaks very clearly about "you can do in your network whatever you want, but nobody else wants to see that" Given the age of the document, the language used to be less STRONG back then. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Gert Doering <gert@space.net> writes:
On Fri, Sep 17, 2021 at 03:43:29PM +0200, Bjørn Mork wrote:
Gert Doering <gert@space.net> writes:
On Fri, Sep 17, 2021 at 03:23:24PM +0200, Bjørn Mork wrote:
Using RFC1918 on links in your own network is fine. And having them show up in traceroutes is a feature.
s/feature/sign of sloppy network design, and a RFC1918 violation/
Sorry for being slow. but I need a direct reference to the paragraph this violates.
Section 5:
It is strongly recommended that routers which connect enterprises to external networks are set up with appropriate packet and routing filters at both ends of the link in order to prevent packet and routing information leakage.
I think that speaks very clearly about "you can do in your network whatever you want, but nobody else wants to see that"
This fails to consider the situation where you are using RFC1918 addresses on that link, which is common for mobile network access today. My example didn't make that clear, but the traceroute probes are sent from an RFC1918 address: bjorn@miraculix:~$ ip route get 130.67.15.198 130.67.15.198 dev wwan0 src 10.82.241.88 uid 1000 cache So you should drop packets using RFC1918 addresses on that link?
Given the age of the document, the language used to be less STRONG back then.
Sure. Assigning RFC1918 addresses to customers was also unheard of, and didn't even need to be mentioned. Bjørn
Hi, On Fri, Sep 17, 2021 at 04:17:47PM +0200, Bjørn Mork wrote:
Section 5:
It is strongly recommended that routers which connect enterprises to external networks are set up with appropriate packet and routing filters at both ends of the link in order to prevent packet and routing information leakage.
I think that speaks very clearly about "you can do in your network whatever you want, but nobody else wants to see that"
This fails to consider the situation where you are using RFC1918 addresses on that link, which is common for mobile network access today.
It doesn't. It is very clear that *if* you do, it's your responsibility to ensure ICMP packets are not sent from a RFC1918 address. This is not a fault in the RFC, it's a fault in the way these people build their networks.
My example didn't make that clear, but the traceroute probes are sent from an RFC1918 address:
bjorn@miraculix:~$ ip route get 130.67.15.198 130.67.15.198 dev wwan0 src 10.82.241.88 uid 1000 cache
So you should drop packets using RFC1918 addresses on that link?
"What happens inside your network happens inside your network" (and the RFC explicitly permits that, of course), but we do not want to see it on someone else's network.
Given the age of the document, the language used to be less STRONG back then.
Sure. Assigning RFC1918 addresses to customers was also unheard of, and didn't even need to be mentioned.
If that is CGN'ed, it's not violating the RFC. Leaking packets from addresse that do not belong to you does. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Gert Doering <gert@space.net> writes:
"What happens inside your network happens inside your network" (and the RFC explicitly permits that, of course), but we do not want to see it on someone else's network.
Exactly my point. My traceroute example is all in "my" network if you include the mobile access endpoint. The usage of RFC1918 for PGW pools and links is co-ordinated. There is no reason you should not see RFC1918 adresses as source here.
Given the age of the document, the language used to be less STRONG back then.
Sure. Assigning RFC1918 addresses to customers was also unheard of, and didn't even need to be mentioned.
If that is CGN'ed, it's not violating the RFC.
Of course it's not. But you'll have to define the customer endpoint as part of the RFC1918 "enterprise" network.
Leaking packets from addresse that do not belong to you does.
Yes. And the point is that you cannot tell if there is a leakage unless you are able to detect the network borders. Which you can't by counting traceroute hops. Bjørn
On 20210917, at 21:20, Bjørn Mork <bjorn@mork.no> wrote: [..]
Leaking packets from addresse that do not belong to you does.
Yes. And the point is that you cannot tell if there is a leakage unless you are able to detect the network borders. Which you can't by counting traceroute hops.
In the example I gave it is all quite obvious actually... as there are multiple ASNs in the path and then RFC1918... Noting again, that seeing RFC1918 a few hops away is a clear indicator that there is likely a possibility to spoof as they don't even bother to filter the obvious. Greets, Jeroen
participants (6)
-
Bjørn Mork -
Gert Doering -
Jeroen Massar -
Max Grobecker -
Philip Homburg -
Stephen Strowes