Tagging probes which are using the ISP's DNS server?
Hi, a few days ago I wanted to debug a name resolution problem of one of our domains. For this reason, I wanted to test if probes inside a specific ASN are having difficulties to resolve a specific name (because only customers of this ISP were complaining). This lead to very mixed results, mostly because some of the selected probes did queries to a public DNS service like Google, Quad9 and so on. The problem existed only with the provider's DNS servers for some reason. It did take some time to make a script which tried to filter out these probes, so I wondered if anyone else had the same use-case and problem. Is there a way to automatically tag probes, which are (seemingly) using the ISP's own DNS servers, or, at least, not a well-known public service? This could be done maybe by querying a special DNS name which returns the IP address from where the query was received (like "whoami.akamai.net"). By comparing the ASN of the probe and the ASN of the IP address returned by the DNS query, one could determine, if the ISP's servers are used. This would also be true for people running their own recursor, but this could be filtered as well very easy. If an ISP is using multiple ASN, this could be a problem. Maybe there's an easy solution for this as well. Probes which pass this test, could then be tagged with "DNS-using-ISP-server" or something like that and explicitly be selected for specific DNS resolution tests. Greetings, Max
On this topic: Is there an Atlas-preferred setting for this? Should the probe I host ideally use the ISP's DNS, or ideally a DNS from a global player? Or ideally an academic/non-profit DNS? BR Daniel AJ On 10/5/22 18:38, Max Grobecker wrote:
Hi,
a few days ago I wanted to debug a name resolution problem of one of our domains. For this reason, I wanted to test if probes inside a specific ASN are having difficulties to resolve a specific name (because only customers of this ISP were complaining). This lead to very mixed results, mostly because some of the selected probes did queries to a public DNS service like Google, Quad9 and so on. The problem existed only with the provider's DNS servers for some reason.
It did take some time to make a script which tried to filter out these probes, so I wondered if anyone else had the same use-case and problem. Is there a way to automatically tag probes, which are (seemingly) using the ISP's own DNS servers, or, at least, not a well-known public service? This could be done maybe by querying a special DNS name which returns the IP address from where the query was received (like "whoami.akamai.net"). By comparing the ASN of the probe and the ASN of the IP address returned by the DNS query, one could determine, if the ISP's servers are used. This would also be true for people running their own recursor, but this could be filtered as well very easy. If an ISP is using multiple ASN, this could be a problem. Maybe there's an easy solution for this as well.
Probes which pass this test, could then be tagged with "DNS-using-ISP-server" or something like that and explicitly be selected for specific DNS resolution tests.
Greetings, Max
Hi Daniel, On 06/10/2022 07:41, Daniel AJ Sokolov wrote:
On this topic: Is there an Atlas-preferred setting for this? Should the probe I host ideally use the ISP's DNS, or ideally a DNS from a global player? Or ideally an academic/non-profit DNS?
Although you are completely free to choose whichever DNS server you think is best for your probe, we generally advise to use the same one that you assign to the rest of the network that the probe is connected to. Cheers, Johan ter Beest RIPE Atlas Team
BR Daniel AJ
On 10/5/22 18:38, Max Grobecker wrote:
Hi,
a few days ago I wanted to debug a name resolution problem of one of our domains. For this reason, I wanted to test if probes inside a specific ASN are having difficulties to resolve a specific name (because only customers of this ISP were complaining). This lead to very mixed results, mostly because some of the selected probes did queries to a public DNS service like Google, Quad9 and so on. The problem existed only with the provider's DNS servers for some reason.
It did take some time to make a script which tried to filter out these probes, so I wondered if anyone else had the same use-case and problem. Is there a way to automatically tag probes, which are (seemingly) using the ISP's own DNS servers, or, at least, not a well-known public service? This could be done maybe by querying a special DNS name which returns the IP address from where the query was received (like "whoami.akamai.net"). By comparing the ASN of the probe and the ASN of the IP address returned by the DNS query, one could determine, if the ISP's servers are used. This would also be true for people running their own recursor, but this could be filtered as well very easy. If an ISP is using multiple ASN, this could be a problem. Maybe there's an easy solution for this as well.
Probes which pass this test, could then be tagged with "DNS-using-ISP-server" or something like that and explicitly be selected for specific DNS resolution tests.
Greetings, Max
On 10/6/22 01:35, Johan ter Beest wrote:
Although you are completely free to choose whichever DNS server you think is best for your probe, we generally advise to use the same one that you assign to the rest of the network that the probe is connected to.
Thank you. To avoid any misunderstanding: When you say "rest of the network" you mean the LAN/WAN-router, not the ISP's AS, correct? :-) Daniel AJ
Hello, This seems to be an interesting question. We can certainly apply some (system) tags for probes that have the popular resolvers 8.8.8.8, 9.9.9.9 and so on in the resolver configuration. This would allow users like you to easily filter for, or filter out, probes that do this. One complication is that in many cases probes (an by extension, the users) have such a public resolver *in addition to* whatever else they use - which complicates the semantics of what resolver was actually used. But I guess one can accept that as a fact and still consider the feature to be useful. As an extension, we can, if that's deemed useful, tag other resolvers, along the lines of: * resolvers on private IPs (ie. on-net in the home?) * mixed private-and-quadX * mixed private-and-public If we go this far, a probe could have multiple tags, eg. uses-8888 + uses-private + mixed-private-and-quad8888. This may be overdoing it... We'd be curious about what you think. Regards, Robert On 2022-10-06 03:38, Max Grobecker wrote:
Hi,
a few days ago I wanted to debug a name resolution problem of one of our domains. For this reason, I wanted to test if probes inside a specific ASN are having difficulties to resolve a specific name (because only customers of this ISP were complaining). This lead to very mixed results, mostly because some of the selected probes did queries to a public DNS service like Google, Quad9 and so on. The problem existed only with the provider's DNS servers for some reason.
It did take some time to make a script which tried to filter out these probes, so I wondered if anyone else had the same use-case and problem. Is there a way to automatically tag probes, which are (seemingly) using the ISP's own DNS servers, or, at least, not a well-known public service? This could be done maybe by querying a special DNS name which returns the IP address from where the query was received (like "whoami.akamai.net"). By comparing the ASN of the probe and the ASN of the IP address returned by the DNS query, one could determine, if the ISP's servers are used. This would also be true for people running their own recursor, but this could be filtered as well very easy. If an ISP is using multiple ASN, this could be a problem. Maybe there's an easy solution for this as well.
Probes which pass this test, could then be tagged with "DNS-using-ISP-server" or something like that and explicitly be selected for specific DNS resolution tests.
Greetings, Max
Hi, Since a lot of probes use RFC1918 DNS resolvers (like home DSL/Cable routers etc.) you can't tell, if an ISP-resolver or Public-resolver is actually used. Another thing I noticed is, that some eyeball providers stopped provisioning their own DNS resolvers. Instead, they assing public resolvers like Cloudflare to their customers. If the distinction isn't to difficult to implement, I would prefer these three types as system tags: Inside-AS DNS Outside-AS DNS RFC1918 DNS Best Regards, Simon On 6 October 2022 09:23:15 UTC, Robert Kisteleki <robert@ripe.net> wrote:
Hello,
This seems to be an interesting question.
We can certainly apply some (system) tags for probes that have the popular resolvers 8.8.8.8, 9.9.9.9 and so on in the resolver configuration. This would allow users like you to easily filter for, or filter out, probes that do this.
One complication is that in many cases probes (an by extension, the users) have such a public resolver *in addition to* whatever else they use - which complicates the semantics of what resolver was actually used. But I guess one can accept that as a fact and still consider the feature to be useful.
As an extension, we can, if that's deemed useful, tag other resolvers, along the lines of: * resolvers on private IPs (ie. on-net in the home?) * mixed private-and-quadX * mixed private-and-public
If we go this far, a probe could have multiple tags, eg. uses-8888 + uses-private + mixed-private-and-quad8888. This may be overdoing it...
We'd be curious about what you think.
Regards, Robert
On 2022-10-06 03:38, Max Grobecker wrote:
Hi,
a few days ago I wanted to debug a name resolution problem of one of our domains. For this reason, I wanted to test if probes inside a specific ASN are having difficulties to resolve a specific name (because only customers of this ISP were complaining). This lead to very mixed results, mostly because some of the selected probes did queries to a public DNS service like Google, Quad9 and so on. The problem existed only with the provider's DNS servers for some reason.
It did take some time to make a script which tried to filter out these probes, so I wondered if anyone else had the same use-case and problem. Is there a way to automatically tag probes, which are (seemingly) using the ISP's own DNS servers, or, at least, not a well-known public service? This could be done maybe by querying a special DNS name which returns the IP address from where the query was received (like "whoami.akamai.net"). By comparing the ASN of the probe and the ASN of the IP address returned by the DNS query, one could determine, if the ISP's servers are used. This would also be true for people running their own recursor, but this could be filtered as well very easy. If an ISP is using multiple ASN, this could be a problem. Maybe there's an easy solution for this as well.
Probes which pass this test, could then be tagged with "DNS-using-ISP-server" or something like that and explicitly be selected for specific DNS resolution tests.
Greetings, Max
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
On Thu, 6 Oct 2022 at 13:32, Simon Brandt via ripe-atlas <ripe-atlas@ripe.net> wrote:
If the distinction isn't to difficult to implement, I would prefer these three types as system tags:
Inside-AS DNS Outside-AS DNS RFC1918 DNS
Agreed, these three tags would IMO satisfy *most* use cases. Certainly mine, too.
On Thu, 6 Oct 2022 at 13:32, Simon Brandt via ripe-atlas
<ripe-atlas@ripe.net> wrote:
If the distinction isn't to difficult to implement, I would prefer these three types as system tags:
Inside-AS DNS Outside-AS DNS RFC1918 DNS
Agreed, these three tags would IMO satisfy *most* use cases. Certainly mine, too. I'm now curious how that would work with dual-stack probes using different
On Thursday, 6 October 2022 19:42:58 EEST netravnen+ripelist@gmail.com wrote: providers for v4 and v6... My probe uses my local Provider for v4 and HE for IPv6 (static /48 network with reverse dns delegation would never be possible from my local provider). According to that logic, using my local router as dns server for the probe, could set all three tags depending on the used dns server. As an example depending on the transport protocol used for dns: - v4 uses RFC1918 IP as resolver IP and inside AS DNS Server - v6 uses public IP of subnet and resolves via a different AS if the query is sent over v6 It could also be argued, that the v6 case is still Inside-AS DNS, but it should be clear how the tags are determined. Or is this case in the meantime special enough to ignore it due to the rising native v6 rollout by providers?
On 6 Oct 2022, at 13:32, Simon Brandt via ripe-atlas wrote:
If the distinction isn't to difficult to implement, I would prefer these three types as system tags:
Inside-AS DNS Outside-AS DNS RFC1918 DNS
At least these are necessary, but I'm not sure that they are sufficient. For example, "Inside-AS" masks the distinction between on-site and ISP-provided resolvers, which are distinct on my domestic network (RFC1918 + GLA) and still (I believe) on the campus network where I once had a day job (Gv4 + GLA). Apart from that, should there be separate tags for RFC1918 and ULA? €0,02 Niall
Inside-AS DNS Outside-AS DNS RFC1918 DNS
We probably need a "localhost DNS" too I would still argue that the "big" public resolvers warrant their own tags.
Apart from that, should there be separate tags for RFC1918 and ULA?
Paraphrasing a quote from may decades ago: "probe DNS tagging is like math - a simple idea but it can get complicated" :-) We could separate IPv4 and IPv6 tags in this context. Which is clearer but also more complicated to process. Still, for each af, probes could get multiple of these tags. I believe it would be informative if we assembled some statistics about the current state, ie. how many probes would get what kind of tags if we applied them today. Perhaps Max already has some up his sleeve? Regards, Robert
I believe it would be informative if we assembled some statistics about the current state, ie. how many probes would get what kind of tags if we applied them today. Perhaps Max already has some up his sleeve?
Here's a quick-and-dirty answer to this: what the tags could look like if we applied them. This is about the ~12K connected probes, which collectively have ~23K resolver entries. Note: in this experiment the "more specific tags win", i.e. if quad1 is applied, then outside-asn tag is not. 6718 system-resolv-v4-private 4502 system-resolv-v4-inside-asn 2834 system-resolv-v4-google 1247 system-resolv-v4-outside-asn 1020 system-resolv-v4-loopback 933 system-resolv-v4-quad1 338 system-resolv-v4-quad9 31 system-resolv-v4-1-24 1745 system-resolv-v6-inside-asn 968 system-resolv-v6-ula 560 system-resolv-v6-loopback 473 system-resolv-v6-quad9 461 system-resolv-v6-google 335 system-resolv-v6-outside-asn 319 system-resolv-v6-linklocal 187 system-resolv-v6-quad1 Cheers, Robert
On 10 Oct 2022, at 13:42, Robert Kisteleki wrote:
Here's a quick-and-dirty answer to this: what the tags could look like if we applied them. This is about the ~12K connected probes, which collectively have ~23K resolver entries. Note: in this experiment the "more specific tags win", i.e. if quad1 is applied, then outside-asn tag is not.
Thanks for this quick and useful report, Robert. What I take from it is that - (a) for most of the categories of interest, the one which (most closely) matches a given probe may readily be determined, and - (b) only one category seems ambiguous with regard to Max's original question, since "inside ASN" doesn't necessarily correspond to "using the ISP's DNS server". I'm wondering whether this ambiguity matters in the context of Max's question, and reckon that his opinion on this is a significant one. For my own network, I have always the option of reconfiguring to use ULA/RFC1918 (both private) instead of GLA/RFC1918 (inside-ASN/private). Thanks again, Niall (hat off)
Max Grobecker <max.grobecker@ml.grobecker.info> writes:
This could be done maybe by querying a special DNS name which returns the IP address from where the query was received (like "whoami.akamai.net"). By comparing the ASN of the probe and the ASN of the IP address returned by the DNS query, one could determine, if the ISP's servers are used.
There should be no need for a new service. The SOS queries already provides the necessary raw data. You can see resolver addresses in the probe's "SOS History". Someone "just" has to process the data and produce a "Resolver-in-same-AS" tag.
This would also be true for people running their own recursor, but this could be filtered as well very easy.
How? Reject resolvers which are only used by a single probe? Or did you have something smarter in mind? If not, I fear it would produce a large number of false positives. Many ISPs will have a relatively large resolver to probe ratio (when counting resolver addresses visible to authoritative servers).
If an ISP is using multiple ASN, this could be a problem. Maybe there's an easy solution for this as well.
Geoff Huston has tried to analyze this as part of open resolver measurements: https://www.potaroo.net/ispcol/2019-09/centrality.html Doing a "same CC and not well-kown public resolver" might do it. Bjørn
participants (9)
-
Bjørn Mork
-
Daniel AJ Sokolov
-
Johan ter Beest
-
Karsten Thomann
-
Max Grobecker
-
netravnen+ripelist@gmail.com
-
Niall O'Reilly
-
ripe.net@toppas.net
-
Robert Kisteleki