Easy way to view DNS results?
Hi, I’m wondering if I’m missing something - I ran a DNS measurement with 110 probes, and the results are in, but the site UI seems to only show the result status and time elapsed until response, but not the results themselves. Looking at the JSON from the result endpoint I’m seeing that it seems to return the raw binary response - is there a particular reason it can’t be parsed and displayed in a more human-readable format? I know there are libraries and tools that do result parsing, but being on mobile at the moment (and often) means those are somewhat less accessible as far as I’m aware.
Micha Bailey writes:
Hi, I’m wondering if I’m missing something - I ran a DNS measurement with 110 probes, and the results are in, but the site UI seems to only show the result status and time elapsed until response, but not the results themselves. Looking at the JSON from the result endpoint I’m seeing that it seems to return the raw binary response - is there a particular reason it can’t be parsed and displayed in a more human-readable format? I know there are libraries and tools that do result parsing, but being on mobile at the moment (and often) means those are somewhat less accessible as far as I’m aware.
Hi Micha, Do you mean the way that the JSON contains the "abuf" field with the base64-encoded DNS response? I have actually not worked with these before, but I was able to make some progress in Python as follows (using the python3-dnspython package). You could modify the "to_text()" call to print out some more specific information from the DNS reply of interest to you, or modify the enclosing print() to save it elsewhere. I don't know why there isn't a parsed version of the reply included in the JSON, but perhaps the idea is that the literal details are of interest to some researchers. One example that I happened to notice in trying to answer your question: in parsing a sample DNS measurement this way, I notice the use of DNS case randomization (also called "0x20 randomization") in some replies but not in others. Having the literal DNS query reply could help with analyzing the prevalence of this mechanism, whereas it might be obscured by a parser that was written by someone who believed that DNS replies are not case insensitive (which is true from one point of view, but not from another point of view!). #!/usr/bin/env python3 import json import base64 from dns import message import sys measurements = json.load(open(sys.argv[1])) for measurement in measurements: if "resultset" in measurement: for resultsetitem in measurement["resultset"]: if "result" in resultsetitem: abuf = resultsetitem["result"]["abuf"] msg = message.from_wire(base64.b64decode(abuf)) print(msg.to_text()) print()
On 25. 07. 23 7:31, Seth David Schoen wrote:
I don't know why there isn't a parsed version of the reply included in the JSON, but perhaps the idea is that the literal details are of interest to some researchers. One example that I happened to notice in trying to answer your question: in parsing a sample DNS measurement this way, I notice the use of DNS case randomization (also called "0x20 randomization") in some replies but not in others. Having the literal DNS query reply could help with analyzing the prevalence of this mechanism, whereas it might be obscured by a parser that was written by someone who believed that DNS replies are not case insensitive (which is true from one point of view, but not from another point of view!).
For people interested in DNS case (in)sensitivity, have a look at https://datatracker.ietf.org/doc/html/rfc4343 -- Petr Špaček
Hello, We're not parsing and inserting all the bits from the raw response into the JSON mostly because it would inflate the size of the results enormously, while users are generally only interested in specific bits. The UI gives the basics (success, RTT) to show the big picture but exposing all the details would overload the interface, especially on mobile. The "abuf" is always in there, so consumers of the results can parse that (as you already know, and as Seth does below). There are abuf parsers out there, and if you're using Python then I'd recommend using Sagan (https://github.com/RIPE-NCC/ripe-atlas-sagan/) Cheers, Robert On 2023-07-25 07:31, Seth David Schoen wrote:
Micha Bailey writes:
Hi, I’m wondering if I’m missing something - I ran a DNS measurement with 110 probes, and the results are in, but the site UI seems to only show the result status and time elapsed until response, but not the results themselves. Looking at the JSON from the result endpoint I’m seeing that it seems to return the raw binary response - is there a particular reason it can’t be parsed and displayed in a more human-readable format? I know there are libraries and tools that do result parsing, but being on mobile at the moment (and often) means those are somewhat less accessible as far as I’m aware.
Hi Micha,
Do you mean the way that the JSON contains the "abuf" field with the base64-encoded DNS response?
I have actually not worked with these before, but I was able to make some progress in Python as follows (using the python3-dnspython package). You could modify the "to_text()" call to print out some more specific information from the DNS reply of interest to you, or modify the enclosing print() to save it elsewhere.
I don't know why there isn't a parsed version of the reply included in the JSON, but perhaps the idea is that the literal details are of interest to some researchers. One example that I happened to notice in trying to answer your question: in parsing a sample DNS measurement this way, I notice the use of DNS case randomization (also called "0x20 randomization") in some replies but not in others. Having the literal DNS query reply could help with analyzing the prevalence of this mechanism, whereas it might be obscured by a parser that was written by someone who believed that DNS replies are not case insensitive (which is true from one point of view, but not from another point of view!).
#!/usr/bin/env python3
import json import base64 from dns import message import sys
measurements = json.load(open(sys.argv[1]))
for measurement in measurements: if "resultset" in measurement: for resultsetitem in measurement["resultset"]: if "result" in resultsetitem: abuf = resultsetitem["result"]["abuf"] msg = message.from_wire(base64.b64decode(abuf)) print(msg.to_text()) print()
On Tue, Jul 25, 2023 at 09:40:34AM +0200, Robert Kisteleki <robert@ripe.net> wrote a message of 88 lines which said:
We're not parsing and inserting all the bits from the raw response into the JSON mostly because it would inflate the size of the results enormously, while users are generally only interested in specific bits.
Also: 1) The DNS format continues evolving, for instance with new EDNS options, so it would require the Atlas team to follow. 2) Sometimes, people are interested in invalid DNS answers, that cannot be parsed.
The "abuf" is always in there, so consumers of the results can parse that (as you already know, and as Seth does below). There are abuf parsers out there, and if you're using Python then I'd recommend using Sagan
Or use Blaeu <https://labs.ripe.net/author/stephane_bortzmeyer/creating-ripe-atlas-one-off-measurements-with-blaeu/>: % blaeu-resolve --measurement-ID 57629583 check.ns1.dtc.dnssec.lab.nic.cl. [::2] : 3 occurrences [] : 1 occurrences Test #57629583 done at 2023-07-25T00:15:02Z
On 7/25/23 00:56, Stephane Bortzmeyer wrote:
The "abuf" is always in there, so consumers of the results can parse that (as you already know, and as Seth does below). There are abuf parsers out there, and if you're using Python then I'd recommend using Sagan Or use Blaeu <https://labs.ripe.net/author/stephane_bortzmeyer/creating-ripe-atlas-one-off-measurements-with-blaeu/>:
% blaeu-resolve --measurement-ID 57629583 check.ns1.dtc.dnssec.lab.nic.cl. [::2] : 3 occurrences [] : 1 occurrences Test #57629583 done at 2023-07-25T00:15:02Z
Also if you happen to have a GCP account, the bigquery interface has a view that tries to decode the abufs:
select * from `ripencc-atlas.measurements.dns_decoded` where date(start_time) = "2023-07-25" limit 100
You can filter down to your own measurement ID if you like, though as a platform it's probably better aligned to work with all the abufs within a timespan. S.
I see - yeah, I can definitely see the case for not including every bit of fully parsed data from the response. I would, however, posit that besides success and RTT, the actual response content itself (especially for simple A/AAAA records, perhaps not necessarily any query) should be considered part of the basics, and be at the very least presented in the UI (and as a wishlist item, aggregated/processed visually, as I believe some other measurement types are, e.g. in order to more easily spot anomalies). On a PC it would definitely not be a big deal to use libraries such as that (and I believe there are also CLI tools) to extract the data, but there are scenarios where that’s not an option, in my case I’m not often at my own PC recently so most non-work activities are on a mobile device where that’s not as easily achieved. On Tue, Jul 25, 2023 at 10:40 AM Robert Kisteleki <robert@ripe.net> wrote:
Hello,
We're not parsing and inserting all the bits from the raw response into the JSON mostly because it would inflate the size of the results enormously, while users are generally only interested in specific bits.
The UI gives the basics (success, RTT) to show the big picture but exposing all the details would overload the interface, especially on mobile.
The "abuf" is always in there, so consumers of the results can parse that (as you already know, and as Seth does below). There are abuf parsers out there, and if you're using Python then I'd recommend using Sagan (https://github.com/RIPE-NCC/ripe-atlas-sagan/)
Cheers, Robert
Micha Bailey writes:
Hi, I’m wondering if I’m missing something - I ran a DNS measurement with 110 probes, and the results are in, but the site UI seems to only show
result status and time elapsed until response, but not the results themselves. Looking at the JSON from the result endpoint I’m seeing
seems to return the raw binary response - is there a particular reason it can’t be parsed and displayed in a more human-readable format? I know
are libraries and tools that do result parsing, but being on mobile at
On 2023-07-25 07:31, Seth David Schoen wrote: the that it there the
moment (and often) means those are somewhat less accessible as far as I’m aware.
Hi Micha,
Do you mean the way that the JSON contains the "abuf" field with the base64-encoded DNS response?
I have actually not worked with these before, but I was able to make some progress in Python as follows (using the python3-dnspython package). You could modify the "to_text()" call to print out some more specific information from the DNS reply of interest to you, or modify the enclosing print() to save it elsewhere.
I don't know why there isn't a parsed version of the reply included in the JSON, but perhaps the idea is that the literal details are of interest to some researchers. One example that I happened to notice in trying to answer your question: in parsing a sample DNS measurement this way, I notice the use of DNS case randomization (also called "0x20 randomization") in some replies but not in others. Having the literal DNS query reply could help with analyzing the prevalence of this mechanism, whereas it might be obscured by a parser that was written by someone who believed that DNS replies are not case insensitive (which is true from one point of view, but not from another point of view!).
#!/usr/bin/env python3
import json import base64 from dns import message import sys
measurements = json.load(open(sys.argv[1]))
for measurement in measurements: if "resultset" in measurement: for resultsetitem in measurement["resultset"]: if "result" in resultsetitem: abuf = resultsetitem["result"]["abuf"] msg = message.from_wire(base64.b64decode(abuf)) print(msg.to_text()) print()
participants (6)
-
Micha Bailey -
Petr Špaček -
Robert Kisteleki -
Seth David Schoen -
Stephane Bortzmeyer -
Stephen Strowes