Querying popular domains with RIPE Atlas
Dear colleagues, As you are probably aware, DNSMON* and RIPE Atlas built-in** measurements are often used when analysing outages and other issues with important DNS services, particularly the root zone servers. These measurements do not provide a direct indication of the impact on actual end-user experience because they involve directly querying authoritative name servers, thereby ignoring the effects of having multiple authoritative name servers as well as caching local recursive resolvers. To improve upon our DNS monitoring, we will initiate two built-in DNS measurements from all RIPE Atlas probes that use the probes' default resolvers instead of authoritative name servers. The first measurement will query for random top-level domains with the intention of avoiding caches and therefore hitting at least one root name server. The second measurement will query for popular domain names, with the intention of hitting caches where appropriate and getting an idea of the true impact on users. As a starting point for the second measurement, we will use a recent snapshot of the global top 50 visited sites according to the Alexa top sites list***, which combines unique visitors and individual page views in its calculations. Probes will cycle through these domains, querying for a different A record every 10 minutes. Since DNS queries containing these domains will be made from every connected RIPE Atlas probe, we would like to ask for feedback on whether querying the DNS for any of these domains could pose legal or political problems for probe hosts in certain parts of the world. I have also included the next 25 most popular domains below, which will be used to fill any gaps caused by removing items from the main list. Kind regards, Chris Amin RIPE NCC * https://atlas.ripe.net/dnsmon/ ** https://atlas.ripe.net/docs/built-in/ *** https://en.wikipedia.org/w/index.php?title=List_of_most_popular_websites&oldid=762455912 Provisional 50 domains to query =============================== google.com. youtube.com. facebook.com. baidu.com. wikipedia.org. yahoo.com. google.co.in. amazon.com. qq.com. google.co.jp. live.com. taobao.com. vk.com. twitter.com. instagram.com. hao123.com. sohu.com. sina.com.cn. reddit.com. google.de. linkedin.com. tmall.com. yahoo.co.jp. weibo.com. google.fr. 360.cn. google.co.uk. google.ru. google.com.br. yandex.ru. ebay.com. bing.com. msn.com. google.it. soso.com. t.co. wordpress.com. google.es. microsoft.com. tumblr.com. aliexpress.com. blogspot.com. netflix.com. amazon.co.jp. ok.ru. google.com.hk. stackoverflow.com. google.ca. google.com.mx. imgur.com. Next 25 most popular domains, excluding pornography =================================================== apple.com. Naver.com. mail.ru. imdb.com. popads.net. onclickads.net. office.com. google.co.kr. github.com. pinterest.com. paypal.com. tianya.cn. google.com.tw. google.com.tr. google.com.au. diply.com. amazon.de. google.co.id. microsoftonline.com. onclckds.com. twitch.tv. adobe.com. amazon.co.uk. wikia.com. cnzz.com.
should be fine from uk perspective, note you need to be careful about dos protection issues when probes behind provider mobile nat Colin
On 7 Feb 2017, at 14:16, Chris Amin <camin@ripe.net> wrote:
Dear colleagues,
As you are probably aware, DNSMON* and RIPE Atlas built-in** measurements are often used when analysing outages and other issues with important DNS services, particularly the root zone servers. These measurements do not provide a direct indication of the impact on actual end-user experience because they involve directly querying authoritative name servers, thereby ignoring the effects of having multiple authoritative name servers as well as caching local recursive resolvers.
To improve upon our DNS monitoring, we will initiate two built-in DNS measurements from all RIPE Atlas probes that use the probes' default resolvers instead of authoritative name servers. The first measurement will query for random top-level domains with the intention of avoiding caches and therefore hitting at least one root name server. The second measurement will query for popular domain names, with the intention of hitting caches where appropriate and getting an idea of the true impact on users.
As a starting point for the second measurement, we will use a recent snapshot of the global top 50 visited sites according to the Alexa top sites list***, which combines unique visitors and individual page views in its calculations. Probes will cycle through these domains, querying for a different A record every 10 minutes.
Since DNS queries containing these domains will be made from every connected RIPE Atlas probe, we would like to ask for feedback on whether querying the DNS for any of these domains could pose legal or political problems for probe hosts in certain parts of the world. I have also included the next 25 most popular domains below, which will be used to fill any gaps caused by removing items from the main list.
Kind regards, Chris Amin RIPE NCC
* https://atlas.ripe.net/dnsmon/ ** https://atlas.ripe.net/docs/built-in/ *** https://en.wikipedia.org/w/index.php?title=List_of_most_popular_websites&oldid=762455912
Provisional 50 domains to query =============================== google.com. youtube.com. facebook.com. baidu.com. wikipedia.org. yahoo.com. google.co.in. amazon.com. qq.com. google.co.jp. live.com. taobao.com. vk.com. twitter.com. instagram.com. hao123.com. sohu.com. sina.com.cn. reddit.com. google.de. linkedin.com. tmall.com. yahoo.co.jp. weibo.com. google.fr. 360.cn. google.co.uk. google.ru. google.com.br. yandex.ru. ebay.com. bing.com. msn.com. google.it. soso.com. t.co. wordpress.com. google.es. microsoft.com. tumblr.com. aliexpress.com. blogspot.com. netflix.com. amazon.co.jp. ok.ru. google.com.hk. stackoverflow.com. google.ca. google.com.mx. imgur.com.
Next 25 most popular domains, excluding pornography =================================================== apple.com. Naver.com. mail.ru. imdb.com. popads.net. onclickads.net. office.com. google.co.kr. github.com. pinterest.com. paypal.com. tianya.cn. google.com.tw. google.com.tr. google.com.au. diply.com. amazon.de. google.co.id. microsoftonline.com. onclckds.com. twitch.tv. adobe.com. amazon.co.uk. wikia.com. cnzz.com.
Dear Chris. It seems fine from my perspective. The second measurement should be answered by resolver's cache, being top-resolved names. But what are you thinking on the first measurement? I'd try to reduce to a minimun sending random queries to root servers, an infrastructure frequently abused :( Also for the benefit of transparency and politeness you could prepend some string on the form "ripe-atlas-test-measurement.<random>." Best, Hugo On 14:52 07/02, Colin Johnston wrote:
should be fine from uk perspective, note you need to be careful about dos protection issues when probes behind provider mobile nat
Colin
On 7 Feb 2017, at 14:16, Chris Amin <camin@ripe.net> wrote:
Dear colleagues,
As you are probably aware, DNSMON* and RIPE Atlas built-in** measurements are often used when analysing outages and other issues with important DNS services, particularly the root zone servers. These measurements do not provide a direct indication of the impact on actual end-user experience because they involve directly querying authoritative name servers, thereby ignoring the effects of having multiple authoritative name servers as well as caching local recursive resolvers.
To improve upon our DNS monitoring, we will initiate two built-in DNS measurements from all RIPE Atlas probes that use the probes' default resolvers instead of authoritative name servers. The first measurement will query for random top-level domains with the intention of avoiding caches and therefore hitting at least one root name server. The second measurement will query for popular domain names, with the intention of hitting caches where appropriate and getting an idea of the true impact on users.
As a starting point for the second measurement, we will use a recent snapshot of the global top 50 visited sites according to the Alexa top sites list***, which combines unique visitors and individual page views in its calculations. Probes will cycle through these domains, querying for a different A record every 10 minutes.
Since DNS queries containing these domains will be made from every connected RIPE Atlas probe, we would like to ask for feedback on whether querying the DNS for any of these domains could pose legal or political problems for probe hosts in certain parts of the world. I have also included the next 25 most popular domains below, which will be used to fill any gaps caused by removing items from the main list.
Kind regards, Chris Amin RIPE NCC
* https://atlas.ripe.net/dnsmon/ ** https://atlas.ripe.net/docs/built-in/ *** https://en.wikipedia.org/w/index.php?title=List_of_most_popular_websites&oldid=762455912
Provisional 50 domains to query =============================== google.com. youtube.com. facebook.com. baidu.com. wikipedia.org. yahoo.com. google.co.in. amazon.com. qq.com. google.co.jp. live.com. taobao.com. vk.com. twitter.com. instagram.com. hao123.com. sohu.com. sina.com.cn. reddit.com. google.de. linkedin.com. tmall.com. yahoo.co.jp. weibo.com. google.fr. 360.cn. google.co.uk. google.ru. google.com.br. yandex.ru. ebay.com. bing.com. msn.com. google.it. soso.com. t.co. wordpress.com. google.es. microsoft.com. tumblr.com. aliexpress.com. blogspot.com. netflix.com. amazon.co.jp. ok.ru. google.com.hk. stackoverflow.com. google.ca. google.com.mx. imgur.com.
Next 25 most popular domains, excluding pornography =================================================== apple.com. Naver.com. mail.ru. imdb.com. popads.net. onclickads.net. office.com. google.co.kr. github.com. pinterest.com. paypal.com. tianya.cn. google.com.tw. google.com.tr. google.com.au. diply.com. amazon.de. google.co.id. microsoftonline.com. onclckds.com. twitch.tv. adobe.com. amazon.co.uk. wikia.com. cnzz.com.
Hi Hugo, On 21/02/2017 15:16, Hugo Salgado-Hernández wrote:
Dear Chris. It seems fine from my perspective. The second measurement should be answered by resolver's cache, being top-resolved names.
But what are you thinking on the first measurement? I'd try to reduce to a minimun sending random queries to root servers, an infrastructure frequently abused :( Also for the benefit of transparency and politeness you could prepend some string on the form "ripe-atlas-test-measurement.<random>."
The intention of the first measurement is to check the actual reachability of the root zone name servers from probe resolvers. This is a measurement which was requested by the K-Root operations team within the RIPE NCC, and is intended to show the reachability of the root zone servers in general given issues with a particular root letter. These measurements will indeed be constructed in a way similar to how you suggested: <probe_id>.atlas.ripe.net.<random>. Kind regards, Chris Amin RIPE NCC
participants (3)
-
Chris Amin -
Colin Johnston -
Hugo Salgado-Hernández