whois-server friendly client programming
Hi, I am writing a tool to generate OpenBGPD filters from the policy specification in aut-num objects. Naturally, this involves sending a lot of queries to the whois servers: -one for the aut-num in question -one for every as-set given in the policy -one for every member of these as-sets, and recursively for their members, until we're down to an all-aut-num list -for each aut-num, a query for the routes it exports (-i origin <AS>) I do have extensive caching in place, all responses are beeing cached (or rather, the parse result), so no query is ever sent twice. nontheless, a lot of queries remain. right now, I have whois.ripe.net hardwired, of course this is going to change before the tool is released. I am using "-r -a -T <objtype> <object>" for aut-num and as-set, and "-r -a -T <objtype> -i origin <as>" for route objects. Questions: can I make life easier for the whois servers by using another set of query options? what is a good default whois server? whois.ripe.net? whois.radb.net? of course, I run into the penalty delays the whois server imposes because of the amount of queries I send. How do I deal with that? do nothing, just accept the delay (that's how it is now)? is it possible (and feasable!) to mirror the entire IRR? is this documented somewhere? I'd happily point to that documentation and recommend running a local mirror in the manpage if that is feasable. the forced delays also make development pretty painful ;( Thanks & Greetz Henning Brauer -- Henning Brauer, hb@bsws.de, henning@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
On 2 Mar 2007, at 03:31, Henning Brauer wrote:
right now, I have whois.ripe.net hardwired, of course this is going to change before the tool is released. I am using "-r -a -T <objtype> <object>" for aut-num and as-set, and "-r -a -T <objtype> -i origin <as>" for route objects.
you can reduce the information you get back by asking the RIPE whois server to give you only the primary keys using the -K flag. In route objects that would give you the prefix and the origin only, and not the rest of the ancillary info. The option was put there just for this use.
Questions: can I make life easier for the whois servers by using another set of query options?
what is a good default whois server? whois.ripe.net? whois.radb.net?
they tend to mirror each other and be sync'ed. By using the -a option above, you are specifying that the server search them all, so it doesn't matter.
of course, I run into the penalty delays the whois server imposes because of the amount of queries I send. How do I deal with that?
you contact the RIPE NCC and sign an AUP, so the limit is removed.
do nothing, just accept the delay (that's how it is now)?
is it possible (and feasable!) to mirror the entire IRR? is this documented somewhere? I'd happily point to that documentation and recommend running a local mirror in the manpage if that is feasable.
Again, yes, possible. Ask the people who run the IRRs (each of the ones you want to mirror) Joao
* Joao Damas <Joao_Damas@isc.org> [2007-03-02 03:58]:
On 2 Mar 2007, at 03:31, Henning Brauer wrote:
right now, I have whois.ripe.net hardwired, of course this is going to change before the tool is released. I am using "-r -a -T <objtype> <object>" for aut-num and as-set, and "-r -a -T <objtype> -i origin <as>" for route objects.
you can reduce the information you get back by asking the RIPE whois server to give you only the primary keys using the -K flag. In route objects that would give you the prefix and the origin only, and not the rest of the ancillary info. The option was put there just for this use.
excellent, doing that now.
Questions: can I make life easier for the whois servers by using another set of query options?
what is a good default whois server? whois.ripe.net? whois.radb.net? they tend to mirror each other and be sync'ed. By using the -a option above, you are specifying that the server search them all, so it doesn't matter.
the question was less about the data they carry - I know they should be in sync. I was more looking for things like "don't ever use X by default because they can't handle the load without you already" or the like :)
of course, I run into the penalty delays the whois server imposes because of the amount of queries I send. How do I deal with that? you contact the RIPE NCC and sign an AUP, so the limit is removed.
so that is what I need to recommend in the manpage? I have looked a bit more and spend some time with http://www.ripe.net/ripe/docs/db-query-manual.html
From that document it sounds like the penalty is only there if I query person/role data. That does not match what I see tho - I do not query anything but aut-num, as-set and route objects. Still, I am suspect to the delay.
Greetings Henning Brauer -- Henning Brauer, hb@bsws.de, henning@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
Henning Brauer wrote:
* Joao Damas <Joao_Damas@isc.org> [2007-03-02 03:58]:
On 2 Mar 2007, at 03:31, Henning Brauer wrote:
right now, I have whois.ripe.net hardwired, of course this is going to change before the tool is released. I am using "-r -a -T <objtype> <object>" for aut-num and as-set, and "-r -a -T <objtype> -i origin <as>" for route objects.
you can reduce the information you get back by asking the RIPE whois server to give you only the primary keys using the -K flag. In route objects that would give you the prefix and the origin only, and not the rest of the ancillary info. The option was put there just for this use.
excellent, doing that now.
Questions: can I make life easier for the whois servers by using another set of query options?
what is a good default whois server? whois.ripe.net? whois.radb.net?
they tend to mirror each other and be sync'ed. By using the -a option above, you are specifying that the server search them all, so it doesn't matter.
the question was less about the data they carry - I know they should be in sync. I was more looking for things like "don't ever use X by default because they can't handle the load without you already" or the like :)
of course, I run into the penalty delays the whois server imposes because of the amount of queries I send. How do I deal with that?
you contact the RIPE NCC and sign an AUP, so the limit is removed.
so that is what I need to recommend in the manpage?
I have looked a bit more and spend some time with http://www.ripe.net/ripe/docs/db-query-manual.html
From that document it sounds like the penalty is only there if I query person/role data. That does not match what I see tho - I do not query anything but aut-num, as-set and route objects. Still, I am suspect to the delay.
May I suggest that you forward this mail to <ripe-dbm@ripe.net>. We will then discuss it with you off the list. There are things we can do to help you with this situation. regards DenisWalker Database Group RIPE NCC
Greetings
Henning Brauer
Hi Henning, On 3/1/07, Henning Brauer <hb@bsws.de> wrote:
Hi,
I am writing a tool to generate OpenBGPD filters from the policy specification in aut-num objects.
Naturally, this involves sending a lot of queries to the whois servers: -one for the aut-num in question -one for every as-set given in the policy -one for every member of these as-sets, and recursively for their members, until we're down to an all-aut-num list [...]
Thanks & Greetz
Henning Brauer [...]
For this, you can use something like: $ whois -h whois.radb.net '!ias-kpn,1' For further information, see the "Summary of Commands" section in Appendix B at: http://www.irrd.net/irrd-user.pdf I hope this helps. Pierre.
Teilnehmer (4)
-
Denis Walker -
Henning Brauer -
Joao Damas -
Pierre Baume