whois queries with respect to RIPE policies
Hi list, During the next days, I'm planning to analyze some suspicious traffic that I collected during an execution of malware. One of my goal is to retrieve additional information for every IP address that is in that traffic. For this, I believe that the whois data is a very valuable source of information. However, I realize that I should not overdo in performing whois queries to RIPE (and RIR databases in general). I expect to query the database ~ 20,000 times per day. Here comes my problem: According to the RIPE policy, I am allowed to query the database even frequently. At the same time, one should avoid querying personal data too often. I am trying now to find a program (e.g., jwhois, the 'usual' UNIX whois client, with some weird parameters) that allow me to comply to the RIPE policies. For my convenicen, if I can somehow avoid it, I'd rather use the online database instead of syncing it and performing requests locally. In addition, I checked the whois usage policies of other RIRs. Admittedly, RIPE has got the far most transparent ones! Others don't give explicit limits in their usage, and I have to hope that I won't be blocked when using whois extensively. Would you please share your experiences, and maybe even give hints about a 'correct' usage of a whois client? Thanks in advance, Chris
* chris:
Would you please share your experiences, and maybe even give hints about a 'correct' usage of a whois client?
What type of objects do you need? If inetnum:s and route:s are sufficient, you could use the database dump on ftp.ripe.net. -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99
Florian, thanks for your reply.
Would you please share your experiences, and maybe even give hints about a 'correct' usage of a whois client?
What type of objects do you need? If inetnum:s and route:s are sufficient, you could use the database dump on ftp.ripe.net. I'd like to get as much information out of whois as possible, without breaking the RIPE policies. Downloading the dump via FTP is an option, yes, but as I wrote I'd prefer to avoid it. The main reasons for this are that a) parsing this database is not very convenient and b) I'd need to synchronize also with the other RIRs.
Chris
Dear Chris, There is only one way to get permanently banned by the RIPE DB: repeatedly ignoring its error messages. All other oversteps of limits result in a warning message only, one that users can and should react on. For your case, specifically, you should take care of the following: - always use the flags "-rR" to avoid querying for private date and avoid using "-B"; - do not query for person, role, ... objects - only internet resources like inetnum or aut-num; - If querying the RIPE DB in parallel, you should take care not to open more than 4 connections to whois at the same time. If you keep yourself to this, there is absolutely no problem firing 20K queries a day. As suggested by Florian earlier, if you have high volumes of queries, though, you should consider using the split files available on the ftp site. There is no exact definition of "high volume", as it also depends on what our servers can handle. At the moment I would say we will probably ask you to consider using split files if you have more than a couple hundred thousand queries a day. If you have any more questions, feel free to ask. Kind regards, Agoston Horvath Database Group RIPE NCC On 2010-01-27 11:46 AM, chris wrote:
Florian, thanks for your reply.
Would you please share your experiences, and maybe even give hints about a 'correct' usage of a whois client?
What type of objects do you need? If inetnum:s and route:s are sufficient, you could use the database dump on ftp.ripe.net. I'd like to get as much information out of whois as possible, without breaking the RIPE policies. Downloading the dump via FTP is an option, yes, but as I wrote I'd prefer to avoid it. The main reasons for this are that a) parsing this database is not very convenient and b) I'd need to synchronize also with the other RIRs.
Chris
Dear Agoston,
For your case, specifically, you should take care of the following:
- always use the flags "-rR" to avoid querying for private date and avoid using "-B";
- do not query for person, role, ... objects - only internet resources like inetnum or aut-num; Thanks a lot for these very helpful hints! I realized that -rR does exactly what I was looking for. Correct me if I am wrong: Using -rR implies that I comply to your second point, doesn't it?
If you keep yourself to this, there is absolutely no problem firing 20K queries a day. Great, that's good news. However, I realized that there are two additional problems:
1) I also would like to query IP addresses of other RIRs. No matter what IP address I choose, my whois client (jwhois) always queries ARIN first, which redirects to the correct RIR. Since I'd like to avoid these redirects, my idea is to use the -h (host) option (e.g., -h whois.ripe.net) and vary the option depending on which IP address I query. I would use [1] as source to vary this parameter. Do you see any problems doing so? 2) As I mentioned already, RIPE whois policies are very transparent compared to other RIRs' ones. Whereas -rR works seamlessly for RIPE, it does not work for other RIRs (which is at least unfortunate). As I have to vary parameters for the whois call anyway (see 1), I could also switch off -rR for other RIRs. However, this would query personal data from other RIRs. Although this is not RIPE business now, is there any good practice in complying also to the whois policies of other RIRs? Thanks again for helping out. Cheers, Chris [1]: http://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xml
chris wrote:
Dear Agoston,
For your case, specifically, you should take care of the following:
- always use the flags "-rR" to avoid querying for private date and avoid using "-B";
- do not query for person, role, ... objects - only internet resources like inetnum or aut-num;
Thanks a lot for these very helpful hints! I realized that -rR does exactly what I was looking for. Correct me if I am wrong: Using -rR implies that I comply to your second point, doesn't it?
Yes the -r flag will prevent person objects being returned with your query results. You only need -R if you are querying for forward domain objects to prevent referral queries to domain registries. But this information is unreliable and will be removed from the RIPE Database soon.
If you keep yourself to this, there is absolutely no problem firing 20K queries a day.
Great, that's good news. However, I realized that there are two additional problems:
1) I also would like to query IP addresses of other RIRs. No matter what IP address I choose, my whois client (jwhois) always queries ARIN first, which redirects to the correct RIR. Since I'd like to avoid these redirects, my idea is to use the -h (host) option (e.g., -h whois.ripe.net) and vary the option depending on which IP address I query. I would use [1] as source to vary this parameter. Do you see any problems doing so?
You may find this list is not exactly up to date. There are some ranges that are moved from one RIR to another. For example we recently moved some more legacy ranges from RIPE to AfriNIC. But you can handle these exceptions when you find them.
2) As I mentioned already, RIPE whois policies are very transparent compared to other RIRs' ones. Whereas -rR works seamlessly for RIPE, it does not work for other RIRs (which is at least unfortunate). As I have to vary parameters for the whois call anyway (see 1), I could also switch off -rR for other RIRs. However, this would query personal data from other RIRs. Although this is not RIPE business now, is there any good practice in complying also to the whois policies of other RIRs?
APNIC and AfriNIC run similar whois code as RIPE. These flags may work with their systems. For ARIN and LACNIC you will have to check with them about their options and limits. Regards Denis Walker
Thanks again for helping out.
Cheers, Chris
[1]: http://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xml
participants (4)
-
Agoston Horvath -
chris -
Denis Walker -
Florian Weimer