So the new kid on the block for issuing IRR querys is bgpq3: http://snar.spb.ru/prog/bgpq3/ (Thanks to Job Snijders for pointing it out). Here are some comparative results for bgpq3 and irrtoolset using a random large-ish as-set with ~5000 ASNs:
ubuntu1:/home/nick/bgpq3-0.1.16> time bgpq3 AS-NEOT > /dev/null 0.144u 0.032s 0:04.26 3.9% 0+0k 0+0io 0pf+0w
ubuntu1:/home/nick/trunk/src/peval> time ./peval -h whois.radb.net -protocol irrd AS-NEOT > /dev/null 8.600u 0.516s 10:14.14 1.4% 0+0k 0+0io 0pf+0w
ubuntu1:/home/nick/trunk/src/peval> time ./peval -h whois.ripe.net -protocol ripe AS-NEOT > /dev/null 2.900u 0.304s 12:21.72 0.4% 0+0k 0+0io 0pf+0w
Due to its use of pipelining it can return IRRD query results ridiculously quickly, and it's pretty clear that irrtoolset is being completely trashed by comparison because it doesn't support pipelining. The difference between the RIPE and RADB whois server is related to the fact that the RADB server supports recursive as-set expansion. In fact, a more interesting question is not why bgpq3 is 175 times faster than irrtoolset, but rather why it takes as long as 4.26 seconds to complete an inverse query for the as-set in question. If I did ran a SQL query on a single term with indexed lookups and which returned 50k results, I'd wonder why it was so slow. In this case, it turns out that the answer mostly relates to the amount of data returned (~750k), and the rtt to the whois server from my query box (~150ms), rather than as a result of the query latency. Anyway, it should be clear that for large query sets, the RIPE server is extremely slow because of its lack of command pipelining and lack of support for recursive expansion of certain types of objects. This directly affects INEX because we run a full rebuild of our route server configuration three times daily. Our route server as-set (AS-SET-INEX-RS) contains about 5400 ASNs and the consequent serialisation delays are currently causing our rebuild scripts to take slightly more than 30 minutes to complete. This varies depending on client policies: earlier this year, rebuilds were taking about 2 hours. In fact for performance reasons, we need to query whois.radb.net instead of RIPE for the larger AS-SETs. This is because of the recursive as-set performance issue which affects queries to whois.ripe.net In order to solve this problem, what we'd really love to see is: 1. support for command pipelining 2. support for recursive expansion of as-set to a list of ASNs. 3. support for recursive inverse key expansion of an as-set to a list of prefixes These would mean a massive performance gain for INEX's use case (and a bunch of other IXPs around europe) and #3 would mean that the RIPE whois server could leapfrog IRRD in terms of performance. Would it be possible for the DB people to look into this? Nick