Re: To get things started
Matti Rendahl from NORDUnet, who's also on this list, is managing the nameserver at nic.nordu.net, and promised to give some input on the impact of having a rootserver, to the group.
I know it was something that had high priority when I got back from vacation, sorry about the delay, and sorry about this brief and hasty contribution (I really was planning to get to Paris, but there is no time for that).
One of the "hot" items that have to be discussed is wether we need another root name server in Europe. My guess would be yes.
Yes, I really think we do. And the larger the European IP community gets, the more we need the redundancy in root servers here.
I think it is the task of this working group to get input from people wanting to set up another root-server, make them aware of the consequences in terms of manpower, hardware, connectivity (especially the last one, since a root server must be reachable from EVERY single host on the Internet).
First let me summarize: Regarding manpower, not very much is needed in hours, more in alertness. And when looking at hardware, a SparcStation 1 with 500M disk and 16M memory will do. The crucial point is the connectivity, this must be good, otherwise there is not much use adding one more root server. 1) manpower It is not that much work to get the server to run, the only thing that must be done/checked is to pickup the new zone files from nic.ddn.mil (with ftp, zone-transfers won't do). This can be made with a script (a mail message is sent when the new files are available), even if I prefer to do it manually as the transefer fails every now and then (or the zone files themself can be corrupt/ truncated). The important thing is that the new zone files is loaded ASAP, and that the servers continues to work after this (can be checked with e.g. doc). Checking that the server is running is easy with a script, and as BIND crashes every now and then this is vital. The zone files are updated twice a week (Tuesday and Thursday, late evening MET). 2) hardware NIC.NORDU.NET is a SparcStation 1, and this works nice (we tried an ELC, but that didn't really work, don't know if it was the e-net interface or what, but it didn't really manage the load). If it is only used for the name server, I would say that 16M memory will do, BIND tends to get big, lots of cache. The zone files does not require much disk, but if one like to log questions (LOGIT patches) a few 100M disk is needed for the server. (We scratch the log every 24 hours at nordunic, but the daily log can get > 100M.) (To process these large logfiles you need more memory...) The root servers are all (?) running BIND 4.8.3, but there is no real coordination that they have the same patches applied &c. There are som patches available to disable zone transfers except from "trusted" hosts (to get some load of the network), and to disable recursive queries. 3) connectivity This is really important, there is no use in adding one more root server if it is not well connected. (Networks that have problems with bad connectivity should set up a fake root server, like we did here at nordunic, not apply for a root server. There are ways to manage a fake server so that it in all except the NS list for `.' is a mirror of the real root servers. The fake root server we had on nordunic was most of the time more in sync with NIC.DDN.MIL that the real ones :-) It is also required for the root server to act as a primary server for the 3-letter zones and ARPA/IN-ADDR.ARPA as well as the root zone. And it is preferred that is also a server for the zone it resides in. To get an idea of the load, here is the monthly statistics from nordunic (note: nordunic does recursive lookup, but those questions is not included in this, this is only the questions received bye nordunic). I haven't really checked, but I think one can see an increase in load that to some extent reflects the upgrades of lines to both Europe and US. Month Questions Day min Day max Day mean Q/sec 9107 8315817 22211 981077 277193 3 9108 10171786 58963 797348 339059 4 9109 8997370 2882 957857 299912 3 9110 11810123 8167 3178018 393670 5 9111 6837735 28871 1046749 227924 3 9112 10124765 24931 1665318 337492 4 9201 33076339 11873 3510106 1102544 13 9202 31598520 16778 3478439 1053284 12 9203 17007590 8843 1628033 566919 7 9204 24153201 29870 4394418 805106 9 9205 26104016 27969 3461392 870133 10 9206 49711587 939 4426938 1657052 19 9207 43547168 30481 5350535 1451572 17 9208 76186421 92550 7481229 2539547 29 I'm working with a program that should be able to give more information from the logs than this and Top-10 lists. Would be nice to get some kind of indication about from where the questions are generated, I know that we see a lot of questions from the US, but I have no real feeling for how the geographic distribution looks. I have some figures about DNS/UDP traffic also, but I can't find them right now. But if I remember correct, it was a few % of the total use (which is "normal" DNS usage), with ftp(.funet.fi) as the winner :-) Matti ------------------------------------------------------------------------------ Matti Rendahl | SUNET/NORDUnet Operation Center | Phone: +46 8 7907224 | Royal Institute of Technology, KTH | Fax: +46 8 7230302 | S-100 44 Stockholm, Sweden | Internet: matti@nordu.net
Below I'll make a comparison with what you state in your message and the situation on ns.EU.net. First let me summarize: Regarding manpower, not very much is needed in hours, more in alertness. Clear, but part of it can be automated. And when looking at hardware, a SparcStation 1 with 500M disk and 16M memory will do. See below. The crucial point is the connectivity, this must be good, otherwise there is not much use adding one more root server. Speaks for itself. 1) manpower ...the only thing that must be done/checked is to pickup the new zone files from nic.ddn.mil (with ftp, zone-transfers won't do). This can be made with a script (a mail message is sent when the new files are available), even if I prefer to do it manually as the transfer fails every now and then (or the zone files themself can be corrupt/ truncated). .... This can mostly be reliably automated; it's comparable to how ns.EU.net constructs its named.boot file every night from -amongst other things- a list of domains it runs secondary for and that it receives from uunet. 2) hardware NIC.NORDU.NET is a SparcStation 1, and this works nice (we tried an ELC, but that didn't really work, don't know if it was the e-net interface or what, but it didn't really manage the load). Interesting, but: If it is only used for the name server, I would say that 16M memory will do, BIND tends to get big, lots of cache. Maybe it will do for a machine that runs *only* root server, but not if it has to run primary/secondary for a bunch of other domains too. By comparison: ns.EU.net is an ELC; it runs BIND 4.8.3; it runs primary/secondary for some 1700 domains; the SIZE and RSS are around 10M on the average but can go well above 15M; queries/day average between 200,000 and 1,200,000. Crucial point is swapping, which has to be avoided at all cost. Which means 16M of memory just isn't enough, especially when you take into consideration that things like zone transfers by secondary servers off ns.EU.net cause the named process to fork off *the complete core image*. Even the current 24M in ns.EU.net isn't enough to cope with that, so we're about to upgrade it to 40M of memory. Piet
[after finishing I see that Berthold already send out info an important part of my text below...]
Interesting, but: If it is only used for the name server, I would say that 16M memory will do, BIND tends to get big, lots of cache. Maybe it will do for a machine that runs *only* root server, but not if it has to run primary/secondary for a bunch of other domains too. By comparison: ns.EU.net is an ELC; it runs BIND 4.8.3; it runs primary/secondary for some 1700 domains; the SIZE and RSS are around 10M on the average but can go well above 15M; queries/day average between 200,000 and 1,200,000. Crucial point is swapping, which has to be avoided at all cost. Which means 16M of memory just isn't enough, especially when you take into consideration that things like zone transfers by secondary servers off ns.EU.net cause the named process to fork off *the complete core image*. Even the current 24M in ns.EU.net isn't enough to cope with that, so we're about to upgrade it to 40M of memory.
deins.Informatik.Uni-Dortmund.DE is just a 3/60 with 16 MB physical. It carries some 500 zones, and the rate of queries varies between 0,5 to some 10 per second in peaks with an average of 1 query/sec. Virtual memory of the named process is just below 10 MB with just the zones loaded and usually is about 12.5 MB (with very rare peaks into the 14 MB range), usually the RSS is in the 3 MB range. Low horse power is no problem: the named process including children just consumes < 5% of the cpu cycles. We do NOT get paging problems (though overall we are close to one disk I/O per query handled). We experienced severe memory problems on outgoing zone transfers - but they actually were NOT paging but running out of swap space (after updating one of our primary zones we saw a lot of requests to pull that zone - and each child process allocates swap space for the complete activated virtual address space of course needing almost none). With a bit of hacking elimanated that problem. Our BIND version has a number of home-grown patches, most important in this context: - log some statistics on resources used and traffic supported - (in a rather crude way) identify the most frequently asked queries/most frequently asking clients [points out a lot of actual or latent problems without extensive logging of full queries] - logging of what zone transfers are done (want to see about unpublished secondaries) - outgoing zone transfers NOT handled by a forked copy of of the regular named but by extra program pulling the zone data from the disk (also eliminates bad extra glue as accumulated in the cache) So, DE-NIC experience is a bit different from the problems Piet sees, though the DE-NIC server falls into the same class of (the few) servers that carry a lot of zones and accumulate considerable cache data for a large set of clients. So my conclusion is that requirements are indeed not too high. However I think that Piet's and DE-NIC's configurations and should not be the modell for a root name server. Root name servers are not required to carry a lot of zones. They don't need to acquire a huge cache (at least not if you choose to reject recursive queries). In fact I think that a root server better be a small system and configured so the small system is sufficient - and completely dedicated (with only very few very seriously considered exceptions) to be root server (i.e. hardware cost for a complete server could be below 10 K$). If you want to go for being everybodys secondary use a separate system for that. If you want to be everybodys forwarder and support the apropriate cache. In particular with the current bad state of the names server software with regards to using only authoritative data as authoritative this kind of separation one should avoid to have the root servers answer much they are not really authoritative on. The most important question regarding root NS experiences has not been addressed so far: what are the statistics for the traffic generated by BIND's regular queries to all name servers (to learn which one is closest), and what levels of traffic are generated by the usual kinds of misconfigurations. There was a report - I think from ISI - with in depth analysis of root name server traffic (and conclusions like: we are seeing about 20fold the traffic that really is neccessary and negative caching would NOT improve things significantly from a global point of view). In my opinion it would be best to base considerations about how many root servers should be placed where mainly on data and arguments as provided by that report and some check how that relates to experience with NORDUnet's server. Ruediger Ruediger Volk Universitaet Dortmund, Informatik IRB DE-NIC Postfach 500 500 D-W-4600 Dortmund 50 Germany E-Mail: rv@Informatik.Uni-Dortmund.DE Phone: +49 231 755 4760 Fax: +49 231 755 2386
Piet Beertema <Piet.Beertema@cwi.nl> writes: Crucial point is swapping, which has to be avoided at all cost. Which means 16M of memory just isn't enough, especially when you take into consideration that things like zone transfers by secondary servers off ns.EU.net cause the named process to fork off *the complete core image*.
Ruediger has a patch for that.
Piet Beertema <Piet.Beertema@cwi.nl> writes: Crucial point is swapping, which has to be avoided at all cost. Which means 16M of memory just isn't enough, especially when you take into consideration that things like zone transfers by secondary servers off ns.EU.net cause the named process to fork off *the complete core image*.
Ruediger has a patch for that.
created by Berthold Paffrath who also posted it to this list the evening just before the Paris RIPE meeting. Berthold called it a "dirty patch" - but it's in production use on the DE-NIC name server for several months and did not create any problems so far, but solved a severe memory problem. That patch also has the advantage, that only clean data coming straight from the zone file and keeping out any nasty cache data. Ruediger Ruediger Volk Universitaet Dortmund, Informatik IRB DE-NIC Postfach 500 500 D-W-4600 Dortmund 50 Germany E-Mail: rv@Informatik.Uni-Dortmund.DE Phone: +49 231 755 4760 Fax: +49 231 755 2386
participants (4)
-
Daniel Karrenberg -
Matti.Rendahl＠bion.kth.se -
Piet Beertema -
rv＠deins.informatik.uni-dortmund.de