- db-wg - mailman.ripe.net

Towards a better distributed database model..
by Cengiz Alaettinoglu 16 Jul '95

16 Jul '95

Hi, Here is a draft David and I put together. Comments/suggestios are welcome. Thanks. Cengiz -- Cengiz Alaettinoglu Information Sciences Institute (310) 822-1511 University of Southern California http://www.isi.edu/div7/people/cengiz.home ================================================== Draft: 4 Working Title: Towards a better distributed database model.. Date: 950707 Authors: David Kessens Cengiz Alaettinoglu Introduction: ------------- This document describes the new proposed synchronization and distribution model. Note that we state here distribution model, not distributed model. This builds further on the proposed synchronized database model for the short term. For convenience this model is also briefly discussed below. Current model: -------------- Internet Routing Registry (IRR) consists of several registries. Each registry maintains a separate and distinct database. Each database contains a set of objects (e.g. route objects, aut-num objects [Ripe-181]) which are identified by a key. Each object in a database also has a "source" attribute which is not a part of the object's key and identifies the registry whose database contains this object. Same object may occur in multiple databases (of different registries) with conflicts. However, the two objects when presented to a user (e.g. upon a whois query) will have different source attributes, and the user can determine which registry's data to trust more. Unfortunately, with the current model, the databases contain a lot of inconsistent objects. Hence, the current model has proven to be inadequate. The databases maintained by each registry are not distributed in any form. Registries keep a mirror copy of the databases of other registries and search for objects in all of the databases when replying to user queries. The mirroring is done by ftping the databases once in a predetermined interval (currently once a day). The following short term solution for getting a more synchronized database mirroring is currently being implemented at RIPE by the first author and is described below. However, we believe that this solution is not sufficient for the long term needs. Every registry makes once in a predetermined interval its full database available at a well known ftp site. In addition, it maintains and makes available a differences file (described below) which records the updates to its database since the last ftp-copy of its database was made available. When an update request is received by a registry, the database is updated if the object is syntactically correct and the registry is the registry specified in the source attribute. If the registry is not the registry specified in the source attribute, the update is forwarded to the correct registry or refused and returned to the user. Once the update is processed, a transaction is written to the differences file(s). It is possible to immediately forward the transactions to the other registries to keep the mirror copies up-to-date. Or once in a predefined interval, the registries may check the differences files of other registries and resynchronize if necessary. The full database copies (ftped periodically) and differences files enable one to access the states of the databases at a user specified time. This feature is often desired for configuring routers. Towards the long term solution: ------------------------------- For a long term solution the following preconditions are set: Transparency: We expect the number of registries to grow significantly. This means that it will be increasingly more difficult for users to keep track of every registry. Therefore, it should be possible for users to interact only with one registry while accessing and registering data and transparently getting the information from all different registries (this may involve a referral mechanism in the client programs and protocols) or transparently registering data in the database of the authoritative registry. Scalability: The current (synchronized short term) database/registry model requires each registry to obtain each other's databases and differences file(s) by some mechanism such as ftp. This requires NxN ftp (or other protocol) peering, where N is the number of registries, which does not scale. The current model has no hierarchy or other methods for scaling. Reliability/robustness: The model should be as reliable and as robust as possible. Since the system will run in an in principle evil and unreliable network, special measures for security should be built in from the start. Update and access speed: Updates should propagate as fast as possible. Users access and update the data interactively and do not accept that the updates take more then few seconds (and will not tolerate if the updates are slower in the new system than the current system). However, it may be acceptable to take more time to update the mirror copies, or it may be acceptable to take more time to update certain object which are not updated frequently (e.g. maintainer objects). Since several programs that use the system now need a (nearly) real-time access, we cannot sacrifice any performance. Distribution versus distributed: The IRR data is used for configuring routers. To be able to do this, two requirements are important: First, at least the maintainer of a database should be able to access the database in the current state but also at other time and date instances specified by her/him (earlier, but may be even later in time with delayed updates). Second, all data should be retrievable at each state, not just part of it. These requirements speak clearly against aggressive distribution of the data since there will always be problems with connectivity with some remote hosts and we need to be able to gather all data without any exceptions. Synchronization: It is desirable to have the databases and their mirrors as synchronized as possible. However, if this requirement slows the update speeds considerably, it can be compromised. The proposed model: ------------------- Why not DNS: Several people have suggested to use DNS. However, there are several problems with using the DNS: DNS is a very complex system and difficult to maintain. It does not support the needed ability to get the state of the database at a certain point in time. It does not support searches on arbitrary attributes such as locating all routes with community HEPNET or finding the person objects with name 'David Kessens' within a reasonable amount of resources and nearly real-time response. Furthermore, tools like prtraceroute have their good performance by accessing all the data from one nearby registry. Since DNS has no easy way of collecting all (without anything missing) the data from all the hosts on the Internet, tool performances would degrade. Furthermore, DNS security problems are not solved. Of course DNS is a working system in the real world, and we will benefit from it in designing our registry/database architecture. Distribution Topology: Full mesh ftp (or other protocol) peering of current model does not scale. A hierarchical peering topology where each registry only receives and passes data to and from its children and parent would scale to the large number of registries. However, a hierarchical peering topology has robustness problems since a down registry would make the complete subtree beneath it disconnected. These considerations led us to a non-hierarchical graph peering of registries. In this graph, each node represents a registry, and a link represents an ftp (or other protocol) peering. There are no roots. A registry can become neighbors with any other registry provided that the graph remains connected (perhaps bi-connected). Each registry only communicates with its neighbor registries. The neighboring registries exchange all data from all known registries. This makes it very easy to add new registries to the IRR: just become a neighbor of an existing registry. Consistency: ------------ Current model allows registering an object with the same key in two registries with conflicts. The consistency of the information is very important since this data is being used to configure operational routers and to diagnose operational routing problems. Currently, conflicts are being resolved by manual inspection which is very slow and tedious. One way to achieve consistency is to have a true distributed database with atomicity and synchronization. However, this solution does not meet our preconditions listed above (for example updates would be very slow, perhaps at the order of days, when there are network partitions). In the new model, we are debating between two solutions that should be able to avoid this conflicts. The first solution is based on multi-valued source attributes. The second solution is based on removing the source attribute altogether. Registries and multi-valued source attributes: Registries will store a new object which describes the registry. The registry description may contain information on which type of objects that are stored locally. If a request comes for information that might be stored in another registry, the requester will be referred to that registry. The client programs should cache this referral information (not the actual data returned!). The source attribute of an object now contains a list of registry names. An object can only be updated if all the registries mentioned in this attribute authorize that. The source attribute has no other semantics. This may require a two phase commit protocol run between the registries mentioned in the source attribute. A special case of this is when the source attribute contains only one registry which is identical to the current model. There can be other implicit semantics in the source attribute. Some registries might require to be one of the sources for certain objects. Also the type of object might require (as defined in the object definition) that object updates should be approved by some additional source(s) (the authoritative sources). Updating: The update process will be controlled by a modified two phase commit protocol. The user will send an update request to one of the registries. The registry will send the request to all the registries specified in the object's source attribute and to additional registries if required by the definition of the object's type. After the registries determine whether to accept the object, the first registry will acknowledge this to the user. After this, all the involved registries (including the one that received the request) will update their local database(s), and flood the update (with serial&timestamp of first source&signature) to the other registries. No source attributes: This approach views the whole IRR data as being one database (though it may still be maintained in pieces) with certain registries authoritative on certain parts of it. Since it is one database, there is no need for the source attribute to identify which database, and no two objects with the same key and conflicting information can be registered (i.e. one of them will overwrite the other one). Our solution is as follows: Maintainer objects are globally unique and are created if all (or an authoritative subset) registries approve it. This can be achieved by a two phase commit protocol. Note that the creation of the maintainer objects will be slow (and is slow currently since it is done manually). However, this is tolerable since maintainer objects are not created very often. Each maintainer object specifies a registry which is authoritative for the objects that are maintained by itself. When a user sends an update to a registry, that registry will forward it to the registry which is authoritative for the object. The authoritative registry updates its local copy of the database and invokes the flooding protocol for other registries to pick up this change. Note that, if an object is registered in the database with maintainer X who authorizes registry Y, all updates will be performed by registry Y. This will ensure consistency within Y's copy of the database. If someone tries to register the same object with a different maintainer than X, Y (or any other registry) will deny the update with a maintainer failure error. There is one problem with this approach. If the mirroring takes a long time, within that period, one can create two objects with the same key, different maintainers which authorize different registries (and possibly with conflicts) in the local databases of two registries. Hence, it is essential to keep the mirroring as synchronized as possible. In this case, when the next mirroring is done and the conflict is detected, the copy which is created later is removed and returned to the user. We think that the probability of this case happening will be small.

3 3

Re: Towards a better distributed database model..
by martin＠terena.nl 14 Jul '95

14 Jul '95

Maybe I'm barking up the wrong tree, (please let me know!), but seems to me that one could use WHOIS++ for this sort of distribution. (Yes, I _know_ it is distributed rather than distribution model). Bunyip have recently released 'digger' a WHOIS++ implementation: http://services.bunyip.com:8000/products/digger/digger.html Rgds, John

1 0

Draft Minutes RIPE21, Database-WG
by Wilfried Woeber, UniVie/ACOnet 02 Jun '95

02 Jun '95

This is the draft minutes for the DB-WG meeting at RIPE 21 in Rome. They are quite belated (and probably full of typos), but unfortunately nobody volunteered this time to act as a scribe... Comments, corrections, additions -as always- very welcome. Wilfried. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Draft Minutes: RIPE 21, Database-WG Rome, 8.5.1995 Chair: Wilfried Woeber, Scribe: Wilfried Woeber ------------------------------------------------------ 0. Administrative stuff Unfortunately nobody volunteered to take the minutes. I had to guide the discussions and to take the notes at the same time. Thus these minutes are probably not as comprehensive as we are used to. Apologies. The agenda was accepted as proposed, without changes. 1. DB-SW review David Kessens, RIPE NCC, reported on the current state of the database software and the environment at the NCC for providing maintenance and doing development work. Activities since RIPE 20 (as Marten and Tony having left the NCC/PRIDE Project) centered on becoming familiar with the software, performing bug fixes and implementing minor extensions, like the "network update" feature whois -U and whois -tv (for template verbose). Other aspects worth mentioning: . reverse lookup technology . hierarchical authorization (initially for domains) . referrals are being researched and estimates for the necessary implementation efforts are obtained. David confirmed that the official distributioin kit available (anonymous FTP) at the NCC always contains the most recent fixes issued and additions. For more details please refer to the slides supporting David's presentation which are available from the RIPE NCC FTP Server in the presentations directory ftp://ftp.ripe.net/ripe/presentations/ripe-m21-david-DB-REPORT.ps.Z Merit is also working on the databse software and has produced some tools. These things are available from: ftp://ftp.ra.net/routing.arbiter/tools/RAToolSet 2. Security While there was no concern regarding the functionality and usefulness of the authorisation methods already implemented (mail-from:, crypt-pw:) there is a need for even stronger authentication of updates. PGP seems the way to go, although there is still quite a bunch of open questions and a need for logistic (e.g. key distribution) development. It is not clear at the moment whether this should be integrated (DB, RIPE NCC) or be provided by mechanisms outside the scope of RIPE. SURFnet reported it's intent to use the RIPE-DB for routing configuration as soon as PGP would be available to provide the data reliability necessary. Others expressed strong interest, as well. Merit (LJ) did part of the work already to provide PGP authorisation for the email addresses in updates. Comments from the group stressed that it is essential to *not* mix the concepts of authentication and of encryption. Encryption is a logistic and/or legal problem in some regions. The group is requested to persue this aspect on the relevant mailing lists. *Action on David Kessens and Wilfried Woeber: Investigate the deployment of PGP for the RIPE Database and to draft a proposal. 3. Review of beta test for RIPE-Handles The method for generating RIPE-Handles was discussed once again, there are still open issues when trying to fully automate handle assignement because this would require both a DB-commit function as well as require the modification of submitted objects on-the-fly. The group did not resolve this issues, thus the current procedure (finger) remains in place. It was noted that the scripts involved should be made available to all interested parties. 4. External interfaces There is an on-going discussion (in the light of more than one organisation using the RIPE DB-SW now) on how to keep the various databases in synch. One of the ideas is to accept updates for each and every database on all interfaces, to analyze the "source:" attribute and then to automatically forward the update to the database as indicated by "source:". While this is probably quite interesting, it might not always be what the users intend. The alternative is to reject updates when submitted for the "wrong" "source:". This is the safe way. To meet the need of regularly transferring updates from one database to (an)other(s), a method using FTP and providing "deltas" is investigated. 5. Input for and review of objects Due to the scheduling of the working groups, there was no input at the DB-WG meeting. A proposal to expand the internet-router object to describe Mbone routing was circulated after the meeting, as drafted by the Mbone WG. 6. AOB None. Thus the meeting was closed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ WW. 2.6.95

1 0

Re: Delete syntax (again)
by David.Kessens＠ripe.net 18 May '95

18 May '95

Hi Dale, > Dale S. Johnson writes : > > Sorry; I think I asked this once but lost the answer: > > Where in RIPE documentation does anything describe how to use the "delete:" > attribute? (Both for us, so we know what it really wants, and even more > for the users who keep asking us). >From the source code (Probably most definitive ;-)): "delete attribute must contain email address and reason for delete" I cannot find any document describing it. May be anybody else knows documentation ? Marten/Tony ? I proposed the RIPE community a new whois flag combination. We now have : $ whois -t aut-num aut-num: [mandatory] [single] as-name: [optional] [single] descr: [mandatory] [multiple] as-in: [optional] [multiple] [..] I proposed to the RIPE database working group the new (verbose) combination : $ whois -tv aut-num aut-num: [mandatory] [single] The 'aut-num' objects describes the ... Syntax: as-name: [optional] [single] The 'as-name' attributes describes the ... Syntax: ... I think most people liked this idea and it should solve this never ending story about 'what is this object/attribute for ???' When I have time I will start implementing this. Kind regards, David Kessens RIPE NCC Database maintainer

1 0

Re: just an FYI - don't panic
by Dale S. Johnson 16 May '95

16 May '95

David, AAAAAARRRRRRRRRRRRGGGGGGGGGGGHHHHHHHHHHH!!!! THAT SIMPLE!!! Thanks extremely for the extra set of eyes. (sigh). --Dale > The inet-rtr support is there, although it is not very visible since it > uses only generic attributes as admin-c and others. Please note that you > forgot a colon after the admin-c attribute in your previous mail. After > adding that the inet-rtr object was properly updated : > > ------------------------------------------------------------------------ > Update OK: [inet-rtr] nss11.ans.net > > Delete OK: [inet-rtr] nss11.ans.net > ------------------------------------------------------------------------

1 0

Re: just an FYI - don't panic
by RIPE NCC Staff 16 May '95

16 May '95

Dear Dale, * "Dale S. Johnson" <dsj(a)merit.edu> writes: * Steve, * * > Who will act as coordinator to get this stuff distributed? I assume * > that the RA in the U.S. can take the lead in this role if we were to * > provide source. * * The only thing close to an official answer to this question at the moment * is that RIPE is the home and custodian of the official distribution * source. * * At the RA, we've been trying rather hard to minimize any modifications to * RIPE code, if only to simplify importing further revisions of their code. Thank you ! * (But not making modifications is getting harder, as we get user pressure * to make email address parsing case-insensitive, add PGP, [add inet-rtr * support, if we aren't just looking for it in the wrong places today], * etc). I think that most of things you mention here are exactly the same as the wishlist of the RIPE community so for that there is no need for a divergence ;-) The inet-rtr support is there, although it is not very visible since it uses only generic attributes as admin-c and others. Please note that you forgot a colon after the admin-c attribute in your previous mail. After adding that the inet-rtr object was properly updated : ------------------------------------------------------------------------ Update OK: [inet-rtr] nss11.ans.net Delete OK: [inet-rtr] nss11.ans.net ------------------------------------------------------------------------ * The RA does post the radbserver (rpslserver) and RtConfig tookkit * distributions, which are separable packages from the RIPE distribution. * If we were to package and post our own "RIPE-181-modified by the RA" * distribution today, there might be some considerable confusion. * * The RA is generally willing to install Curtis' changes in our * implementation, though this needs discussion with the list since * our accepting such non-181 syntax would probably break implementations * and/or tools of other registries (such as RIPE itself) which import * that data and make it available to their tools (like prtraceroute). (Syntax) changes (not the implementation details!) are normally discussed in the RIPE database working group. After we reach consensus the changes will be implemented. I would strongly ask you all to join the discussion on the <db-wg(a)ripe.net> as that is probably the most appropriate platform to discuss these issues. For example the case-sensitive E-mail parsing could be changed if our contributors and you all ask for it ! * Clearly this all needs sorting out. (Maybe starting in the halls at * NANOG?) But for the moment, the answer is that RIPE is the custodian. I hope to see you all in the halls ;-), Kind regards, David Kessens RIPE NCC Database maintainer ----------- * ============ From: Steve Heimlich * * > From heimlich(a)ans.net Mon May 15 21:21:43 1995 * > To: "Dale S. Johnson" <dsj(a)merit.edu> * > Cc: cengiz(a)isi.edu, curtis(a)ans.net, Tony.Bates(a)ns.mci.net, rr-impl(a)ripe.ne * t, * > selina(a)ans.net * > Subject: Re: just an FYI - don't panic * > Date: Mon, 15 May 1995 21:20:44 -0400 * > * > Dale, * > * > Who will act as coordinator to get this stuff distributed? I assume * > that the RA in the U.S. can take the lead in this role if we were to * > provide source. * > * > Thanks, * > Steve * > * > > > Is it OK if I added about 4 lines of code in enparse.pl to do Unix * > > > style backslash at the end of the line continuation? * > > * > > This is a great idea; I think it should be done. But... * > > * > > in order for this idea to be completely implemented, the change has to * > > be installed in Amsterdam, Reston, and Toronto as well as Ann Arbor; * > > and probably several places (ESNet, DSI, NASA, APNIC) that we aren't * > > sure of yet. * > > * > > Also, is enparse.pl part of any of the tools? Then it needs to be * > > changed in all the copies of those tools that anyone has FTP'd as * > > well.

1 0

Number of domain objects in RIPE database
by RIPE NCC Staff 04 May '95

04 May '95

Action: 20.11 RIPE NCC To investigate (as time permits) distribution of the domain information in the RIPE database and to investigate a "private" approach for referrals. Dear working group members, Relating to Action Item 20.11 from the previous RIPE meeting I have been asked to produce and send out some statistics about the number of domain objects in the RIPE database. You will find them below. I hope they are self-explanatory. Regards, Roderik Muit RIPE NCC AT : 132 objects, 25 different second-level domains BE : 481 objects, 458 different second-level domains BG : 28 objects, 28 different second-level domains CH : 709 objects, 709 different second-level domains COM : 5 objects, 5 different second-level domains CS : 4 objects, 4 different second-level domains CZ : 276 objects, 276 different second-level domains DE : 1512 objects, 1510 different second-level domains DK : 219 objects, 219 different second-level domains EE : 5 objects, 3 different second-level domains ES : 1 objects, 1 different second-level domains FI : 166 objects, 165 different second-level domains FO : 13 objects, 13 different second-level domains FR : 878 objects, 869 different second-level domains GA : 2 objects, 2 different second-level domains GL : 2 objects, 2 different second-level domains GR : 6 objects, 6 different second-level domains HU : 260 objects, 259 different second-level domains IE : 10 objects, 10 different second-level domains IL : 270 objects, 7 different second-level domains IS : 63 objects, 52 different second-level domains IT : 713 objects, 487 different second-level domains LI : 5 objects, 5 different second-level domains LT : 60 objects, 60 different second-level domains LU : 6 objects, 6 different second-level domains LV : 2 objects, 2 different second-level domains NET : 13 objects, 12 different second-level domains NL : 1559 objects, 1558 different second-level domains NO : 936 objects, 771 different second-level domains PL : 7 objects, 4 different second-level domains PT : 105 objects, 105 different second-level domains RO : 1 objects, 1 different second-level domains RU : 1 objects, 1 different second-level domains SE : 7 objects, 7 different second-level domains SK : 165 objects, 165 different second-level domains SU : 4 objects, 4 different second-level domains TN : 3 objects, 3 different second-level domains TR : 6 objects, 3 different second-level domains UA : 3 objects, 2 different second-level domains UK : 20 objects, 3 different second-level domains YU : 7 objects, 7 different second-level domains 164: 1 reverse delegations 192: 22 reverse delegations 193: 297 reverse delegations 194: 105 reverse delegations

1 0

Draft agenda DB WG meeting at RIPE 21, Rome
by Wilfried Woeber, UniVie/ACOnet 03 May '95

03 May '95

The *draft* agenda for the DB-WG at RIPE 21 in Rome. As always, additons, comments, etc. welcome! Apologies if I missed something obvious... Wilfried. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Draft Agenda: Database-WG, RIPE 21, Rome, IT ------------------------------------------------------ 0. Administrative stuff . who's taking the minutes? . WG-agenda 1. DB-SW review . new or proposed functionality . time-stamps . reverse links (for persons, at least) . distribution kit and integrating modifications . any loose ends? 2. Security . experiences with mail-from:, crypt-pw: . stronger authentication of updates (PGP?) 3. Review of beta test for RIPE-Handles . status 4. External interfaces . DB-exchange . first steps towards distributing the database 5. Input for and Review of objects . extensions for the router object (input from Mbone WG?) 6. AOB WW, 3.5.95 -------------------------------------------------------------------------- Wilfried Woeber : e-mail: Woeber(a)CC.UniVie.ac.at Computer Center - ACOnet : Vienna University : Tel: +43 1 4065822 355 Universitaetsstrasse 7 : Fax: +43 1 4065822 170 A-1010 Vienna, Austria, Europe : NIC: WW144 --------------------------------------------------------------------------

1 0

DNS vs. RIPE DB checking tool
by Havard Eidnes 12 Mar '95

12 Mar '95

Hi, as promised at the last RIPE meeting, please find below a copy of the perl script I have which checks for the consistency between what is registered in the RIPE database for the domains and what is registered in the DNS. Do note, this script does nothing more than report what it thinks is inconsistencies, you still need to sift through the error reports this script produces and fix any inconsistencies yourselves. This script relies on a number of conventions and is a bit picky about what needs to be in the RIPE database if it is to not complain too much. First, the script has to know which domains it should check. It performs this decision by "traversing" the domain objects in the RIPE database starting from the object for the top-level domain. If the object in question has at least one sub-dom: attbibute, my script assumes that all names registered in this domain should be registered in the RIPE database with it's own domain object. This way, you use the "sub-dom" attbiutes to span out a subset of the DNS hierarchy under a given top-level domain where administrative registration in the RIPE database is required for DNS registration. The script subsequently does zone transfers for the DNS domains corresponding to the domains which should be checked as decided above. When querying the RIPE database, the set of name server records for each domain is recorded, so that the script can compare these with what is registered in the DNS. This is one point where my script deviates slightly from or is slightly more strict than the RIPE-49 document, in that it 1) does not permit IP addresses in the nserver: attribute (since you do not delegate to an IP address in the DNS, this makes sense) 2) does not permit more than one word in each nserver: attribute This makes registration such as: nserver: aun.uninett.no / 129.241.1.99 illegal, and nserver: aun.uninett.no nserver: 129.241.1.99 would also be flagged as an error when compared to the DNS output. It is my beleif that the RIPE database does not adequatly support the registration of glue records (I do not think it needs to either), and putting the glue record information in together in the nserver: attribute is bad when seen from the database normalization and maintenance side. If a domain is registerd in the DNS with only MX information, my script demands that the corresponding registration in the RIPE database should contain the string "MX-only" in one of the remarks: attributes of the domain. Subsequently, what has been harvested via the RIPE database and what has been fetched from the DNS is compared, and any inconsistencies are reported. These are: 1) Check for domains registered in the RIPE database but not in the DNS, report these as errors 2) Check for domains registered in the DNS but not in the RIPE database, report these as errors 3) For delegated domains, report any inconsistencies between the set of nserver: attribute values and the NS records in the DNS 4) Check for domains with both MX and NS records in the DNS, report these as errors 5) Check that only domains marked in the RIPE database as "MX-only" really have an MX-only registration in the DNS. Lastly, this script depends on dig version 2.0 (forget the one which comes with BIND 4.9.3 -- the external interface changed for the worse, IMHO) and on a whois client which uses getopt() to parse it's arguments (so that I can use the '--' construct to separate the options and the arguments). If you have another version of whois than this, the fix should be trivial (at the top of "get_ripe_data". Note: the script creates a "cache file" with name cc.ripe-cache in the current directory containing all the domain objects without sub-dom attributes, so that multiple runs in quick succession can be a little faster (do however remember to remove any objects you sent in updates for, or remove the cache after substantial modifications). Well, other than this, read the script -- it's some 450 lines of simple straight-forward Perl code. - Håvard #!/local/bin/perl # $Id: domain-check.pl,v 1.7 1995/03/12 18:37:22 he Exp $ # # Perl script to check the consistency between what is registered in the # RIPE database and what is registered in the DNS for the domain objects # under a given domain. Missing pieces (i.e. inconsistencies) for either # registration is reported. # Pull in data for a given domain from the DNS using dig sub get_dns_data { local($domain) = @_; local($ns, $label, $zoneok); local($wildcard_label); $zoneok = 0; $ns = &get_primary_ns($domain); if (!defined($ns)) { &err("Could not get SOA for $domain\n"); return undef; } $digcmd = "dig @$ns $domain axfr +nodefname +pfset=0xa024"; $digcmd = $digcmd . " | tr '[A-Z]' '[a-z]' |"; open(dig, $digcmd) || return undef; while(<dig>) { chop; split; if ( $_[1] eq "in" ) { chop($_[0]); $label = $_[0]; $wildcard_label = 0; if ($label =~ /^\*\./) { $label = substr($label, 2); $wildcard_label = 1; } } else { if (( $_[1] eq "matching" && $_[0] eq ";" ) || ( $_[1] eq "received" && $_[0] eq ";;" && $_[3] eq "records" )) { $zoneok = 1; } next; } if ( $_[2] eq "a" ) { $dns_glued{$label} = 1; } elsif ( $_[2] eq "ns" ) { chop($_[3]); $dns_ns{$label} = $dns_ns{$label} . ":" . $_[3]; $dns_delegated{$label} = 1; $in_dns{$label} = 1; } elsif ( $_[2] eq "mx" ) { $dns_mx_domain{$label} = 1; if ($wildcard_label) { $dns_mx_wildcard{$label} = 1; } $in_dns{$label} = 1; } elsif ( $_[2] eq "cname" ) { $dns_cname{$label} = 1; } elsif ( $_[2] eq "soa" ) { # nothing to do, ignore } elsif ( $_[2] eq "hinfo" ) { # nothing to do, ignore } elsif ( $_[2] eq "wks" ) { # nothing to do, ignore } else { printf stderr "spurious data: $_\n"; } } close(dig); if ($zoneok) { return 1; } else { &err("No matching SOA record found from $ns\n"); return undef; } } sub get_primary_ns { local($domain) = @_; local($ns, $digcmd); $digcmd = "dig $domain soa +nodefname +aa +pfset=0xa020"; $digcmd = $digcmd . " | tr '[A-Z]' '[a-z]' |"; open(dig, $digcmd) || return undef; while(<dig>) { chop; split; if ( $_[1] eq "soa" ) { close(dig); return $_[2]; } } close(dig); return undef; } # Pull in any cached data from the RIPE database previously saved to disk # by get_ripe_data sub get_ripe_cached_data { local($domain) = @_; local($tag); open(cache, $ripe_cache_name) || return undef; while (<cache>) { tr/A-Z/a-z/; chop; split; chop($_[0]); $tag = $_[0]; if ( $tag eq "nserver" ) { $ripe_ns{$domain} = $ripe_ns{$domain} . ":" . $_[1]; if ( $#_ > 1 ) { &err("RIPE: more than one nserver on a line for $domain\n"); } } if ( $tag eq "domain" ) { $domain = $_[1]; $in_ripe_db{$domain} = 1; } if ( $tag eq "sub-dom" ) { shift; while ($subdom = shift) { $fqdn = $subdom . "." . $domain; push(domains_to_check, $fqdn); if ( ! $has_subdom{$domain} ) { $has_subdom{$domain} = 1; push(needs_dns_xfer, $domain); } } } if ( $tag eq "remarks" ) { if ( /mx-only/ ) { $ripe_mx_only{$domain} = 1; } } } close(cache); return 1; } # Pull in data for a given domain from the RIPE database using whois # Put received data in "ripe-cache" (append it). The assumption is that # get_ripe_data will not be called unless data isn't found in the cache. sub get_ripe_data { local($domain) = @_; local($tag, $subdom, $fqdn); local($do_cache) = 1; local(@entry, $s); $whoiscmd = "whois -h whois.ripe.net -- '-r '$domain |"; open(whois, $whoiscmd) || return undef; open(cache, ">>$ripe_cache_name") || undef $do_cache; while(<whois>) { if ( /No entries found for the selected source/ ) { close(whois); return undef; } if ($do_cache) { push(@entry, $_); } tr/A-Z/a-z/; chop; split; chop($_[0]); $tag = $_[0]; if ( $tag eq "nserver" ) { $ripe_ns{$domain} = $ripe_ns{$domain} . ":" . $_[1]; if ( $#_ > 1 ) { &err("RIPE: more than one nserver on a line for $domain\n"); } } if ( $tag eq "domain" ) { $in_ripe_db{$domain} = 1; } if ( $tag eq "sub-dom" ) { if ($do_cache) { close(cache); } undef $do_cache; # Do not cache entries with sub-dom entries shift; while ($subdom = shift) { $fqdn = $subdom . "." . $domain; push(domains_to_check, $fqdn); if ( ! $has_subdom{$domain} ) { $has_subdom{$domain} = 1; push(needs_dns_xfer, $domain); } } } if ( $tag eq "remarks" ) { if ( /mx-only/ ) { $ripe_mx_only{$domain} = 1; } } } close(whois); if ($do_cache) { while($s = shift(@entry)) { print cache $s; } close(cache); } return 1; } # Check existence of data in RIPE database for a given domain # (independent of sub-dom tree). Might as well use code above # which pulls in data from RIPE database entry (if it exists). sub in_ripe_db { local($domain) = @_; return &get_ripe_data($domain); } # Get all RIPE data (decending recursively through sub-dom) # for a given domain, noting registered sub-doms that do not have # their own entry in the RIPE database sub get_all_ripe_data { local($domain) = @_; local(@domains_to_check) = @_; &notify("Getting cached RIPE data for domain $domain:"); $ripe_cache_name = $domain . "." . "ripe-cache"; &get_ripe_cached_data($domain); &notify("\nGetting all RIPE DB data: "); while ($domain = shift(domains_to_check)) { if (! $in_ripe_db{$domain} ) { # Not already seen in cache &notify_progress(); if ( ! &get_ripe_data($domain) ) { &err("RIPE database entry missing for $domain\n"); } } } } # Pull inn all relevant data for doing a check. Decend through sub-dom # entries in the RIPE database and pull in data for the relevant DNS zones # as well. sub pull_in_data { local($domain) = @_; &get_all_ripe_data($domain); &notify("Getting corresponding DNS data\n"); while ( $domain = shift(needs_dns_xfer) ) { &notify("Getting zone data for $domain\n"); if (! &get_dns_data($domain) ) { &err("Zone transfer of $domain failed\n"); } } } # Perform an extensive consistency check between the data fetched from DNS # and those fetched from the RIPE database. In particular, check: # # 1) Check that all entries in the RIPE database are in the DNS and # vice versa; report inconsistencies. # # 2) Where entries exist in both RIPE and the DNS, check: # # a) for delegated domains, the set of NS records in DNS and those # registered in the RIPE nserver entries match up # # b) for domains with both NS and MX records in the DNS, report # this as an inconsistency # # c) for MX-only domains (in the DNS), check that the corresponding # entry in the RIPE database contains the remark "MX-only". sub do_consistency_check { &report_missing_ripe(); &report_missing_dns(); &report_ns_inconsistencies(); &report_inconsistent_domains(); &report_mx_comments(); } # Report entries in the DNS which do not exist in the RIPE database sub report_missing_ripe { local($domain); foreach $domain (keys %in_dns) { if ( ! $in_ripe_db{$domain} ) { if ( &in_ripe_db($domain) ) { &err("Missing as sub-dom in parent object: $domain\n"); } else { &err("Missing in RIPE db: $domain\n"); } $ignore{$domain} = 1; } } } # Report entries in the RIPE database missing in the DNS sub report_missing_dns { local($domain); foreach $domain (keys %in_ripe_db) { if ( ! $in_dns{$domain} ) { if ( ! $dns_cname{$domain} ) { &err("Missing in DNS: $domain\n"); $ignore{$domain} = 1; } } } } sub split_ns_list { local($ns_str) = @_; local(%array, @tmp, $elt); $ns_str =~ s/^://; @tmp = split(/[:]+/, $ns_str); while ($elt = shift(@tmp)) { $array{$elt} = 1; } return %array; } # Report inconsistencies between registered name servers DNS/RIPE sub report_ns_inconsistencies { local(%dns_nses, %ripe_nses); local($heading_written) = 0; local($domain, $ns); foreach $domain (keys %in_dns) { if ( ! $ignore{$domain} ) { %dns_nses = &split_ns_list($dns_ns{$domain}); %ripe_nses = &split_ns_list($ripe_ns{$domain}); foreach $ns (keys %dns_nses) { if ( ! $ripe_nses{$ns} ) { &err("Missing NS in RIPE for $domain: $ns\n"); } } foreach $ns (keys %ripe_nses) { if ( ! $dns_nses{$ns} ) { &err("Missing NS in DNS for $domain: $ns\n"); } } } } } # Report domains with both NS and MX records in the DNS # and domains in RIPE database with both nserver and remark: MX-only. sub report_inconsistent_domains { local($domain); foreach $domain (keys %in_dns) { if (defined($dns_ns{$domain}) && defined($dns_mx_domain{$domain})) { &err("Both MX and NS records for $domain in DNS\n"); } } foreach $domain (keys %in_ripe_db) { if (defined($ripe_mx_only{$domain}) && defined($ripe_ns{$domain})) { &err("Both nserver and remark: MX-only in RIPE db for $domain\n"); } } } # Report MX-only domains where RIPE entry doesn't contain remark "MX-only" # and MX-only domains in RIPE database which are fully delegated in DNS sub report_mx_comments { local($domain); foreach $domain (keys %in_dns) { if ($in_ripe_db{$domain}) { if ($dns_mx_domain{$domain}) { if (! $ripe_mx_only{$domain}) { &err("MX-domain in DNS, RIPE db not marked for $domain\n"); } } } } foreach $domain (keys %in_dns) { if ($dns_delegated{$domain}) { if ($ripe_mx_only{$domain}) { &err("MX-domain in RIPE, DNS fully delegated for $domain\n"); } } } } sub err { local(@args) = @_; if ($dot_written) { $dot_written = 0; printf stderr "\n"; } printf stderr @args; } sub notify { local(@args) = @_; if ($dot_written) { $dot_written = 0; printf stderr "\n"; } printf stderr @args; } sub notify_progress { printf stderr "."; $dot_written = 1; } $arg = $ARGV[0]; if (defined($arg) && ($arg =~ /^..$/)) { &pull_in_data($arg); &do_consistency_check(); } else { &err("usage: domain-check.pl cc\n"); &err("\twhere cc is a two-letter country code\n"); }

1 0

draft minutes, DB-WG, RIPE #20
by Wilfried Woeber, UniVie/ACOnet 07 Feb '95

07 Feb '95

Dear DB-WG and David! This is the draft of the minutes for the Database WG. Again many thanks to Andreas Knocke for providing me with the meeting notes in due time. David, I'd be grateful if you could do a spelling and sanity check on the text below. I'm writing too many minutes in a hurry these days :-( As always, any comments welcome. Best regards, Wilfried. PS: I'm off-line from 9.2.95 until 3.3.95, thus I'll be unable to respond to comments during the next 3 weeks. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Draft Minutes: RIPE #20, Database WG 1. Administrative stuff The agenda was agreed upon as proposed. Andreas Knocke (DE-NIC) volunteered to act as scribe for the meeting. 2. The "new" database software: review, future Report by Marten Terpstra: The new software had been announced at the RIPE 19 meeting in Lisboa. After being available for external testing it was put into production in October 1994. Since then some bugs were fixed and some (small) syntax checking features were added, users of the database should not have noticed the change - mostly. Recently the database was moved to a decicated (new) machine (Sparc/5, whois.ripe.net). As a general remark - Marten urged everybody to use the generic names for accessing the individual RIPE NCC services because a number of services (ftp, whois, www) have already been moved to different machines. Renumbering the RIPE network also required to renumber the machines. Documentation and future development of the software package shall be done by Daniel Karrenberg and David Kessens - due to the fact that Marten is leaving the RIPE NCC right after this meeting. Finally the official release will be available from ftp://ftp.ripe.net/ripe/dbase/software/ripe-dbase-1.00.tar.gz Other aspects of general interest: Time stamps (as requested by Merit and agreed upon in Lisbon) have not yet been added. Maybe Merit is going to add that functionality. Still there should only be one common version of the software. Hashed comments (# starting after white space) are allowed in any object submitted to the database although comments are *not* stored in the databse itself. While there was some discussion of allowing "end-of-line comments" this was dropped to avoid breaking scripts. Still an empty line (only white-space) is required to separate objects from each other. 3. RIPE Handles RIPE Handles are necessary to uniquely identify person objects with identical names and are required for the exchange of information with other databases, e.g. the InterNIC. Handles created for use in the RIPE environment database are very similar to the well-known NIC-Handles but have the string "-RIPE" appended to make them unique amongst different registries. Given the more distributed model of operation, the APNIC is going to append the relevant ISO 3166 country codes for the same purpose. Unique RIPE Handles can be obtained by fingering whois.ripe.net: finger handle.XY(a)whois.ripe.net where XY is the initials of the person's name to have a handle assigned. E.g. to request a handle for a given John Doe just do: finger handle.jd(a)whois.ripe.net which returns something similar to: First free handle: JD5-RIPE Using this (interim) method of obtaining unique handles introduces a possible race condition. This is yet to be solved. There are still some loose ends when a person object is updated to get a handle field. Occasionally there may be duplicate person objects created. In this situation the out-dated object can be simply deleted. RIPE Handles are currently considered to be in field test. On a more general level it was proposed to investigate the legal aspects of storing person-related information in a publicly accessible database. 4. External interfaces The SWIP (Shared Whois Project) for exchanging information amongst the (Regional) Registries requires unique handles. The rwhois project (see RFC 1714) - Mark Kosters of InterNIC briefly reported on the resources available (he is working on it in his spare time). A final beta version might be available at the end of Q1 1995. 5. Review of objects and links Inverse lookups: Inverse lookup is very desirable easily to find out where all the references to a given object (persons, but maybe inetnum objects as well) are coming from. Action 20.DB1: on the RIPE NCC, to do an implementation analysis. "status:" attribute Geert Jan de Groot explained that this attribute was indeed required for keeping track of address assignments. Marten had agreed to implement it on the fly to support a new tool for NCC internal use. As implemented, the acceptable values for this attribute are "Delegated" "Assigned" (and "Reserved"). The little sketch below depicts the model: total delegated delegated assigned address to RIPE to local-IR to customer space 193/7 19?.*/16 I I I / I / I I / I / I / I I/ I/ I/ I I\ I\ I\ I I \ I \ I \ I I \ I \ I I I 6. Meta objects Contrary to the current recommendation the person object is "mis-"used by a couple of entities to describe role accounts and similar things. Daniel Karrenberg agreed that this can indeed be useful, even if he opposed this use in the past. However "admin-c" for an object still has to point to a real person object. Action 20.DB2: on HAvard Eidnes to write up a proposal with input from the Local-IR WG and the EOF (European Operators Forum) 7. Input fromt the DNS-WG: domain object In addition to a brief report on the decisions within the DNS-WG, Piet Beertema elaborated on the information scheme for keeping track of domains domains under the TLD ".nl" w2hich has been implemented. Piet intends to remove all objects other than the one for .nl itself. This proposal calls for a forwarding pointer in the RIPE-DB that links to the machine where the data is actually available (the name server for .nl). Questions discussed and issues arising from this proposal: - How can compatible schemas be maintained? With the updated domain object this can be achieved, although the representation for persons are different - tools have to "keep this in mind". - The referral mechanism . The rwhois package is not yet ready for production use and has to be extended for CIDRization (whois++ the same), . The RIPE DB software currently lacks the concept of forwarding queries (comparable to proxy forwarding in BIND) for forwarding to other whois servers. There was a general consensus that the domain object is a very good candidate for starting a pilot to distribute information to the environment where it is maintained. With regard to person objects, the allocation registry and the routing registry, the RIPE database should not be modified right now (production use and automatic configurations!). Action 20.DB3: on the RIPE NCC to investigate (as time permits) distribution of the domain information and to investigate a "private" approach for referrals. Havard Eidnes reported on a tool to check the primary zone file for .no against the registered domain objects. Action 20.DB4: on Havard Eidnes to publish this tool for the benefit of other top level registrars 8. AOB - any other business None, thus the WG meeting was closed. Z. Report on an out of band discussion for the minutes "An additional, informal BOF happened just before the lunch break - inheritance of authorisation was proposed and discussed for hierarchical objects which would resolve a common question: the authorisation of allocations from a delegated block and of the addition of subdomains within a registered top level domain." -------------------------------------------------------------------------- Wilfried Woeber : e-mail: Woeber(a)CC.UniVie.ac.at Computer Center - ACOnet : Vienna University : Tel: +43 1 4065822 355 Universitaetsstrasse 7 : Fax: +43 1 4065822 170 A-1010 Vienna, Austria, Europe : NIC: WW144 --------------------------------------------------------------------------

1 0