- db-wg - mailman.ripe.net

1st report on db indexing problems
by Geert Jan de Groot 21 Sep '95

21 Sep '95

This was sent to me; in order to be able to see patterns I think it is useful to post this to the list. many thanks to Arnold for the report; we do our best. Again: will people who have problems with too large database index files please step forward? Please give us a chance to break our database too (the test-version, that is) Geert Jan ------- Forwarded Message From: Arnold Nipper <nipper(a)xlink.net> We are using perl.c,v $$Revision: 4.0.1.8. Currently I try to index with perl, version 5.000. We are using: adderror.pl (1.1) addkey.pl (0.16) cldb.pl (1.10) cleandb.pl (0.44) cleanhandledb.pl (1.1) dbadd.pl (0.23) dbclose.pl (0.14) dblock.pl (0.13) dbmatch.pl (1.2) dbopen.pl (0.15) dbupdate.pl (0.34) defaults.pl (0.13) defines.pl (0.18) donetdbm.pl (1.5) encmp.pl (0.13) endiffs.pl (0.12) enkeys.pl (0.18) enparse.pl (1.8) enread.pl (0.17) entype.pl (0.13) enukey.pl (0.13) enwrite.pl (0.23) handle.pl (0.13) maintainer.pl (1.6) misc.pl (0.45) net2net.pl (0.14) netdbm.pl (0.15) newdb.pl (0.13) notify.pl (1.8) rconf.pl (0.34) rfc822.pl (0.18) showdbm.pl (0.12) sortdb.pl (0.17) syntax.pl (0.55) syslog.pl (0.14) template.pl (1.4) Warning: time.pl is not a RCS file updatecheck.pl (1.20) whoisd.pl (0.52) Machine is a Sparc10 clone running SunOS Release 4.1.4 with more or less standard kernel (only modified udp_checksum and MTU) Regards, Arnold P.S.: I know that DE-NIC has the same problem with indexing the DB.

1 0

Re: http://www.rmnet.it/cgi-bin/whois (improved)
by Paolo Bevilacqua 21 Sep '95

21 Sep '95

At 11:14 AM 9/20/95 DST, Antonio_Blasco Bonito wrote: > ... >1- some additional source value will be useful in the source popup menu > (for example MCI and CANET) >2- domain is missing from the query-type popup menu > I have added these values. If someone knows other let me know. >3- if you specify server-to-search/other than some options are not used, > namely -s (source) and -r (do not recurse) > This is wanted, because 'non-RIPE' servers won't understand such options. Until now i've found RIPE and APNIC running this kind of code, others can be easily added. Greetings, /pab -- Paolo Bevilacqua RMnet Communications ph/fax: +39 6 85302774 http://www.rmnet.it

3 3

http://www.rmnet.it/cgi-bin/whois (improved)
by Paolo Bevilacqua 20 Sep '95

20 Sep '95

Hi, I've slighty modified this form, now: - as somebody suggested non standard ports are now supported. A patched version of the whois client with alternate port is at http://www.rmnet.it/cgi-bin/whois.c - consistent security checks on client execution and error return. - minor cosmetics. Greetings all, /pab

2 1

WWWhois
by Paolo Bevilacqua 18 Sep '95

18 Sep '95

Hello, An HTML form to query the database is running at: http://www.rmnet.it/cgi-bin/whois Comments to me, improvements to the code. /pab

1 0

New Document available: ripe-130
by RIPE NCC Document Annoucement Service 29 Aug '95

29 Aug '95

New/Revised RIPE Document Announcement -------------------------------------- A revised/new document is available from the RIPE document store. Ref: ripe-130 Title: The advisory attribute Author: David Kessens, RIPE NCC Date: 1 August 1995 Format: TXT=3230 bytes Obsoletes: Obsoleted by: Updates: ripe-181 Updated by: Old: Short content description ------------------------- This document describes the advisory attribute of the route object. FTP Access ---------- All RIPE documents and Internet RFC`s are available via anonymous FTP from host ftp.ripe.net. Type "ftp ftp.ripe.net". Login with username "anonymous" supplying your email address as the password. After logging in, type "cd ripe/docs/" followed by the command "get filename". The relevant filenames for this document are: ripe-130.txt for the ASCII version ripe-130.ps for the PostScript version Electronic Mail Retrieval of Documents -------------------------------------- Documents can also be retrieved from the RIPE document store using a mail server program. For more information on how to use the program, send email to: mail-server(a)ripe.net with "send HELP" in the body text. RIPE NCC Interactive Information Server --------------------------------------- Type "telnet info.ripe.net". This is a menu driven service allows the document store to be browsed. After reading documents you are prompted as to whether you would like to receive an email copy of the document you have just read. If you would, you simply enter your email address and the document will be mailed to you. Below are details of alternative methods of access. Gopher Access ------------- The same documents are available via a "gopher" server at "gopher gopher.ripe.net". WAIS Access ----------- There is also a "WAIS" server at wais.ripe.net, where there is a WAIS index for RIPE documents "ripe-docs.src" WWW Access ---------- For those who wish to add this home page at the RIPE NCC to their own customized home pages, it can be accessed as: http://www.ripe.net MIME Mail Reader ---------------- Below is the data which will enable a MIME compliant Mail Reader implementation to automatically retrieve the RIPE document by FTP or mail server. SEND ripe/docs/ripe-130.txt

1 0

RIPE automatic inaddr-arpa checking tool available
by RIPE NCC IN-ADDR. ARPA Role Account 21 Jul '95

21 Jul '95

Dear all, After many requests I have published the inaddr-arpa checking tool I developed at the RIPE NCC. We use it now at RIPE NCC quite for some time and it really saves us a lot of time, although the program is not perfect and still needs some polishing. Beware of changing/adapting the tool too much since I expect to make some big changes in the output format to facilitate further automatic processing. Furthermore, I want to split it in some smaller files and a config file for better maintainability. Of course comments and bug reports are welcome. Please read the included README file for more details. Kind regards, David Kessens RIPE NCC PS I am on vacation during the next two weeks, but I will certainly deal with any comments/bug reports when I return! -------- README for the inaddrtool v0.1 Date: 950719 Author: David Kessens, RIPE NCC NOTE: THE TOOL HAS BEEN MADE PUBLICLY AVAILABLE BECAUSE OF THE LARGE NUMBER OF REQUESTS OF OUR CUSTOMERS TO DO SO. THE TOOL ITSELF HAS PROVEN IT'S USEFULNESS, BUT NOTE THE FACT THAT I EXPECT TO MAKE SOME BIG CHANGES BEFORE RELEASING A FINAL VERSION. ALSO SOME ADD-ON PROGRAMS THAT WILL MAKE IT POSSIBLE TO ADD THE REVERSE DELEGATION CHANGES/REQUESTS AUTOMATICALLY TO (Y)OUR ZONEFILES ARE STILL MISSING. THE PROGRAM IS DESIGNED AS SITE INDEPENT AS POSSIBLE. PLEASE USE THE NORMAL PROCEDURE AS DESCRIBED IN RIPE-105 IF YOU WANT TO DO A REVERSE DELEGATION REQUEST FOR A REVERSE DOMAIN MANAGED BY THE RIPE NCC, BUT SEND YOUR REQUEST TO <AUTO-INADDR(a)RIPE.NET> INSTEAD OF <HOSTMASTER(a)RIPE.NET> Reverse delegation tool (Alpha version!) ---------------------------------------- As of today the RIPE NCC offers an automated method for the submission of reverse zone delegations in 193.in-addr.arpa and 194.in-addr.arpa. The reverse delegation requests and the zone files on all nameservers will be checked automatically. The diagnostics generated by these checks will be returned to you automatically too. This will make you aware of any problems very quickly, so that you can correct them and re-submit your request. The most recent version of the tool is now publicly available at: ftp://ftp.ripe.net/tools/inaddrtool0.1.tar.gz The tool uses some other external programs for gathering the information it needs for checking your zone files: ftp://ftp.ripe.net/tools/ping.tar.Z ftp://ftp.ripe.net/tools/dns/host.tar.Z ftp://ftp.ripe.net/pride/tools/prtraceroute-2.0beta3.shar.gz Installing the software: - uncompress/unzip and untar the tool and the external programs - edit the variables in the source code (if needed): $TESTMODE=1 for testing, 0 for normal use $UPDLOGDIR="Directory for logging the incoming requests"; $ACKLOGDIR="Directory for logging the outgoing acknowledgements"; $FWLOGDIR="Directory for logging the outgoing approved requests"; $MSGQUEUEDIR="Directory for temporarily storing the requests"; $NSNAME="The name of your nameserver"; $NAMESERVERCHECK=; see for details the source code $MAILCMD=; see for details the source code $NICECMD="Your nice program if you want" else use ""; $HUMANMAIL="The E-mail address of the human processing the approved requests and answering questions. When $TESTMODE=1, all mail will be send to this address"; $AUTOMAIL="The E-mail address of the mail box that will auto process incoming requests"; - Put something like: "|/home/user/bin/inaddrtool 2>/dev/null" in the .forward file of user $AUTOMAIL Note: there is *NO* queuing mechanism yet, so beware of overloading your machine with a large number of requests. The input: The tool expects to read an E-mail message from standard input although there is no problem when using it stand-alone. The E-mail message should contain a valid RIPE database object as described in the ripe-105 (ftp://ftp.ripe.net/ripe/docs/ripe-105) procedure. When the automated procedure does not detect any errors, the request is forwarded to the $HUMANMAIL role account person for some additional manual checks and the processing of the delegation itself. An acknowledgement of this fact is also sent to the people mentioned in the From:/Reply-To: and Cc: field in the E-mail message. The tool will return an error report if errors are found. If $TESTMODE=1 all mail will be send to $HUMANMAIL. You can use some keywords in the 'Subject:' line of your E-mail to control the checking process. The use of the LONGACK keyword is very recommended. HELP - will send you a (patched) ripe-105 document CHANGE - is needed if you want to change an existing reverse delegation LONGACK - will give you the most verbose output as possible TEST - do the checks, but sent only a report back to the user even if no errors are found You also might want to use the special variable NSNAME that's documented in the source code itself (for experts only). RIPE document ripe-105 requires you to send in a RIPE database 'inetnum' object with a 'rev-srv' attribute for each nameserver for single/multiple C's reverse delegation requests and for whole blocks 'domain' objects with 'nserver' attributes for each nameserver. I am neither a DNS expert or native English writer ;-) so all your comments are welcome! Please send them together with complaints, bug reports or special requests to <inaddr(a)ripe.net>.

1 0

TEST database
by RIPE Database Manager 20 Jul '95

20 Jul '95

Dear all, Last week, I have installed a test database on test-whois.ripe.net. This database is intended for people who want to play around with some test database objects and to learn how the RIPE database works. Updates should be send to: <testdb(a)ripe.net> Do queries with: whois -h test-whois.ripe.net SearchString The creation of maintainers is allowed for everybody for your (and my) convenience but the software will warn you that maintainer creation is normally done by human intervention. It is possible to update two different databases with source names: source: TEST OR source: TEST2 Of course, I might use this database for some tests I do myself with the database software so don't be surprised to see some new features that are not present in the production database, Kind regards, David Kessens RIPE NCC Database maintainer --------

1 0

Towards a better distributed database model..
by Cengiz Alaettinoglu 16 Jul '95

16 Jul '95

Hi, Here is a draft David and I put together. Comments/suggestios are welcome. Thanks. Cengiz -- Cengiz Alaettinoglu Information Sciences Institute (310) 822-1511 University of Southern California http://www.isi.edu/div7/people/cengiz.home ================================================== Draft: 4 Working Title: Towards a better distributed database model.. Date: 950707 Authors: David Kessens Cengiz Alaettinoglu Introduction: ------------- This document describes the new proposed synchronization and distribution model. Note that we state here distribution model, not distributed model. This builds further on the proposed synchronized database model for the short term. For convenience this model is also briefly discussed below. Current model: -------------- Internet Routing Registry (IRR) consists of several registries. Each registry maintains a separate and distinct database. Each database contains a set of objects (e.g. route objects, aut-num objects [Ripe-181]) which are identified by a key. Each object in a database also has a "source" attribute which is not a part of the object's key and identifies the registry whose database contains this object. Same object may occur in multiple databases (of different registries) with conflicts. However, the two objects when presented to a user (e.g. upon a whois query) will have different source attributes, and the user can determine which registry's data to trust more. Unfortunately, with the current model, the databases contain a lot of inconsistent objects. Hence, the current model has proven to be inadequate. The databases maintained by each registry are not distributed in any form. Registries keep a mirror copy of the databases of other registries and search for objects in all of the databases when replying to user queries. The mirroring is done by ftping the databases once in a predetermined interval (currently once a day). The following short term solution for getting a more synchronized database mirroring is currently being implemented at RIPE by the first author and is described below. However, we believe that this solution is not sufficient for the long term needs. Every registry makes once in a predetermined interval its full database available at a well known ftp site. In addition, it maintains and makes available a differences file (described below) which records the updates to its database since the last ftp-copy of its database was made available. When an update request is received by a registry, the database is updated if the object is syntactically correct and the registry is the registry specified in the source attribute. If the registry is not the registry specified in the source attribute, the update is forwarded to the correct registry or refused and returned to the user. Once the update is processed, a transaction is written to the differences file(s). It is possible to immediately forward the transactions to the other registries to keep the mirror copies up-to-date. Or once in a predefined interval, the registries may check the differences files of other registries and resynchronize if necessary. The full database copies (ftped periodically) and differences files enable one to access the states of the databases at a user specified time. This feature is often desired for configuring routers. Towards the long term solution: ------------------------------- For a long term solution the following preconditions are set: Transparency: We expect the number of registries to grow significantly. This means that it will be increasingly more difficult for users to keep track of every registry. Therefore, it should be possible for users to interact only with one registry while accessing and registering data and transparently getting the information from all different registries (this may involve a referral mechanism in the client programs and protocols) or transparently registering data in the database of the authoritative registry. Scalability: The current (synchronized short term) database/registry model requires each registry to obtain each other's databases and differences file(s) by some mechanism such as ftp. This requires NxN ftp (or other protocol) peering, where N is the number of registries, which does not scale. The current model has no hierarchy or other methods for scaling. Reliability/robustness: The model should be as reliable and as robust as possible. Since the system will run in an in principle evil and unreliable network, special measures for security should be built in from the start. Update and access speed: Updates should propagate as fast as possible. Users access and update the data interactively and do not accept that the updates take more then few seconds (and will not tolerate if the updates are slower in the new system than the current system). However, it may be acceptable to take more time to update the mirror copies, or it may be acceptable to take more time to update certain object which are not updated frequently (e.g. maintainer objects). Since several programs that use the system now need a (nearly) real-time access, we cannot sacrifice any performance. Distribution versus distributed: The IRR data is used for configuring routers. To be able to do this, two requirements are important: First, at least the maintainer of a database should be able to access the database in the current state but also at other time and date instances specified by her/him (earlier, but may be even later in time with delayed updates). Second, all data should be retrievable at each state, not just part of it. These requirements speak clearly against aggressive distribution of the data since there will always be problems with connectivity with some remote hosts and we need to be able to gather all data without any exceptions. Synchronization: It is desirable to have the databases and their mirrors as synchronized as possible. However, if this requirement slows the update speeds considerably, it can be compromised. The proposed model: ------------------- Why not DNS: Several people have suggested to use DNS. However, there are several problems with using the DNS: DNS is a very complex system and difficult to maintain. It does not support the needed ability to get the state of the database at a certain point in time. It does not support searches on arbitrary attributes such as locating all routes with community HEPNET or finding the person objects with name 'David Kessens' within a reasonable amount of resources and nearly real-time response. Furthermore, tools like prtraceroute have their good performance by accessing all the data from one nearby registry. Since DNS has no easy way of collecting all (without anything missing) the data from all the hosts on the Internet, tool performances would degrade. Furthermore, DNS security problems are not solved. Of course DNS is a working system in the real world, and we will benefit from it in designing our registry/database architecture. Distribution Topology: Full mesh ftp (or other protocol) peering of current model does not scale. A hierarchical peering topology where each registry only receives and passes data to and from its children and parent would scale to the large number of registries. However, a hierarchical peering topology has robustness problems since a down registry would make the complete subtree beneath it disconnected. These considerations led us to a non-hierarchical graph peering of registries. In this graph, each node represents a registry, and a link represents an ftp (or other protocol) peering. There are no roots. A registry can become neighbors with any other registry provided that the graph remains connected (perhaps bi-connected). Each registry only communicates with its neighbor registries. The neighboring registries exchange all data from all known registries. This makes it very easy to add new registries to the IRR: just become a neighbor of an existing registry. Consistency: ------------ Current model allows registering an object with the same key in two registries with conflicts. The consistency of the information is very important since this data is being used to configure operational routers and to diagnose operational routing problems. Currently, conflicts are being resolved by manual inspection which is very slow and tedious. One way to achieve consistency is to have a true distributed database with atomicity and synchronization. However, this solution does not meet our preconditions listed above (for example updates would be very slow, perhaps at the order of days, when there are network partitions). In the new model, we are debating between two solutions that should be able to avoid this conflicts. The first solution is based on multi-valued source attributes. The second solution is based on removing the source attribute altogether. Registries and multi-valued source attributes: Registries will store a new object which describes the registry. The registry description may contain information on which type of objects that are stored locally. If a request comes for information that might be stored in another registry, the requester will be referred to that registry. The client programs should cache this referral information (not the actual data returned!). The source attribute of an object now contains a list of registry names. An object can only be updated if all the registries mentioned in this attribute authorize that. The source attribute has no other semantics. This may require a two phase commit protocol run between the registries mentioned in the source attribute. A special case of this is when the source attribute contains only one registry which is identical to the current model. There can be other implicit semantics in the source attribute. Some registries might require to be one of the sources for certain objects. Also the type of object might require (as defined in the object definition) that object updates should be approved by some additional source(s) (the authoritative sources). Updating: The update process will be controlled by a modified two phase commit protocol. The user will send an update request to one of the registries. The registry will send the request to all the registries specified in the object's source attribute and to additional registries if required by the definition of the object's type. After the registries determine whether to accept the object, the first registry will acknowledge this to the user. After this, all the involved registries (including the one that received the request) will update their local database(s), and flood the update (with serial&timestamp of first source&signature) to the other registries. No source attributes: This approach views the whole IRR data as being one database (though it may still be maintained in pieces) with certain registries authoritative on certain parts of it. Since it is one database, there is no need for the source attribute to identify which database, and no two objects with the same key and conflicting information can be registered (i.e. one of them will overwrite the other one). Our solution is as follows: Maintainer objects are globally unique and are created if all (or an authoritative subset) registries approve it. This can be achieved by a two phase commit protocol. Note that the creation of the maintainer objects will be slow (and is slow currently since it is done manually). However, this is tolerable since maintainer objects are not created very often. Each maintainer object specifies a registry which is authoritative for the objects that are maintained by itself. When a user sends an update to a registry, that registry will forward it to the registry which is authoritative for the object. The authoritative registry updates its local copy of the database and invokes the flooding protocol for other registries to pick up this change. Note that, if an object is registered in the database with maintainer X who authorizes registry Y, all updates will be performed by registry Y. This will ensure consistency within Y's copy of the database. If someone tries to register the same object with a different maintainer than X, Y (or any other registry) will deny the update with a maintainer failure error. There is one problem with this approach. If the mirroring takes a long time, within that period, one can create two objects with the same key, different maintainers which authorize different registries (and possibly with conflicts) in the local databases of two registries. Hence, it is essential to keep the mirroring as synchronized as possible. In this case, when the next mirroring is done and the conflict is detected, the copy which is created later is removed and returned to the user. We think that the probability of this case happening will be small.

3 3

Re: Towards a better distributed database model..
by martin＠terena.nl 14 Jul '95

14 Jul '95

Maybe I'm barking up the wrong tree, (please let me know!), but seems to me that one could use WHOIS++ for this sort of distribution. (Yes, I _know_ it is distributed rather than distribution model). Bunyip have recently released 'digger' a WHOIS++ implementation: http://services.bunyip.com:8000/products/digger/digger.html Rgds, John

1 0

Draft Minutes RIPE21, Database-WG
by Wilfried Woeber, UniVie/ACOnet 02 Jun '95

02 Jun '95

This is the draft minutes for the DB-WG meeting at RIPE 21 in Rome. They are quite belated (and probably full of typos), but unfortunately nobody volunteered this time to act as a scribe... Comments, corrections, additions -as always- very welcome. Wilfried. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Draft Minutes: RIPE 21, Database-WG Rome, 8.5.1995 Chair: Wilfried Woeber, Scribe: Wilfried Woeber ------------------------------------------------------ 0. Administrative stuff Unfortunately nobody volunteered to take the minutes. I had to guide the discussions and to take the notes at the same time. Thus these minutes are probably not as comprehensive as we are used to. Apologies. The agenda was accepted as proposed, without changes. 1. DB-SW review David Kessens, RIPE NCC, reported on the current state of the database software and the environment at the NCC for providing maintenance and doing development work. Activities since RIPE 20 (as Marten and Tony having left the NCC/PRIDE Project) centered on becoming familiar with the software, performing bug fixes and implementing minor extensions, like the "network update" feature whois -U and whois -tv (for template verbose). Other aspects worth mentioning: . reverse lookup technology . hierarchical authorization (initially for domains) . referrals are being researched and estimates for the necessary implementation efforts are obtained. David confirmed that the official distributioin kit available (anonymous FTP) at the NCC always contains the most recent fixes issued and additions. For more details please refer to the slides supporting David's presentation which are available from the RIPE NCC FTP Server in the presentations directory ftp://ftp.ripe.net/ripe/presentations/ripe-m21-david-DB-REPORT.ps.Z Merit is also working on the databse software and has produced some tools. These things are available from: ftp://ftp.ra.net/routing.arbiter/tools/RAToolSet 2. Security While there was no concern regarding the functionality and usefulness of the authorisation methods already implemented (mail-from:, crypt-pw:) there is a need for even stronger authentication of updates. PGP seems the way to go, although there is still quite a bunch of open questions and a need for logistic (e.g. key distribution) development. It is not clear at the moment whether this should be integrated (DB, RIPE NCC) or be provided by mechanisms outside the scope of RIPE. SURFnet reported it's intent to use the RIPE-DB for routing configuration as soon as PGP would be available to provide the data reliability necessary. Others expressed strong interest, as well. Merit (LJ) did part of the work already to provide PGP authorisation for the email addresses in updates. Comments from the group stressed that it is essential to *not* mix the concepts of authentication and of encryption. Encryption is a logistic and/or legal problem in some regions. The group is requested to persue this aspect on the relevant mailing lists. *Action on David Kessens and Wilfried Woeber: Investigate the deployment of PGP for the RIPE Database and to draft a proposal. 3. Review of beta test for RIPE-Handles The method for generating RIPE-Handles was discussed once again, there are still open issues when trying to fully automate handle assignement because this would require both a DB-commit function as well as require the modification of submitted objects on-the-fly. The group did not resolve this issues, thus the current procedure (finger) remains in place. It was noted that the scripts involved should be made available to all interested parties. 4. External interfaces There is an on-going discussion (in the light of more than one organisation using the RIPE DB-SW now) on how to keep the various databases in synch. One of the ideas is to accept updates for each and every database on all interfaces, to analyze the "source:" attribute and then to automatically forward the update to the database as indicated by "source:". While this is probably quite interesting, it might not always be what the users intend. The alternative is to reject updates when submitted for the "wrong" "source:". This is the safe way. To meet the need of regularly transferring updates from one database to (an)other(s), a method using FTP and providing "deltas" is investigated. 5. Input for and review of objects Due to the scheduling of the working groups, there was no input at the DB-WG meeting. A proposal to expand the internet-router object to describe Mbone routing was circulated after the meeting, as drafted by the Mbone WG. 6. AOB None. Thus the meeting was closed. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ WW. 2.6.95

1 0