- db-wg - mailman.ripe.net

Re: Object Metadata Proposal
by Janos Zsako 15 Nov '01

15 Nov '01

> From owner-db-wg(a)ripe.net Thu Nov 8 17:53:37 2001 > From: Andrei Robachevsky <andrei(a)ripe.net> Dear Andrei, I think both proposals (this one and the garbage collection) are very good and very useful. I just have a further suggestion and a question (I apologize if this has already been discussed). > Metadata Contents > ------------------ > Other meta-attributes may be added as needed. My suggestion would be to add a meta-attribute "creation-time:" which would store the date and time the object was created (as opposed to "update-time:", which only shows when it was last updated). This attribute would never change. The rationale behind having this attribute is that one would have a proof of the existence of the object in the database from (at least) creation-date (until at least query time). This may help for example auditing LIR assignments to the same end-user (which is limited to one AW worth in any 12 months). My question would be about the generation of the initial values (of update-time and ref-time) for the _existing_ objects. One aproach would be to use the "changed:" value for "update-time", and for "ref-time" the most recent "changed" value of the objects referencing the given object. The only problem is that we know "changed" is not (very) reliable. An other approach would be to use the time of the (initial) creation of the meta-attributes as initial value both for "update-time" and "ref-time". This is probably misleading. A better approach would probably be to leave these values empty, but this would not meet the current spec. The most exact approach would be to retrieve the exact values from the update logs, but this is probably too expensive. My question is which approach do you plan to take (one of the above or one I did not think about)? Thanks and regards, Janos

2 1

automatic DB cleanup proposal (v2)
by Andrei Robachevsky 08 Nov '01

08 Nov '01

Dear Colleagues, This is a slightly modified version of Ulf Kieber's proposal based now on metadata concept (please see "Object Metadata" proposal on the list next to this one). Your comments and suggestions are welcome. Thanks, Andrei Robachevsky RIPE NCC ----------------------------------------------------- Automatic DB Cleanup proposal ============================= Motivation ---------- In the light of data privacy issues raised by governmental organizations and in RIPE Database Working Group sessions during RIPE-39 and RIPE-40, as well as following the Great Dangling Person Objects Deletion after de.* ccTLD move-out it is necessary to implement an automized garbage collection procedure to remove personal data that is unreferenced for a certain amount of time. Proposed Change --------------- Four meta-attibutes "update-time:", "ref-time:", "ref-count:" and "expire-time:" should be maintained and used for this purpose (*). The values of these meta-attributes would be a modification timestamp, reference counter modification timestamp, reference counter and object expiration time (if applicable) respectively. The reference counter would contain the number of objects still refering to this object, and thus be similar to the link counter of an inode in a UNIX file system. The modification timestamp would be updated whenever the object is updated. The reference counter modification timestamp would be updated whenever link counter changes. The analogy in the UNIX file system would be sort of mtime and ctime. The expiration time will be updated for person and role object only. For other object types it will be emplty meaning "not set". The expiration time will be updated whenever link counter reaches zero or changes from zero to a positive value. In the first case the expiration time will be calculated by adding the expiration period to the time when this event occured. In the second case the expiration time will be unset (empty value). A garbage collection process running on a periodic basis should then generate a deletion update (transaction) overriding authorization checks (**) for all person and role objects that reached their expiration time. The expiration period is to be discussed and agreed upon in the community. The initial proposal is 90 days. Explicit generation of an update (transaction) is needed for tracking/debugging purposes. (*) Please see the "Object Metadata" proposal. (**) Overriding authorization allows deletion of a maintained object to be performed. Respective maintainers will be notified. Overriding authorization still doesn't allow deletion of a referenced object.

1 0

Object Metadata Proposal
by Andrei Robachevsky 08 Nov '01

08 Nov '01

Dear Colleagues, Please review the proposal below. This proposal is supposed to provide means for reliable timestamping, and also support garbage collection procedure (please see "Automatic DB cleanup proposal" following this one). But of course the functionality can be expanded when needed. Your comments are welcome as always, Best regards, Andrei Robachevsky RIPE NCC ------------------------------------------------------------- Object metadata proposal ======================== Motivation ---------- Several proposals already exist that require reliable status information about an object in the database, such as last update time, number of references, etc. These are requests for reliable timestamping and automatic cleanup procedure. In order to be reliable this information should be automatically generated by the database software and be outside users' control. Storing this information as part of the object data was generally considered not a good idea since this requires modifications to the object not solicited by the user, and breaks some basic principles of the RIPE Database. But in fact these attributes/facilities do not represent an object, but rather status information about the object. Extending Ulf Kieber's analogy with the Unix FS, they are part of the inode, not the data blocks. Object metadata --------------- Status information, or object metadata, will be automatically generated by the database, but stored and accessed separately from the object itself. This will not affect any object template, preserve users' data and add flexibility to the RIPE Database. Metadata Contents ------------------ Metadata contents will be presented as metadata attribute-value pairs, or meta-attributes. The initial set of meta-attributes is required by the above mentioned proposals and will include: update-time: <YYYY-MM-DD> <HH:MM:SS> This meta-attribute will store the time when the update (object creation or subsequent updates) was processed by the database software (when modification was made to the Database). ref-count: <number> The number of references to this object from other objects in the RIPE Database. Zero means the object is not referenced. ref-time: <YYYY-MM-DD> <HH:MM:SS> Last time the "ref-count:" was updated. It is empty if ref-count has never been updated. expire-time: <YYYY-MM-DD> <HH:MM:SS> This meta-attribute displays the time after which the object will be automatically deleted from the database overriding authorisation checks. Empty value of the meta-attribute means "not set". Please see "Automatic DB cleanup" proposal for more information. Other meta-attributes may be added as needed. Metadata will not propagate through NRTM. It is up to a NRTM client to generate the metadata as updates are processed. Querying Metadata ------------------ Object metatdata may be accessed by using "-q meta" flag when interrogating the database. This flag is can be used together with other queries, except other "-q" flags. Metadata of all objects returned will be displayed. An object metadata will be displayed after the object with a blank like between them. It is displayed as meta-attributes preceded by a '%' sign. According to RIPE whois definition a line starting with '%' sign should be generally considered as a comment and should be parsed at client's own risk. For example: $ whois -q meta RC1234-RIPE ... person: Referenced Contact nic-hdl: RC1234-RIPE ... changed: refcon(a)somedom.dom 20011005 source: RIPE %metainfo: RC1234-RIPE %update-time: 2001-10-05 16:50:07 %ref-time: 2001-10-06 09:17:33 %expire-time: %ref-count: 3 Using Metadata -------------- The metadata may be consulted by humans or metadata-aware clients for different purposes. The database software may use metadata for its garbage collection routine, or other internal processes.

1 0

LIR-PARTITIONED - Final version
by Nurani Nimpuno 23 Oct '01

23 Oct '01

Dear all, Following the announcement 28 Sep 2001 on these lists, please find below the final details of the LIR-PARTITIONED PROPOSAL. This proposal will be implemented by the RIPE NCC in the coming period and presented at the RIPE 41 meeting in January 2002. Comments and suggestions made on this list have been incorporated in the proposal below. +-----------------+ | LIR-PARTITIONED | +-----------------+ Background ----------- This proposal (initially named "LIR-ALLOCATED") was originally discussed at September 1999. James Aldridge later sent out a first proposal to the LIR-WG mailing list, 14 Jan 2000. (See: http://www.ripe.net/ripe/mail-archives/lir-wg/20000101-20000401/msg00007.ht…) The full final proposal (15 Jan 2001) by Aldridge can be found at: http://www.ripe.net/ripe/mail-archives/lir-wg/20010101-20010401/msg00003.ht… At the RIPE 38 meeting,(22-26 Jan 2001), the RIPE NCC accepted the action to implement this proposal. Motivation (as expressed by James Aldridge) -------------------------------------------- "For various reasons large registries often need to distribute their allocated address space between various parts of their organisation (for example we have separate national operations in 20 or so countries obtaining address space through the eu.eunet registry). In these situations it makes sense to document this redistribution in the RIPE database but there is currently no way to do this without throwing up errors in the RIPE NCC's auditing procedures or by removing flexibility and adding more work to the already busy NCC hostmasters by having each local operation treated as having their own allocation. Previously the RIPE database software allowed anyone to register an object with a status value of "ALLOCATED" but this was abused by people who didn't know what they were doing and so this is no longer possible and all non-NCC registered inetnum objects must now have the status of "ASSIGNED". However, having multiple levels of assignments in the RIPE database causes error reports from the NCC's auditing processes which now see anything under the higher lever sub-allocation as being a duplicate (or overlapping) assignment which makes finding "real" problem assignments difficult or impossible. By allowing a local IR to register an inetnum object with a new status value of "LIR-ALLOCATED" it becomes possible to differentiate between a higher level sub-allocation ("status: LIR-ALLOCATED") and a final assignment by a LIR ("status: ASSIGNED"). By allowing this registration of intermediate objects it would also be possible to restrict changes to lower-level objects differently in different blocks of addresses within the LIR's allocation by setting different "mnt-lower" values without having to open the whole allocated block up for changes by anyone who might have a valid reason for updating a particular (range of) inetnum object(s)." Database Objects affected ------------------------- Only inetnum objects may have the "LIR-PARTITIONED" status value. Usage ----- The "LIR-PARTITIONED" feature allows LIRs to delegate maintenance of lower-level objects representing assignments within the address space specified by the inetnum object with the status "LIR-PARTITIONED PA" OR "LIR-PARTITIONED PI". Functionality ------------- When creating or modifying the inetnum object the database will check the value of the "status:" attribute according to the following rules: - The value of "ALLOCATED PA" or "ALLOCATED PI" is allowed if the object is maintained by the RIPE-NCC-HM-MNT mntner (this name is specified in ALLOCMNT variable of the configuration file) - The value of "LIR-PARTITIONED PA" is allowed if one level less specific inetnum object contains a "status:" attribute with the value of "ALLOCATED PA" or "LIR-PARTITIONED PA". - The value of "LIR-PARTITIONED PI" is allowed if one level less specific inetnum object contains a "status:" attribute with the value of "ALLOCATED PI" or "LIR-PARTITIONED PI". Additional clarification ------------------------ This proposal addresses the added inetnum status value as proposed by James Aldridge. This is strictly a database feature, allowing LIRs to delegate maintenance within their allocations in the RIPE Database. The implementation of this proposal will not affect IP address allocation policy. This means that it will not affect the responsibility LIRs have for allocations and assignments held by them. Furthermore, it will not change the manner in which the RIPE NCC verifies allocation utilisation. There is an on-going policy discussion addressing this specific issue. Kind regards, Nurani Nimpuno *-----------------------------------* | Nurani Nimpuno <nurani(a)ripe.net> | | Internet Address Policy Manager | | RIPE Network Co-ordination Centre | | http://www.ripe.net | *-----------------------------------*

1 0

changed attribute proposal and not only
by Andrei Robachevsky 11 Oct '01

11 Oct '01

Dear Colleagues, Please find enclosed the proposal of modification of the changed attribute that was presented at the RIPE-40 meeting in Prague. However, another proposal (or rather idea) came up in light of recent discussions regarding timestamping and cleanup procedures in the RIPE Database. Let me present you this idea first. As was pointed by the two proposals there is a need to store and display status information about an object in the database, such as last update time, number of references, etc. At the same time in order to be reliable this information should be automatically generated by the database software and be outside users' control. Regarding "changed attribute" proposal people felt that this changes some of the basic principles of the RIPE Database. Recent Ulf's proposal makes things even worse since it requires that modification to an object may be caused by an update of another object. But in fact these attributes/facilities do not represent an object, but rather status information about the object. Extending Ulf's analogy with the Unix FS, they are part of the inode, not the data blocks. So status information (object metadata) may be stored and displayed separately from the object itself by, say, using a flag. Much like we display the contents using cat, but status using ls. And this information will be used for cleanup procedure, reliable timestamping, etc. This will not affect any object template, preserve users' data and add flexibility to the database. Something like this: $ whois -q meta RC1234-RIPE ... person: Referenced Contact nic-hdl: RC1234-RIPE ... changed: refcon(a)somedom.dom 20011005 source: RIPE %metainfo: RC1234-RIPE %update-time: 2001-10-05 16:50:07 %ref-time: 2001-10-06 09:17:33 %ref-count: 3 Regards, Andrei Robachevsky DB Group Manager RIPE NCC

1 0

Reminder: RIPE-181 to RPSL Migration; Phase 3.
by Andrei Robachevsky 09 Oct '01

09 Oct '01

Dear Colleagues, [Apologies for multiple messages] On and from Monday, 15th October, 2001, the RIPE Database will no longer accept updates in RIPE-181 format. Mail robot at <auto-181(a)ripe.net> will be shutdown. The update path for the TEST database will also be changed. We plan to discontinue <test-181(a)ripe.net> service on 11th October, 2001. If you are still sending updates in RIPE-181 format, we advise you to switch to RPSL before 15th October. Please send your objects in RPSL format to <auto-dbm(a)ripe.net>. You may use ripe2rpsl utility to convert your RIPE-181 objects to RPSL format before sending them to the RIPE Database. The utility can be found at ftp://ftp.ripe.net/tools/ripe2rpsl-3.0.2.tar.gz. As part of this final step we will also make some cleanup of the ftp.ripe.net site. - The top directory for the RIPE Database software and flat object files is ftp://ftp.ripe.net/ripe/dbase. - "Short" versions of flat object files [*] will not be supported. [*]Version of a flat object file where attribute names are abbreviated to 2 letter code starting with '*'. The name of the object file also contains the object type in abbreviated form. For example ripe.db.in is the "short" version of the flat file for inetnum objects, while ripe.db.inetnum is the "long " version that is RPSL compliant. For more information please visit http://www.ripe.net/rpsl If you have any questions, please contact <ripe-dbm(a)ripe.net>. Regards, Andrei Robachevsky DB Group Manager RIPE NCC

2 1

automatic DB cleanup proposal
by kieber＠gatel.net 09 Oct '01

09 Oct '01

Re everybody, following up to some personal talks during the RIPE meeting I propose an automatic cleanup of orphaned person and role objects in the RIPE DB. Motivation ---------- In the light of data privacy issues raised by governmental organizations and in RIPE Database Working Group sessions during RIPE-39 and RIPE-40, as well as following the Great Dangling Person Objects Deletion after de.* ccTLD move-out I propose an automized garbage collection procedure to remove personal data that is unreferenced for a certain amount of time. Proposed Change --------------- An additional attibute ``expire:'' with two values should be added to all person and role objects. The values to this attribute would be a reference counter and a modification timestamp. The reference counter would contain the number of objects still refering to this object, and thus be analog to the link counter of an inode in a UNIX file system. The modification timestamp would be updated whenever the object is updated or the link counter changes. The analogy in the UNIX file system would be sort of a mix of ctime and mtime. Both values could be combined into a single value, but each value of it's own may be of interest to some people. Combining both values would leave the resulting value useless to human interpretation. A garbage collection process running on a periodic basis should then delete all objects with a reference counter of ``0'' and a modification timestamp of more than e.g. 8000000 seconds (approx. 3 month) ago. The expiration period is to be discussed and agreed upon in the community. An additional switch to the whois client should be added to make the otherwise invisible attribute visible, the DB should consequently not accept ``expire:'' attributes in updates. (The DB could certainly be made more forgiving about this, but since the move to v.3 tightened the syntax we should probably not weaken it again.) Database Objects affected ------------------------- An additional attribute would be added to all person and role objects. role: [mandatory] [single] [lookup key] address: [mandatory] [multiple] [ ] phone: [optional] [multiple] [ ] fax-no: [optional] [multiple] [ ] e-mail: [mandatory] [multiple] [lookup key] trouble: [optional] [multiple] [ ] admin-c: [mandatory] [multiple] [inverse key] tech-c: [mandatory] [multiple] [inverse key] nic-hdl: [mandatory] [single] [primary/look-up key] remarks: [optional] [multiple] [ ] notify: [optional] [multiple] [inverse key] mnt-by: [optional] [multiple] [inverse key] expire: [generated] [single] [ ] changed: [mandatory] [multiple] [ ] source: [mandatory] [single] [ ] person: [mandatory] [single] [lookup key] address: [mandatory] [multiple] [ ] phone: [mandatory] [multiple] [ ] fax-no: [optional] [multiple] [ ] e-mail: [optional] [multiple] [lookup key] nic-hdl: [mandatory] [single] [primary/look-up key] remarks: [optional] [multiple] [ ] notify: [optional] [multiple] [inverse key] mnt-by: [optional] [multiple] [inverse key] expire: [generated] [single] [ ] changed: [mandatory] [multiple] [ ] source: [mandatory] [single] [ ] Example ------- inetnum: 10.0.0.0 - 10.0.255.255 netname: XMAS descr: Santa's Workshop Inc. Christmas Toys Manufacturing Facility Northern Nowhere country: NN admin-c: SC12-RIPE tech-c: SWT95-RIPE status: ASSIGNED PA mnt-by: SANTA-SECURITY-MNT changed: jbgood(a)antarctic.nn 19960412 source: RIPE role: Santa Workshop Team address: Santa's Workshop Inc. Jingle Bell Lane 12 1224CH Christmastown Northern Nowhere phone: +12 12 122 3333 fax-no: +12 12 221 4444 e-mail: team(a)antarctic.nn admin-c: SC12-RIPE tech-c: SC12-RIPE tech-c: RD212-RIPE nic-hdl: SWT95-RIPE mnt-by: SANTA-SECURITY-MNT expire: 1 849700000 changed: jbgood(a)antarctic.nn 19960412 source: RIPE person: Santa A. Claus address: Santa's Workshop Inc. Jingle Bell Lane 12 1224CH Christmastown Northern Nowhere phone: +12 12 122 2121 +12 12 122 2211 ext. 1221 fax-no: +12 12 221 1212 e-mail: santa(a)antarctic.nn nic-hdl: SC12-RIPE mnt-by: SANTA-SECURITY-MNT expire: 3 849700000 changed: jbgood(a)antarctic.nn 19960412 source: RIPE person: Rudolf R N Reindeer address: Santa's Workshop Inc. Jingle Bell Lane 21 1224CH Christmastown Northern Nowhere phone: +12 12 122 1111 fax-no: +12 12 122 2222 nic-hdl: RD212-RIPE e-mail: rudolf(a)xmas.nn mnt-by: SANTA-SECURITY-MNT expire: 1 849700000 changed: jbgood(a)antarctic.nn 19960412 source: RIPE -- Ulf Kieber email: kieber(a)gatel.net Senior Network Engineer voice: +49-69-299896-21 Global Access Telecommunications, Inc. fax : +49-69-299896-40 internet solutions for business www : www.gatel.net The Database Toothbrush Proposal

4 4

Re: automatic DB cleanup proposal
by Wilfried Woeber, UniVie/ACOnet 09 Oct '01

09 Oct '01

Dear George, >I am worried that there may be *external* referencees to objects, and while >we have no 'contractual' obligation it might not be nice to do this >to people without some engagement but I expect most people would welcome >losing a path to spam. without going into any detail of the proposal itself, I perceive the fact that there are external references (allowed) _without "consent" and without documentation_ a major flaw in the architecture. In fact, there are mechanisms available already (at least in the RIPE DB-SW V2.x and V3.x) that allow us to deal with that requirement: maintainer objects. >As to automatic sweeping, I think there are problems with link counting >which are similar to those we've faced in discussing cross-database linkage >and references. Too many risks of the link count getting out of whack. Fair enough, a very valid concern! But I think the proposal is trying to describe the "user view" and the "abstract behaviour". >I think you could use it as a guide to a sweep, but not as a determinant >without some review and other checks. Eg to trigger more exhaustive checks >to see if there are are references in twisty ways. I agree. >cheers > -George Cheers, and thanks for partcipating in this discussion! Wilfried.

2 1

Re: automatic DB cleanup proposal
by kieber＠gatel.net 09 Oct '01

09 Oct '01

Somebody @ RIPE NCC, speaking for himself, not for RIPE NCC, writes: > On 2001-10-08 20:02:38 +0200, Ulf Kieber wrote: > > > > The modification timestamp would be updated whenever the object is > > updated or the link counter changes. The analogy in the UNIX file > > system would be sort of a mix of ctime and mtime. > > <snip/> > > > expire: 1 849700000 > > I suggest that this look more like the timestamp in the changed > attribute: > > expire: 1 20011008 > > Human-readable == good. :) Seconded. Daily granularity is good enough, but way more readable. Make is so. > Perhaps this attribute should also be present on maintainer and irt > objects, since they contain contact information as well? Not > necessarily as important, but possible. Then we could put it onto key-cert as well. I'm sort of reluctent to have it placed onto maintainer objects which can not be recreated without interaction with the NCC. I'd at least ask for an extended expiration period. -- Ulf Kieber email: kieber(a)gatel.net Senior Network Engineer voice: +49-69-299896-21 Global Access Telecommunications, Inc. fax : +49-69-299896-40 internet solutions for business www : www.gatel.net

1 0

Re-homing of RAToolSet to RIPE NCC
by Andrei Robachevsky 05 Oct '01

05 Oct '01

Dear Colleagues, [sorry for duplicate messages] Following Cengiz Alaettinoglu's message on this subject on the ratoolset(a)isi.edu mailing list (please find the original message attached below) it is our pleasure to announce that RAToolSet project has moved from ISI to the RIPE NCC. Starting from now the RIPE NCC will take on support and development of these tools. As a part of this move, RAToolSet will be called IRRToolSet. A new web page http://www.ripe.net/ripencc/pub-services/db/irrtoolset/index.html and a mailing list irrtoolset(a)ripe.net have been established. We have subscribed all of the members of ratoolset(a)isi.edu to this mailing list. The original list will be phased out, but will forward any email to the new address. If you would like to subscribe to the IRRToolSet mailing list, put the line "subscribe irrtoolset" in the body of the message to add your address to the list, and send it to majordomo(a)ripe.net. We rely on your feedback and hope to provide the level of support you received from Cengiz and the ISI team. Best regards, Andrei Robachevsky DB Group Manager RIPE NCC

7 11