dp-tf

Download

dp-tf@ripe.net

September 2007

3 participants
2 discussions

planning for RIPE55
by Wilfried Woeber, UniVie/ACOnet 19 Oct '07

19 Oct '07

Folks, as a follow-up on the discussion regarding the availability and quality of "contact data" in the DB, I had an e-chat with Brian, chair of the Anti-Spam WG. We are tossing around an idea to request a slot during the plenary in order to summarize the "problem space". It might be a good idea to put these presentations ("contacts" and "DP-TF") next to each other. What's your opinion? Wilfried.

4 3

Quadlogy of person proposals
by Denis Walker 04 Sep '07

04 Sep '07

Hi all Attached are the new draft versions of 4 proposals concerning person and other objects. They include the two already discussed in the DB-WG session at RIPE 54: Maintaining person objects Cleanup of person objects plus another issue we briefly discussed at the DP-TF meeting at RIPE 54: Authenticating references to person objects and another loosely related proposal: Structuring the address fields of person objects. We have tried to accommodate all the issues raised in the DB-WG session. As a package they address a number of Data Protection and security issues. If all these proposals are accepted by the community we will implement them one at a time. But we would like to plan the code changes based on the full set that are approved as they are all related. Your initial feedback will be appreciated before we release them to the wider community. regards Denis Walker RIPE NCC Maintaining person, role and domain objects Abstract -------- This is a proposal regarding maintaining all objects. * Maintaining person, role and domain objects o Abstract o Intended Audience o Introduction o Proposal o Illustrations of how the checks work + New startup + Closing down + Modifying existing data o Implementation + Stage 1 + Stage 2 o Statistics o Code changes (for our benefit, not part of public proposal) Introduction ------------ It was agreed at RIPE 54 to make "mnt-by:" mandatory on all objects. Currently, for person, role and domain objects, "mnt-by:" is optional. Making "mnt-by:" mandatory on person objects raises two clear issues: 1. It creates a chicken and egg situation between person and mntner objects. 2. Many person objects are not changed for years, so any change to syntax may take a long time to be reflected in the database. So many un-maintained person objects could remain so for many years. Proposal -------- Making "mnt-by:" mandatory on role and domain objects does not cause any technical difficulties. To accommodate all the issues for the person objects, we suggest a slightly different approach. 1. Make "mnt-by:" mandatory in person objects. 2. Provide a mechanism in dbupdate to handle the startup case where a new person and mntner objects are required. 3. Provide a mechanism in dbupdate to handle the closing down situation where the last person and mntner objects are to be deleted. 4. Extend the referential integrity checks so that a person object cannot be referenced unless it is maintained, except in a mntner object 5. Extend the referential integrity checks so that a mntner object cannot be referenced unless it's referenced person objects are maintained, but excluding the cyclic case of a person and mntner object referencing each other. The referencing checks sound complicated in words, but are actually quite simple. The illustrations below explain how it works in practice. This approach has a number of advantages: 1. It accommodates the chicken and egg situation. 2. We already do referential integrity checks. Whenever there is a reference to a person object, the database software checks if the person object exists. We only have to extend this check to see if the person object is also maintained. 3. Whenever there is a reference to a mntner object, the database software checks if the mntner object exists. We only have to extend this check to see if the person objects referenced by the mntner are maintained. 4. Each time any other object (say an inetnum or aut-num) is modified, the referenced person objects need to be maintained. If not, the update will fail. This will not prevent anyone from working. If they are going to modify an inetnum object, they must be authorised to do so. They can apply the same mntner to any un-maintained person objects. This encourages users to maintain the existing person objects. A cgi script will be provided to make this process simple and quick. 5. No un-maintained person object can be linked to the white pages to avoid deletion. Illustrations of how the checks work ------------------------------------ New startup ----------- Send an update message to dbupdate to create a person object and a mntner object. These must be the first two objects in an update message, in any order. The references to each other must also be in place. The database software will accommodate this. person: Denis Walker address: RIPE Network Coordination Centre (NCC) address: Singel 258 address: 1016 AB Amsterdam address: The Netherlands phone: +31 20 535 4444 nic-hdl: DW-RIPE mnt-by: aardvark-mnt notify: denis(a)ripe.net changed: denis(a)ripe.net 20040318 source: RIPE mntner: AARDVARK-MNT descr: Mntner for denis' objects. admin-c: DW-RIPE tech-c: DW-RIPE upd-to: denis(a)ripe.net auth: X509-1 notify: denis(a)ripe.net mnt-by: AARDVARK-MNT referral-by: RIPE-DBM-MNT changed: denis(a)ripe.net 20040225 source: RIPE In dbupdate it will recognise that the first person/mntner object references a non-existent mntner/person object. It will check that this referenced object is next in the update message. If so the "mnt-by:" attribute will be removed from the person object. The person and mntner objects will be created. Then the person object will be modified to add back the "mnt-by:" attribute. The objects are now fully configured and can be used. The person object can be referenced by any other object where a nic-hdl is referenced. It can also be linked to the white pages. The mntner can be used to protect any data in the database. If the non-existent referenced object is not next in the update message then an error message will be generated and the update will fail. This is the current behaviour. Closing down ------------ The same situation occurs when closing down and deleting all the data. It is possible to delete all data except for the last person and mntner objects in the normal way. Send these last two together in an update message, both marked with the "delete:" pseudo attribute. Only these two objects should be in the update message. In dbupdate it will recognise that the first person/mntner object to delete is referenced in a mntner/person object. It will check that this referencing object is next in the update message and also marked for deletion. If so the person object will be modified first to remove the "mnt-by:" attribute. The mntner object will then be deleted followed by the person object. Modifying existing data ----------------------- Suppose some data is to be modified and there is a reference to an un-maintained person object. Suppose you already have this data in the database: inetnum: 193.0.0.0 - 193.0.7.255 netname: RIPE-NCC descr: RIPE Network Coordination Centre descr: Amsterdam, Netherlands remarks: Used for RIPE NCC infrastructure. country: NL admin-c: DW-RIPE tech-c: DW-RIPE status: ASSIGNED PI mnt-by: AARDVARK-MNT mnt-lower: AARDVARK-MNT changed: bit-bucket(a)ripe.net 20060221 source: RIPE person: Denis Walker address: RIPE Network Coordination Centre (NCC) address: Singel 258 address: 1016 AB Amsterdam address: The Netherlands phone: +31 20 535 4444 nic-hdl: DW-RIPE notify: denis(a)ripe.net changed: denis(a)ripe.net 20040318 source: RIPE mntner: AARDVARK-MNT descr: Mntner for denis' objects. admin-c: DW-RIPE tech-c: DW-RIPE upd-to: denis(a)ripe.net auth: X509-1 notify: denis(a)ripe.net mnt-by: AARDVARK-MNT referral-by: RIPE-DBM-MNT changed: denis(a)ripe.net 20040225 source: RIPE The inetnum and mntner objects both reference a person object that is not maintained. This situation can continue for the foreseeable future with no problem. Until some data needs to be changed. An attempt to modify the inetnum object will fail. This is because it references a person object that is not maintained. It also references a mntner object which references un-maintained person objects. These person objects will have to be maintained before the inetnum object can be modified. A cgi script will be provided to allow this to be done quickly and easily. When all referenced person objects are maintained, the original update can be resubmitted and will proceed. Implementation -------------- As with the crypt-pw deprecation this will have a staged rollout. Stage 1 * No new person, role or domain objects can be created without a "mnt-by:" attribute. * Any unmaintained person, role or domain object cannot be modified without adding an "mnt-by:" attribute. * Any update where objects reference an un-maintained person object, either directly or their mntner has such references, will generate a warning message in the acknowledgement. In this stage the acknowledgement message may include these warnings: ***WARNING: Un-maintained person object referenced [DW-RIPE] Stage 2 * Any update where objects reference an un-maintained person object, either directly or their mntner has such references, will generate an error message in the acknowledgement and the update will fail. In this stage the acknowledgement message may include these errors: ***ERROR: Un-maintained person object referenced [DW-RIPE] Statistics ---------- Not a very statistically valid survey, but we looked at a few days just after the RIPE meeting to see how many new person objects were created with and without a mntner. They were queried some time after creation to allow for a "mnt-by:" to be added later. Also how many unique person objects were referenced in any objects in update messages with and without a mntner. AUTO- references were ignored in both creations and updates. Multiple instances of the same person object being referenced many times were counted as one. date created with created without referenced with referenced without 20070515 156 89 925 726 20070516 111 67 1022 486 20070517 85 48 476 185 20070518 82 18 679 445 Clean up of unreferenced person objects Abstract -------- This is a proposal for a one time cleanup of unreferenced person objects. It also addresses the issues of flagging objects prior to deletion and an opt-in white pages. * Clean up of unreferenced person objects o Abstract o Intended Audience o Introduction o Background o Changes to objects o Flagged o White pages o Bulk cleanup procedure + Month 1 + Month 2 + Month 3 + Notes o References o Follow up + WG mailing list references + RIPE Meeting minutes references Introduction ------------ According to our database consistency statistics program (dbconstat [1]) we currently have 460,573 unreferenced person/role objects [2]. Some of these may be maintained, but are still unreferenced. Any personal data not referenced by Internet resources do not fit within the primary purpose of the RIPE Database. They should not be stored in the RIPE Database beyond a reasonable 'work in progress' period. A secondary purpose of the RIPE Database has been proposed. This will be a 'white pages' listing of 'significant' people within the Internet community. Those wishing to be linked to these white pages must 'opt-in' and explicitly consent to their personal data being displayed in a public database. We regard this consent to be given by providing the authentication to modify the person object. Those doing so will be exempt from the cleanup process. One of the other current proposals is to maintain all objects. This may result in many of the currently unreferenced person/role objects being referenced in a maintainer which maintains the person/role object. This pair of mutually referencing objects may not be linked to any other object. This may have a significant effect on the original proposal. We therefore propose a change to this one time cleanup to accommodate this effect and aim to hit the majority of the previously identified objects which should be removed. The rest will be taken care of later with the introduction of a regular cleanup script. The new proposal will still target any unreferenced person/role objects. But also target person objects and their referencing mntner object, providing this pair of objects only reference each other. Even though person/role objects must now be maintained, this does not mean they will all be referenced by a mntner. They may be maintained by an existing mntner which references other person/role objects. This allows for the maintained person/role objects to still be unreferenced. Targeting 'loose' mntner objects will catch the mutually referencing pairs. There may be many more of these when it is required to maintain person objects. In this case we will only target the person/mntner object pairs. To include role objects implies person/role/mntner groups with many more references. This is too complicated to handle within the scope of this one time cleanup process. Background ---------- The RIPE NCC has had a mandate to delete these since RIPE40: http://www.ripe.net/ripe/wg/db/minutes/ripe-40.html (2001) At RIPE 41, the Database Working Group agreed that "maintained objects will now be removed" and "gave a mandate to the RIPE NCC to continue with the cleanup process." http://www.ripe.net/ripe/wg/db/minutes/ripe-41.html (2002) The cleanup process of 2003 (http://www.ripe.net/db/news/unref-cleanup-200304.html) involved using a script to run periodic cleanups. This script was put in place. However, it failed about 18 months ago. Because of other priorities, we have not had the time to examine this issue again until now. The graph showing the increase in these unreferenced person objects [3] indicates that the cleanup script appeared not to be performing correctly. At the start of 2006, there were about 300,000 unreferenced person/role objects. Since then there has been a steady increase with a large increase of almost 50,000 in February 2007. The graph also shows a slightly higher rate of increases in these objects since February 2007. At RIPE 54 (2007) it was agreed that a new cleanup process is needed. Concerns regarding use as 'white pages' and the issue of tagging objects prior to deletion to delay re-use of nic-hdls have been addressed in this new proposal. Because redundant personal data, with no explicit consent to remain, is a serious data protection issue, we want to take a new approach to this in the future. Once the initial cleanup is in progress, we will create a new proposal for a new, regular cleanup procedure. This will be sent to the Data Protection Task Force [4] and then to the rest of the community. Changes to objects ------------------ Add a "not-ref:" attribute to person/role objects. This indicates that the person/role object is not referenced and the date when it last became unreferenced. It is generated by the database software and takes a date stamp. It can also take an extra tag 'FLAGGED'. It cannot be changed in any way by users. person: [mandatory] [single] address: [mandatory] [multiple] phone: [mandatory] [multiple] fax-no: [optional] [multiple] e-mail: [optional] [multiple] org: [optional] [multiple] nic-hdl: [mandatory] [single] not-ref: [generated] [single] remarks: [optional] [multiple] notify: [optional] [multiple] abuse-mailbox: [optional] [multiple] mnt-by: [mandatory] [multiple] changed: [mandatory] [multiple] source: [mandatory] [single] Create a new org-type for organisation objects: org-type: WHITE-PAGES This type can only be set by the database administrators. Flagged ------- When an object is deleted the nic-hdl becomes immediately re-usable by anyone. To allow sufficient time for the user to decide what they want to do with their (unreferenced) person/role objects, they will be flagged before deletion. During this flagged period the user can make appropriate references, link it to the white pages (maintained person objects only) or delete it themselves. If nothing is done it will be automatically deleted after a set flagged period is over. The cleanup script will append the 'FLAGGED' tag to the "not-ref:" generated attribute in the selected, (unreferenced) person/role objects: not-ref: <date> FLAGGED If the object has a "notify:" attribute or is maintained, the user will be notified of this modification by the normal update mechanism. A customised notification message will be used to make it clear to the user why this notification has been sent and reference a web page with further details. If the user wishes to keep this object they can reference it either directly or indirectly from an internet resource or apply to link it to the white pages (maintained person objects only). Either of these operations, if successful, will result in the "not-ref:" attribute being removed from the object by the database software. After an agreed period, if this "not-ref:" attribute is still present with the 'FLAGGED' tag the object will be deleted. The user will be notified by the normal update mechanism with the standard message, if the appropriate contact attributes are present. White pages ----------- There are some people who have a 'significant presence' within the internet community, but who are not directly responsible for any internet resource. These people may want/need to be easily contactable by the community. To facilitate this we will introduce a white pages to the RIPE Database. Users can have their person objects linked to the white pages. The RIPE NCC will create a number of organisation objects to define the categories of the white pages. For example: RIPE NCC board RIPE WG chairs RIPE working groups etc These organisation objects will have a new "org-type:" of 'WHITE-PAGES'. For each category there will be a moderator who will approve an application. The nic-hdl of the moderator(s) will be stored in a "remarks:" attribute in the organisation objects. Details of the white pages and how to apply will be available on the RIPE website. A user can apply to have their person object linked to the white pages. They should select the category and contact the moderator. The user needs to send their full person object to the moderator. This should either include the plain text password or be a signed message providing the authentication to modify this person object. Unmaintained person objects will not be linked to the white-pages. If the moderator approves the application s/he will add an "org:" attribute referencing the appropriate organisation object and the required authentication and submit the message as an update to the RIPE Database software. If the user no longer wishes to be linked to the white pages they can remove the "org:" attribute from the person object. However, if the object is not referenced it will be tagged and then deleted. Person objects that are referenced by internet resources can still be linked to the white pages if the user wishes to be known in a specific category. Unreferenced person objects and person/mntner pairs will be exempt from the cleanup process if linked to the white pages. You can only apply for an existing, maintained person object with an "e-mail:" attribute to be linked to the white pages. The person objects must be created and maintained in the normal way before applying. If the object is later modified to remove the "e-mail:" attribute it will lose it's white pages status. The nominated moderator for the selected categories can be the RIPE NCC for such categories as RIPE NCC board and RIPE WG chairs. The WG chairs (or co-chairs) for their WG category, etc. Requests for additional white pages categories can be sent to Customer Services at RIPE NCC. These requests will be forwarded to the WG chairs mailing list for approval. If approved the RIPE NCC will create the new organisation object, update the web page and notify the moderator. At RIPE 54 there was no discussion on who should be allowed to use the white pages. Without any limits or restrictions applied, it is open to abuse and could become an address book with thousands or tens of thousands of entries. We would simply move the problem of unreferenced person objects from one place to another. Although the RIPE NCC is Data Controller for the RIPE Database, it is not in a position to decide on the usage of the white pages. This is why we have introduced the moderator position to put this responsibility on members of the community who are in a position to make these decisions. We do not expect a high demand for this service or for creating many new white pages categories. If the demand does become excessive then it becomes a social networking service for technical people. In that case we should reassess the purpose and implementation. Our first option is to keep it simple and constrained. Bulk cleanup procedure ---------------------- Month 1 * Select 80,000 unreferenced person/role objects or person/mntner pairs and tag them. (These will be the first 80,000 found by an sql query on the database.) Month 2 * Check selected objects. * Those still unreferenced and tagged: o Delete using normal update process. o Run updates overnight to avoid any unnecessary load on the servers. * Select next 80,000 unreferenced person/role objects or person/mntner pairs and tag them. Month 3 * Repeat process until complete. Notes ----- This procedure will take about 6 months to clear the current backlog, not including the extra time that may be necessary due to the increases over that period. We will announce the start of the cleanup to the Working Group mailing lists and as a news item on our web site home page. References ---------- [1] dbconstat http://www.ripe.net/projects/dbconstat/index.html [2] current unreferenced person/role objects http://www.ripe.net/projects/dbconstat/html/cons-current.html [3] graph of unreferenced person object increase http://www.ripe.net/projects/dbconstat/cons-unrefpero.html [4] Data Protection Task Force http://www.ripe.net/ripe/tf/dp/index.html Authentication for referencing of person and role objects Abstract -------- This is a proposal regarding authentication of references to person and role objects. * Authentication for referencing of person and role objects o Abstract o Intended Audience o Introduction o Background o Details o Implementation o How to implement this in a large company or organisation. o For internal reference (not part of proposal) + Possible extension to the proposal Introduction ------------ There is currently no requirement to authenticate a reference to any person or role object in the database. This can lead to hijacking and the accidental or deliberate misrepresentation of responsibility for data. The database software already includes a mechanism for this type of authentication. References to organisation and irt objects must be authenticated. We propose to extend this list to include person and role objects. Background ---------- There are two ways a reference to the wrong person object can be made - accidental and deliberate. Accidental references can be made by mistyping. Maybe xy58-ripe instead of xy85-ripe. Even though it is accidental it can still cause confusion to anyone wanting to contact the responsible people for some data. Deliberate references to the wrong person may be made to avoid responsibility for use of address space. If someone is doing something nasty on the internet they have three options to try to avoid responsibility for a while. 1. Reference a person object with obviously stupid data in it. But anyone seeing a person object for Donald Duck will waste no more time looking further. 2. Reference a person object with false, but sensible looking data in it. But if a law enforcement agency is looking to make contact with the responsible party they will quickly find out if London Road in Southend has a number 147. If the address does not exist or the phone number does not match the address they will again quickly dismiss this false data. 3. Reference an existing, valid person object belonging to someone else. If a spammer references my person object, the police will come knocking on my door. I will say it has nothing to do with me, which of course I would say if I was the spammer. The police will then waste time investigating me while I try to prove my innocence. This wastes time as well as causing me problems. The RIPE NCC occasionally receives requests from LEAs for information about the responsible persons for address space. So this data is used in investigations. This proposal prevents any deliberate attempt to divert attention as well as avoiding mistakes. Details ------- The proposal is to add an "mnt-ref:" attribute to person and role objects. person: [mandatory] [single] address: [mandatory] [multiple] phone: [mandatory] [multiple] fax-no: [optional] [multiple] e-mail: [optional] [multiple] org: [optional] [multiple] nic-hdl: [mandatory] [single] mnt-ref: [optional] [multiple] remarks: [optional] [multiple] notify: [optional] [multiple] abuse-mailbox: [optional] [multiple] mnt-by: [mandatory] [multiple] changed: [mandatory] [multiple] source: [mandatory] [single] This attribute will specify mntner objects that can authorise a reference to this person/role object. This will only be required when a reference is added to an object. The "mnt-ref:" attribute is optional. The functionality will default to the "mnt-by:" when this new attribute is not present, or if authentication is not passed by any mntners in the "mnt-ref:" attributes. To add a reference to a person object as "admin-c:" of say an inetnum object, authentication will be required from a mntner listed in the "mnt-by:" of the inetnum object, as is the case now. In addition to this, authentication will also be required from a mntner listed in the "mnt-ref:" of the person object, if this attribute is present. If this attribute is not present, or non of the authentications passed, the mntner in the "mnt-by:" of the person object will need to authorise the reference. Implementation -------------- * All new person or role objects can be created with an optional "mnt-ref:" attribute. * Any person or role object can be modified to add an "mnt-ref:" attribute. After an agreed flag date, adding references to person or role objects will require the authentication of the person/role object. How to implement this in a large company or organisation. --------------------------------------------------------- Many organisations have several staff who make changes to their data in the RIPE Database. Their data may also reference many person objects of their customers. If these have been generated by the company on behalf of their customers and all have the same mntner as "mnt-by:", it is a simple matter to add the authentication for this mntner to all updates where a new reference is created. If the person objects have different mntners or are maintained by the customers themselves it requires a little bit more work to set up. One way to accommodate this is to create one new mntner object. Add a pgp key for each staff member to this mntner. Put this mntner in all the referenced person objects as the "mnt-ref:". Or ask all the customers to add this mntner for you. All these staff can now authorise references to all the person objects. If a staff member leaves simply remove their pgp key from the mntner. If a person does not want to be referenced they can either remove or not add this mntner to their person object. Structuring of address attributes in person, role and organisation objects Abstract -------- This is a proposal to properly structure address attributes in person, role and organisation objects. * Structuring of address attributes in person, role and organisation objects o Abstract o Intended Audience o Introduction o Changes to objects o Implementation + Stage 1 + Stage 2 Introduction ------------ Currently there is only a single "address:" attribute. This holds all the address details in free text format. It is almost impossible to break down this single free text into the component parts of an address. We propose to structure the address into separate attributes matching the standard components. It also brings it into line with our internal Registration Database. This will * improve the overall 'usefulness' of the data * allow historical data in the database to be 'cleaned' by removing the personal data but keep the city/country information. This may be useful for future statistical purposes * allow information to be made available to those who insist on doing IP to city mapping without disclosing any personal data. This is a convenient time to do this along with the other changes currently proposed with person and role objects. Changes to objects ------------------ The "address:" attribute will be replaced with: * "street:" * "city:" * "zip:" * "country:" The "country:" attribute will only accept an international country code to avoid differences in spelling. Taking the person object as an example (the same change will apply to role and organisation). person: [mandatory] [single] street: [mandatory] [multiple] city: [mandatory] [single] zip: [optional] [single] country: [mandatory] [single] phone: [mandatory] [multiple] fax-no: [optional] [multiple] e-mail: [optional] [multiple] org: [optional] [multiple] nic-hdl: [mandatory] [single] remarks: [optional] [multiple] notify: [optional] [multiple] abuse-mailbox: [optional] [multiple] mnt-by: [optional] [multiple] changed: [mandatory] [multiple] source: [mandatory] [single] These new attributes will not be inverse searchable. So it will not be possible to find all persons in a city, for example. Implementation -------------- Again this will be rolled out in stages. Stage 1 * Only allow new objects with properly constructed attributes. * Whenever a person/role/organisation object is modified with "address:" attributes a warning message will be added to the acknowledgement. * Whenever a person/role/organisation object is referenced with "address:" attributes a warning message will be added to the acknowledgement. ***WARNING: The person object [DW-RIPE] still contains "address:" attributes. Stage 2 * Whenever a person/role/organisation object is modified with only "address:" attributes an error message will be added to the acknowledgement. * Whenever a person/role/organisation object is referenced with only "address:" attributes an error message will be added to the acknowledgement and the update will fail. ***ERROR: The person object [DW-RIPE] still contains "address:" attributes.

7 53