- db-wg - mailman.ripe.net

Removing personal data from bulk output from the RIPE Database
by Denis Walker 14 May '13

14 May '13

Dear colleagues, We are proposing an improvement to the algorithm used for removing personal data from the bulk provisioning of RIPE Database data. Below you will find an explanation of the current process, why we’re proposing to change it and how the new process would work. ------------------------ Current process ------------------------ At the moment this data is published in three forms: • Complete dump, accessible from: <ftp://ftp.ripe.net/ripe/dbase/ripe.db.gz> • Split dump of each object type, accessible from: <ftp://ftp.ripe.net/ripe/dbase/split/> • A live feed using NRTM protocol, described at: < https://www.ripe.net/data-tools/db/nrtm-mirroring> These data dumps and streams are mainly used for keeping an up-to-date local copy of the RIPE Database or for research and analytical purposes. To prevent the bulk publishing of personal data, for example names, email addresses and phone numbers, we remove all personal data from these datasets. We call this process "dummification of personal data". At the moment it consists of replacing all PERSON and ROLE objects with a single dummy placeholder object as well as changing all references to the personal data objects from all other objects to refer to this dummy placeholder object. ------------------------------------------------- Rationale for Improving the Process -------------------------------------------------- We’ve received feedback from different users and researchers that we are overdoing the dummification. For example, one can obtain all references to personal objects without hitting any personal object result limits by querying the live RIPE Database with proper flags (like -r). This makes the "dummification" of these references in the data dumps meaningless. For more information, see: <http://www.ripe.net/data-tools/support/documentation/aup> ----------------------------------------------------- Proposal for New Dummification Algorithm ----------------------------------------------------- In order to improve the usability of the data dumps and streams, we are proposing to change the "dummification" algorithm to keep the actual personal objects and all references to them and only obfuscate the fields with personal data (for example real names, phone numbers and addresses). The new algorithm will also try to preserve data that is useful for researchers, while not revealing any data that might expose the identity of the date subject. For example, we are proposing to keep the first half of phone number digits or to keep the domain part of email addresses. The proposed changes should not affect any End User scripts currently running because the data will still be in the valid RIPE RPSL format. We propose to run the two datasets in parallel for some time to ensure a smooth transition. Further details and examples of how the dummified objects will change are detailed in a RIPE Labs article at: <https://labs.ripe.net/Members/kranjbar/proposed-improvements-to-dummificati…> Please let us know if you agree with the proposed improvements. Regards, Denis Walker Business Analyst RIPE NCC Database Group

4 4

Re: [db-wg] Removing personal data from bulk output from the RIPE Database
by Tore Anderson 10 May '13

10 May '13

* denis(a)ripe.net > Everyone is correct in what they have said. But when it comes to the > "changed:" attribute you have to rely on assumptions, estimates and 'good > enough' because the dates are not precisely defined. It is possible 90% of > changed dates are spot on, but you don't know that. > > No one would suggest that anyone would deliberately enter the wrong date. > And of course if you don't add a date the software will generate the > correct date for you. But if you wrote a template some time ago for > creating assignments and that template has a date in it, then everytime > you fill in the template to create a new assignment, it will have the > wrong date. Or if you 'change' the date on a single "changed:" attribute > every time you update the object, it will only show the 'last updated' > date. Or as Kaveh pointed out, maybe the object was created 10 years ago, > but you updated the information last week, without altering the changed > date. Do you consider the object to be stale data? There are so many ways > this date can represent something different to what you assume it to be. So while I have no illusions about that I'd get an exact number from looking at the "changed:" attribute, what I was really trying to say is that tried to use the database dumps on ftp.ripe.net for some purpose, but due to excessive dummification I couldn't. Hence I support the proposal to reduce the amount of dummification. Whether or not my desired query would stand up to any "scientific" scrutiny is really besides the point. That said, going slightly off topic wrt. the proposal itself: I'm still no closer to finding an answer to my question, which could be stated as: «How many IPv4 PA Assignments have been made by RIPE NCC LIRs since the 14th of September 2012? (If possible, but less important: How many addresses did these assignments consist of, in total?)» Even a very rough estimation is useful to me. Even just the order of magnitude would be interesting. "A dozen", "about five thousands", or "hundreds of thousands" - these are all results I could put in my slides to underline the point I want to make. While I think the use of the "changed:" attribute I originally planned would give me a useful enough result, I'm in no way married to that particular approach. So if there's some other way to get the number, possibly by having any of you RIPE NCC guys fetch it for me from some internal data source, that would be much appreciated! :-) I'd need it before Wednesday (well, preferably ASAP) if I'm going to be able to make use of it though. > As Kaveh also pointed out, you can query an object with the > '--list-versions' option and get a more accurate, reliable list of dates > for when an object was created and each update. But if you want to find > which PA assignments were created within a time period, you would have to > query them all to get the accurate historical dates. There are 3.7 million > assignment objects. For this type of analysis it might be easier if the > split file had a well defined date(s) you could rely on. No, I really wouldn't want to script 3.7M queries to your whois server...having an auto-generated "created:" attribute in the dumps (and in the whois service too for that matter) would certainly be better. Tore

2 2

Re: [db-wg] Removing personal data from bulk output from the RIPE Database
by James A. T. Rice 09 May '13

09 May '13

On Thu, 9 May 2013, denis(a)ripe.net wrote: > The "changed:" attribute is almost useless. Disagree, it gave a good indication of how stale an object was likely to be. If an object is is 15 years old, one wouldn't be too surprised if it's inaccurate. If it's from last month, then it's a lot more likely to be accurate. Cheers James

3 2

Re: [db-wg] Removing personal data from bulk output from the RIPE Database
by Tore Anderson 09 May '13

09 May '13

* denis(a)ripe.net > Unfortunately no dump will help you with this process. The "changed:" > attribute is almost useless. For what you want to do it IS useless. The > syntax requires all objects to have at least one mandatory "changed:" > attribute. The user can set the date to almost anything. It can be > tomorrow or ten years ago. When the object is changed, there is no > requirement to make any changes to this date or add any additional > "changed:" attributes. It is user generated and only has meaning to the > user. What you want to do would only work if these were auto generated > timestamps by the update software. But that is not the case. That may be so. But to turn it around - are you of the opinion that the LIRs to a great extent are deliberately supplying incorrect changed dates? If "no", it would be good enough for me, really. For my uses, an approximation will suffice. To be honest, I would actually be more concerned about the number being wrong because of LIRs assigning addresses to their customers without registering the assignments in the database, than I would be about LIRs registering their assignments, but with incorrect dates. BTW - the update software *does* generate time stamps. At least, I always send only my e-mail address, the bot does the rest. Tore

1 0

Re: [db-wg] Removing personal data from bulk output from the RIPE Database
by Tore Anderson 09 May '13

09 May '13

(Apologies for breaking threading, I just subscribed to the list.) In general, the proposal seems fine to me and I agree with its goals. In preparing for my APWG presentation next week, I've been wanting to find the answer to the question «how many IPv4 PA Assignments have been registered between dates A and B?». I thought I could do this by examining the inetnum database export - but that didn't work out, due to the fact that all the "changed:" attributes gets rewritten to "unread‌‌(a)ripe.net 20000101". So while I don't care about the e-mail addresses, I'd need the dates to be left intact. Also, the current algorithm folds all changed attributes into a single one. It's not clear from the Labs article whether or not this practise will stop - I hope it will. Would it by any chance be possible to get a sneak preview of the new inetnum dump, so that I could find the numbers I want before heading to Dublin? If so that would be greatly appreciated! Best regards, Tore Anderson

2 1

Reclaim Functionality for Resource Holders
by Denis Walker 29 Apr '13

29 Apr '13

Dear colleagues, With the new RIPE Database software now in place, the RIPE NCC has been able to automate a process that was previously handled manually. This process now allows resource holders to reclaim any operational database object related to the resources allocated and assigned to them by the RIPE NCC. You can find more details in a new article on RIPE Labs: https://labs.ripe.net/Members/denis/reclaim-functionality-for-resource-hold… Regards Denis Walker Business Analyst RIPE NCC Database Group

2 1

RIPE66 DB-WG Draft Agenda V1
by Wilfried Woeber 17 Apr '13

17 Apr '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear DB-WG folks, Chairpersons, "Meeting"! This is the 1st draft of an agenda for the DB-WG Meeting at RIPE66 in Dublin, May 15, 2013. For time slot allocations please refer to the most up-to-date meeting plan at https://ripe66.ripe.net/programme/meeting-plan/ The DB-WG Meeting is scheduled for *Wednesday*, starting at 09:00 local time / 08:00 UTC in the morning. The length of our timeslot is 90 Minutes. The overall (draft) RIPE 66 meeting outline is already available at https://ripe66.ripe.net/programme/meeting-plan/ Please feel free to propose additional topics to be discussed! I'd also suggest that you submit your points of view regarding the topics on the proposed agenda on the mailing list, preferably before the WG meeting's start :-) Best regards, hope to see you soon in Dublin, Wilfried. ________________________________________________________________________ A. Administrative Matters . Welcome . select scribe . finalise agenda . approval of minutes from previous WG meeting(s) . review of action list B. Data Base Operational Update (Kaveh Ranjbar, RIPE NCC) C. New DB-Software functionality - in beta - proposed D. Status update on IETF WEIRDS activities (N.N. t.b.c.) Y. Input from other Working Groups and/or Task Forces - review of abuse contact stuff (N.N. t.b.c.)) Z. AOB ________________________________________________________________________ 17.04.2013 16:00 UTC -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (MingW32) iQCVAwUBUW7IBd/RXX7wrLNpAQJymwP8D1Vt3oXoSkrfFuX05mNS2FLNIwucVYha uiAV89Zzk7uTxXa8dAQDpJgrVXIqaqaUgTjqpl1k8kYC85iuFsC8SEvjqQyx4czI oO99voS2x/qDXKI3nJj++4nuvswdDutYxSS5/ETk46sNWR8lsscMJ3Y8li1ttLEt eAJzPP8s9SQ= =YObZ -----END PGP SIGNATURE-----

1 0

Object Tags: a new feature proposal
by Kaveh Ranjbar 17 Apr '13

17 Apr '13

Dear colleagues, We would like to propose a method for providing metadata about objects in the RIPE Database. This new feature will enable a lot of flexibility to inform users about the state of each object and will help improve the quality of data in the RIPE Database. It can also provide a mechanism to fine-tune RIPE Database query outputs, which will give users who have less knowledge about address hierarchy, placeholder objects, etc. easier access to the data they are looking for. You can find more information about this proposal, along with some examples, at: https://labs.ripe.net/Members/kranjbar/ripe-database-object-tags Please kindly provide us with your feedback. Kind regards, Kaveh Ranjbar RIPE NCC Database Group Manager

1 0

Access to historical data in the RIPE database
by Nigel Titley 16 Apr '13

16 Apr '13

Following the announcement and the general positive feedback on this list, the RIPE NCC will be shortly enabling access to historical data in the RIPE database. Best regards Nigel

2 1

Addition of long query options
by Kaveh Ranjbar 05 Apr '13

05 Apr '13

Dear colleagues, With the latest RIPE Database software release, we introduced the choice of longer names for query options. Now each existing query option has an equivalent long name starting with a double dash. These names are more descriptive than the single character options. For example, "-x", which represents exact match searches, has "--exact" as its longer equivalent. The existing behaviour doesn't change; these long names can be used in addition to the short ones, which are all still valid. Using these longer names allows for more descriptive queries. When a query is passed on to a colleague or when it is coded inside a script, the reader will have a better understanding of what the query is searching for without having to consult the documentation. It also allows more flexibility to introduce new query options as more features are added to the RIPE Database. Below you will find the full list of RIPE Database query server options. This list can be generated any time with the query "whois -h whois.ripe.net help" as well. Kind Regards, Kaveh Ranjbar, RIPE NCC Database Group Manager. ... % The following options are available: % % -x, --exact <ip-lookup> % Requests that only an exact match on a prefix be performed. If no % exact match is found no objects are returned. % % -l, --one-less <ip-lookup> % Returns first level less specific inetnum, inet6num or route(6) % objects, excluding exact matches. % % -L, --all-less <ip-lookup> % Returns all level less specific inetnum, inet6num or route(6) % objects, including exact matches. % % -m, --one-more <ip-lookup> % Returns first level more specific inetnum, inet6num or route(6) % objects, excluding exact matches. % % -M, --all-more <ip-lookup> % Returns all level more specific inetnum, inet6num or route(6) % objects, excluding exact matches. % % -C, --no-irt <ip-lookup> % Turns off default option '-c' or '--irt'. % % -c, --irt <ip-lookup> % Requests first level less specific inetnum or inet6num objects with % the "mnt-irt:" attribute (enabled by default). % % -b, --abuse-contact <ip-lookup> % Requests first level less specific inetnum or inet6num objects with % the "abuse-c:" reference. Only object keys and "abuse-mailbox:" % attributes are visible. % % -d, --reverse-domain <ip-lookup> % Enables use of the -m, -M, -l and -L flags for lookups on reverse % delegation domains. % % -i, --inverse <attribute-name> <inverse-key> % Perform an inverse query. % % -F, --brief % Produce output using short hand notation for attribute names. % % -K, --primary-keys % Requests that only the primary keys of an object to be returned. The % exceptions are set objects, where the members attributes will also % be returned. This flag does not apply to person and role objects. % % -k, --persistent-connection (optional normal query) % Requests a persistent connection. After returning the result the % connection will not be closed by the server and a client may issue % multiple queries on the same connection. % Note, that server implements 'stop-and-wait' protocol, when no next % query can be sent before receiving a reply for the previous one. % Except the first -k query, -k without an argument closes the % persistent connection. % % -G, --no-grouping % Disables the grouping of objects by relevance. % % -B, --no-filtering % Disables the filtering of "notify:", "changed:" and "e-mail:" % attributes. % % -r, --no-personal % Switches off recursion for contact information after retrieving the % objects that match the lookup key. % % -T, --filter-types (comma separated list of object types, no white space is allowed) % Restricts the types of objects to lookup in the query. % % -a, --all-sources % Specifies that the server should perform lookups in all available % sources. See also "-q sources" or "--all-sources" query. % % -s, --sources (comma separated list of sources, no white space is allowed) % Specifies which sources and in which order are to be looked up when % performing a query. % % -q (sources|version|types) % "sources" see list-sources. % "version" see version. % % --list-sources % Returns the current set of sources along with the information % required for mirroring. See [REF], section 2.9 "Other server % features" for more information. % % --version % Displays the current version of the server. % % -t, --template <object-type> % Requests a template for the specified object type. % % -v, --verbose <object-type> % Requests a verbose template for the specified object type. % % -V, --client <client-tag> % Sends information about the client to the server. ...

1 0