- db-wg - mailman.ripe.net

RIPE 66 Draft minutes
by Nigel Titley 15 May '13

15 May '13

Dear WG For your delight and delectation, please find the draft minutes for the DB-WG meeting at RIPE 66 are attached Comments and corrections to me please Nigel

3 2

Removing personal data from bulk output from the RIPE Database
by Denis Walker 14 May '13

14 May '13

Dear colleagues, We are proposing an improvement to the algorithm used for removing personal data from the bulk provisioning of RIPE Database data. Below you will find an explanation of the current process, why we’re proposing to change it and how the new process would work. ------------------------ Current process ------------------------ At the moment this data is published in three forms: • Complete dump, accessible from: <ftp://ftp.ripe.net/ripe/dbase/ripe.db.gz> • Split dump of each object type, accessible from: <ftp://ftp.ripe.net/ripe/dbase/split/> • A live feed using NRTM protocol, described at: < https://www.ripe.net/data-tools/db/nrtm-mirroring> These data dumps and streams are mainly used for keeping an up-to-date local copy of the RIPE Database or for research and analytical purposes. To prevent the bulk publishing of personal data, for example names, email addresses and phone numbers, we remove all personal data from these datasets. We call this process "dummification of personal data". At the moment it consists of replacing all PERSON and ROLE objects with a single dummy placeholder object as well as changing all references to the personal data objects from all other objects to refer to this dummy placeholder object. ------------------------------------------------- Rationale for Improving the Process -------------------------------------------------- We’ve received feedback from different users and researchers that we are overdoing the dummification. For example, one can obtain all references to personal objects without hitting any personal object result limits by querying the live RIPE Database with proper flags (like -r). This makes the "dummification" of these references in the data dumps meaningless. For more information, see: <http://www.ripe.net/data-tools/support/documentation/aup> ----------------------------------------------------- Proposal for New Dummification Algorithm ----------------------------------------------------- In order to improve the usability of the data dumps and streams, we are proposing to change the "dummification" algorithm to keep the actual personal objects and all references to them and only obfuscate the fields with personal data (for example real names, phone numbers and addresses). The new algorithm will also try to preserve data that is useful for researchers, while not revealing any data that might expose the identity of the date subject. For example, we are proposing to keep the first half of phone number digits or to keep the domain part of email addresses. The proposed changes should not affect any End User scripts currently running because the data will still be in the valid RIPE RPSL format. We propose to run the two datasets in parallel for some time to ensure a smooth transition. Further details and examples of how the dummified objects will change are detailed in a RIPE Labs article at: <https://labs.ripe.net/Members/kranjbar/proposed-improvements-to-dummificati…> Please let us know if you agree with the proposed improvements. Regards, Denis Walker Business Analyst RIPE NCC Database Group

4 4

Re: [db-wg] Removing personal data from bulk output from the RIPE Database
by Tore Anderson 10 May '13

10 May '13

* denis(a)ripe.net > Everyone is correct in what they have said. But when it comes to the > "changed:" attribute you have to rely on assumptions, estimates and 'good > enough' because the dates are not precisely defined. It is possible 90% of > changed dates are spot on, but you don't know that. > > No one would suggest that anyone would deliberately enter the wrong date. > And of course if you don't add a date the software will generate the > correct date for you. But if you wrote a template some time ago for > creating assignments and that template has a date in it, then everytime > you fill in the template to create a new assignment, it will have the > wrong date. Or if you 'change' the date on a single "changed:" attribute > every time you update the object, it will only show the 'last updated' > date. Or as Kaveh pointed out, maybe the object was created 10 years ago, > but you updated the information last week, without altering the changed > date. Do you consider the object to be stale data? There are so many ways > this date can represent something different to what you assume it to be. So while I have no illusions about that I'd get an exact number from looking at the "changed:" attribute, what I was really trying to say is that tried to use the database dumps on ftp.ripe.net for some purpose, but due to excessive dummification I couldn't. Hence I support the proposal to reduce the amount of dummification. Whether or not my desired query would stand up to any "scientific" scrutiny is really besides the point. That said, going slightly off topic wrt. the proposal itself: I'm still no closer to finding an answer to my question, which could be stated as: «How many IPv4 PA Assignments have been made by RIPE NCC LIRs since the 14th of September 2012? (If possible, but less important: How many addresses did these assignments consist of, in total?)» Even a very rough estimation is useful to me. Even just the order of magnitude would be interesting. "A dozen", "about five thousands", or "hundreds of thousands" - these are all results I could put in my slides to underline the point I want to make. While I think the use of the "changed:" attribute I originally planned would give me a useful enough result, I'm in no way married to that particular approach. So if there's some other way to get the number, possibly by having any of you RIPE NCC guys fetch it for me from some internal data source, that would be much appreciated! :-) I'd need it before Wednesday (well, preferably ASAP) if I'm going to be able to make use of it though. > As Kaveh also pointed out, you can query an object with the > '--list-versions' option and get a more accurate, reliable list of dates > for when an object was created and each update. But if you want to find > which PA assignments were created within a time period, you would have to > query them all to get the accurate historical dates. There are 3.7 million > assignment objects. For this type of analysis it might be easier if the > split file had a well defined date(s) you could rely on. No, I really wouldn't want to script 3.7M queries to your whois server...having an auto-generated "created:" attribute in the dumps (and in the whois service too for that matter) would certainly be better. Tore

2 2

Re: [db-wg] Removing personal data from bulk output from the RIPE Database
by James A. T. Rice 09 May '13

09 May '13

On Thu, 9 May 2013, denis(a)ripe.net wrote: > The "changed:" attribute is almost useless. Disagree, it gave a good indication of how stale an object was likely to be. If an object is is 15 years old, one wouldn't be too surprised if it's inaccurate. If it's from last month, then it's a lot more likely to be accurate. Cheers James

3 2

Re: [db-wg] Removing personal data from bulk output from the RIPE Database
by Tore Anderson 09 May '13

09 May '13

* denis(a)ripe.net > Unfortunately no dump will help you with this process. The "changed:" > attribute is almost useless. For what you want to do it IS useless. The > syntax requires all objects to have at least one mandatory "changed:" > attribute. The user can set the date to almost anything. It can be > tomorrow or ten years ago. When the object is changed, there is no > requirement to make any changes to this date or add any additional > "changed:" attributes. It is user generated and only has meaning to the > user. What you want to do would only work if these were auto generated > timestamps by the update software. But that is not the case. That may be so. But to turn it around - are you of the opinion that the LIRs to a great extent are deliberately supplying incorrect changed dates? If "no", it would be good enough for me, really. For my uses, an approximation will suffice. To be honest, I would actually be more concerned about the number being wrong because of LIRs assigning addresses to their customers without registering the assignments in the database, than I would be about LIRs registering their assignments, but with incorrect dates. BTW - the update software *does* generate time stamps. At least, I always send only my e-mail address, the bot does the rest. Tore

1 0

Re: [db-wg] Removing personal data from bulk output from the RIPE Database
by Tore Anderson 09 May '13

09 May '13

(Apologies for breaking threading, I just subscribed to the list.) In general, the proposal seems fine to me and I agree with its goals. In preparing for my APWG presentation next week, I've been wanting to find the answer to the question «how many IPv4 PA Assignments have been registered between dates A and B?». I thought I could do this by examining the inetnum database export - but that didn't work out, due to the fact that all the "changed:" attributes gets rewritten to "unread‌‌(a)ripe.net 20000101". So while I don't care about the e-mail addresses, I'd need the dates to be left intact. Also, the current algorithm folds all changed attributes into a single one. It's not clear from the Labs article whether or not this practise will stop - I hope it will. Would it by any chance be possible to get a sneak preview of the new inetnum dump, so that I could find the numbers I want before heading to Dublin? If so that would be greatly appreciated! Best regards, Tore Anderson

2 1

Reclaim Functionality for Resource Holders
by Denis Walker 29 Apr '13

29 Apr '13

Dear colleagues, With the new RIPE Database software now in place, the RIPE NCC has been able to automate a process that was previously handled manually. This process now allows resource holders to reclaim any operational database object related to the resources allocated and assigned to them by the RIPE NCC. You can find more details in a new article on RIPE Labs: https://labs.ripe.net/Members/denis/reclaim-functionality-for-resource-hold… Regards Denis Walker Business Analyst RIPE NCC Database Group

2 1

RIPE66 DB-WG Draft Agenda V1
by Wilfried Woeber 17 Apr '13

17 Apr '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear DB-WG folks, Chairpersons, "Meeting"! This is the 1st draft of an agenda for the DB-WG Meeting at RIPE66 in Dublin, May 15, 2013. For time slot allocations please refer to the most up-to-date meeting plan at https://ripe66.ripe.net/programme/meeting-plan/ The DB-WG Meeting is scheduled for *Wednesday*, starting at 09:00 local time / 08:00 UTC in the morning. The length of our timeslot is 90 Minutes. The overall (draft) RIPE 66 meeting outline is already available at https://ripe66.ripe.net/programme/meeting-plan/ Please feel free to propose additional topics to be discussed! I'd also suggest that you submit your points of view regarding the topics on the proposed agenda on the mailing list, preferably before the WG meeting's start :-) Best regards, hope to see you soon in Dublin, Wilfried. ________________________________________________________________________ A. Administrative Matters . Welcome . select scribe . finalise agenda . approval of minutes from previous WG meeting(s) . review of action list B. Data Base Operational Update (Kaveh Ranjbar, RIPE NCC) C. New DB-Software functionality - in beta - proposed D. Status update on IETF WEIRDS activities (N.N. t.b.c.) Y. Input from other Working Groups and/or Task Forces - review of abuse contact stuff (N.N. t.b.c.)) Z. AOB ________________________________________________________________________ 17.04.2013 16:00 UTC -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (MingW32) iQCVAwUBUW7IBd/RXX7wrLNpAQJymwP8D1Vt3oXoSkrfFuX05mNS2FLNIwucVYha uiAV89Zzk7uTxXa8dAQDpJgrVXIqaqaUgTjqpl1k8kYC85iuFsC8SEvjqQyx4czI oO99voS2x/qDXKI3nJj++4nuvswdDutYxSS5/ETk46sNWR8lsscMJ3Y8li1ttLEt eAJzPP8s9SQ= =YObZ -----END PGP SIGNATURE-----

1 0

Object Tags: a new feature proposal
by Kaveh Ranjbar 17 Apr '13

17 Apr '13

Dear colleagues, We would like to propose a method for providing metadata about objects in the RIPE Database. This new feature will enable a lot of flexibility to inform users about the state of each object and will help improve the quality of data in the RIPE Database. It can also provide a mechanism to fine-tune RIPE Database query outputs, which will give users who have less knowledge about address hierarchy, placeholder objects, etc. easier access to the data they are looking for. You can find more information about this proposal, along with some examples, at: https://labs.ripe.net/Members/kranjbar/ripe-database-object-tags Please kindly provide us with your feedback. Kind regards, Kaveh Ranjbar RIPE NCC Database Group Manager

1 0

Access to historical data in the RIPE database
by Nigel Titley 16 Apr '13

16 Apr '13

Following the announcement and the general positive feedback on this list, the RIPE NCC will be shortly enabling access to historical data in the RIPE database. Best regards Nigel

2 1

Addition of long query options
by Kaveh Ranjbar 05 Apr '13

05 Apr '13

Dear colleagues, With the latest RIPE Database software release, we introduced the choice of longer names for query options. Now each existing query option has an equivalent long name starting with a double dash. These names are more descriptive than the single character options. For example, "-x", which represents exact match searches, has "--exact" as its longer equivalent. The existing behaviour doesn't change; these long names can be used in addition to the short ones, which are all still valid. Using these longer names allows for more descriptive queries. When a query is passed on to a colleague or when it is coded inside a script, the reader will have a better understanding of what the query is searching for without having to consult the documentation. It also allows more flexibility to introduce new query options as more features are added to the RIPE Database. Below you will find the full list of RIPE Database query server options. This list can be generated any time with the query "whois -h whois.ripe.net help" as well. Kind Regards, Kaveh Ranjbar, RIPE NCC Database Group Manager. ... % The following options are available: % % -x, --exact <ip-lookup> % Requests that only an exact match on a prefix be performed. If no % exact match is found no objects are returned. % % -l, --one-less <ip-lookup> % Returns first level less specific inetnum, inet6num or route(6) % objects, excluding exact matches. % % -L, --all-less <ip-lookup> % Returns all level less specific inetnum, inet6num or route(6) % objects, including exact matches. % % -m, --one-more <ip-lookup> % Returns first level more specific inetnum, inet6num or route(6) % objects, excluding exact matches. % % -M, --all-more <ip-lookup> % Returns all level more specific inetnum, inet6num or route(6) % objects, excluding exact matches. % % -C, --no-irt <ip-lookup> % Turns off default option '-c' or '--irt'. % % -c, --irt <ip-lookup> % Requests first level less specific inetnum or inet6num objects with % the "mnt-irt:" attribute (enabled by default). % % -b, --abuse-contact <ip-lookup> % Requests first level less specific inetnum or inet6num objects with % the "abuse-c:" reference. Only object keys and "abuse-mailbox:" % attributes are visible. % % -d, --reverse-domain <ip-lookup> % Enables use of the -m, -M, -l and -L flags for lookups on reverse % delegation domains. % % -i, --inverse <attribute-name> <inverse-key> % Perform an inverse query. % % -F, --brief % Produce output using short hand notation for attribute names. % % -K, --primary-keys % Requests that only the primary keys of an object to be returned. The % exceptions are set objects, where the members attributes will also % be returned. This flag does not apply to person and role objects. % % -k, --persistent-connection (optional normal query) % Requests a persistent connection. After returning the result the % connection will not be closed by the server and a client may issue % multiple queries on the same connection. % Note, that server implements 'stop-and-wait' protocol, when no next % query can be sent before receiving a reply for the previous one. % Except the first -k query, -k without an argument closes the % persistent connection. % % -G, --no-grouping % Disables the grouping of objects by relevance. % % -B, --no-filtering % Disables the filtering of "notify:", "changed:" and "e-mail:" % attributes. % % -r, --no-personal % Switches off recursion for contact information after retrieving the % objects that match the lookup key. % % -T, --filter-types (comma separated list of object types, no white space is allowed) % Restricts the types of objects to lookup in the query. % % -a, --all-sources % Specifies that the server should perform lookups in all available % sources. See also "-q sources" or "--all-sources" query. % % -s, --sources (comma separated list of sources, no white space is allowed) % Specifies which sources and in which order are to be looked up when % performing a query. % % -q (sources|version|types) % "sources" see list-sources. % "version" see version. % % --list-sources % Returns the current set of sources along with the information % required for mirroring. See [REF], section 2.9 "Other server % features" for more information. % % --version % Displays the current version of the server. % % -t, --template <object-type> % Requests a template for the specified object type. % % -v, --verbose <object-type> % Requests a verbose template for the specified object type. % % -V, --client <client-tag> % Sends information about the client to the server. ...

1 0

Access to historical data in the RIPE Database
by Kaveh Ranjbar 04 Apr '13

04 Apr '13

Dear colleagues, We would like to propose making the history of changes to objects available through the RIPE Database. This is one of the most requested features and will help different groups of users by providing more information about objects in the database. Our proposal, along with examples and details about possibilities and restrictions, is published at: https://labs.ripe.net/Members/kranjbar/proposal-to-display-history-of-objec… If accepted, we can enable this feature in a Beta state quite quickly. Please kindly provide us with your feedback. Kind regards, Kaveh Ranjbar, RIPE NCC Database Group Manager

2 1

[news] Phase 1 Update: Abuse Contact Management in the RIPE Database
by Kaveh Ranjbar 03 Apr '13

03 Apr '13

[Apologies for duplicate emails] Dear colleagues, Phase 1 of ripe-563 "Abuse Contact Management in the RIPE Database" is now being implemented, as announced on 19 March 2013. According to ripe-563, all resources allocated or assigned by the RIPE NCC will need to have an abuse contact email address included in their RIPE Database registration. During the first six months of implementation (phase 1), all Local Internet Registries (LIRs) must add an abuse contact attribute ("abuse-c:") to their LIR's ORGANISATION object to provide abuse contact information for all of the allocated address space under their LIR. Since the initial announcement two weeks ago, we are pleased to report that 24.8% of the RIPE NCC's allocated IPv4 address space is already covered with an abuse contact. More statistics will be sent to the RIPE Anti-Abuse Working Group mailing list. In phase 1, we will send notifications to LIRs that haven't added abuse contact information to their LIR's ORGANISATION object. If this information is not added by the LIR by the end of phase 1, we will automatically add the LIR's contact email address (which is already publicly listed at: <https://www.ripe.net/membership/indices/>) as their abuse contact. LIRs can always change this information as needed. More information on the implementation plan, including phase 2 (covering PI space and ASNs) is available on RIPE Labs at: <https://labs.ripe.net/Members/kranjbar/implementation-details-of-policy-201…> ------------------------------- How to add "abuse-c" ------------------------------- An explanation of how to add "abuse-c:" to an LIR's ORGANISATION object as well as how to fine-tune abuse contact information for sub-allocations and assignments is outlined here: <https://labs.ripe.net/Members/denis/creating-and-finding-abuse-contacts-in-…> Please kindly consider adding abuse contact information as soon as possible to the ORGANISATION object(s) you control or maintain in the RIPE Database. Kind Regards, Kaveh Ranjbar, RIPE NCC Database Group Manager

1 0

Call for agenda items, DB-WG Meeting during RIPE66, Dublin, IE
by Wilfried Woeber 22 Mar '13

22 Mar '13

Dear DB-WG folks, planning activities for the next RIPE Meeting - RIPE66 in Dulin, IE, May 13 - May 17, 2013, have already started a while ago - thus this _call for agenda items_ for the meeting of the Database Working-Group. Please submit (to the WG list!) your item requests and|or suggest topics for discussion to be included in the draft agenda! Please note that the DataBase-WG meeting is *at the moment* scheduled for *Wednesday* (starting at 9:00 local time). This is according to the *draft* meeting plan - which is pretty stable already but may still need to be changed as we get closer to the meeting date! The overall (draft) RIPE 66 meeting outline is already available at https://ripe66.ripe.net/programme/meeting-plan/ Looking forward to a successful meeting and see you soon in Ireland, Wilfried.

1 0

Status of the language: attribute
by Jan-Erik Eriksson 21 Mar '13

21 Mar '13

Dear DB-WG members, My name is Jan-Erik Eriksson, representing the ccTLD AX I would like to express our strong support for the continuance of the, currently beta state, attribute 'language:' for the inetnum and inet6num object type in the RIPE DB. Background: AX is the ISO country code for the Åland Islands. The Åland Islands is an autonomous, demilitarized and neutralized region belonging to Finland. Its official language is Swedish. Åland are governed according to the Act on the Autonomy of Åland and international treaties. The autonomy is well developed and in many ways Åland function like a sovereign state. The symbols of the autonomy can be seen in the own national flag, stamps, the Parliament building and internet top-level domain (.ax). Our special situation as a Swedish speaking area with the country of Finland have and still are imposing great challenges for s in the on-line world. Typically web services, ads, media and other on-line content is presented to our users in the wrong language. To solve these issues today the only way is to approach the presenter of that content, for instance Google, and lobby for them to handle this situation. As you all understand, this task is impossible considering the number of parties that we would need to contact. In this respect, the language attribute, if supported by the RIPE/Internet community, could serve as a de facto source of reference for both operators, content providers and geoip suppliers. We also believe there are other multi-language countries in Europe that could benefit from using this attribute, should there be a widespread adoption of its use. Sincerely, Jan-Erik Eriksson CTO Ålands Telekommunikaton Ab

4 4

RIPE abuse finder
by Max Tulyev 21 Mar '13

21 Mar '13

Hello! Recently our company started to receive a lot of strange complaints relating to network we don't operate or route. I fount these complaints was sent to points of contact got by RIPE abuse finder. This is the fresh sample of it: IP: 31.133.58.219 Output of RIPE abuse finder: https://apps.db.ripe.net/whois/use-cases/abuse-finder.json?source=ripe&prim… { "abuse-resources":{ "service":"abuse-finder", "link":{ "xlink:type":"locator", "xlink:href":"http://apps.db.ripe.net/whois/use-cases/abuse-finder.json?source=ripe&prima…"} , "parameters":{ "primary-key":{ "value":"31.133.58.219"} , "sources":{ "source":{ "name":"RIPE", "id":"ripe"} } } , "irt-objects":null, "abuse-mailbox-attributes":{ "attribute":{ "name":"abuse-mailbox", "value":"support(a)netassist.ua"} } , "objects-with-abuse-related-remarks":null} } whois -B 31.133.58.219 % This is the RIPE Database query service. % The objects are in RPSL format. % % The RIPE Database is subject to Terms and Conditions. % See http://www.ripe.net/db/support/db-terms-conditions.pdf % Information related to '31.133.32.0 - 31.133.63.255' inetnum: 31.133.32.0 - 31.133.63.255 netname: DIDAN descr: FOP Trubnikov Valeriy Muhaylovich country: UA org: ORG-FTVM1-RIPE admin-c: TA3197-RIPE tech-c: TA3197-RIPE status: ASSIGNED PI mnt-by: RIPE-NCC-END-MNT mnt-lower: RIPE-NCC-END-MNT mnt-by: TRUBNIKOV-MNT mnt-routes: TRUBNIKOV-MNT mnt-domains: TRUBNIKOV-MNT changed: hostmaster(a)ripe.net 20110415 changed: hostmaster(a)ripe.net 20110415 source: RIPE organisation: ORG-FTVM1-RIPE org-name: FOP Trubnikov Valeriy Muhaylovich org-type: other address: 86156, Ukraine, Makiyivka, Zeleniy, 60/182 admin-c: TV1866-RIPE tech-c: TA3197-RIPE e-mail: core(a)didan.org mnt-ref: TRUBNIKOV-MNT mnt-by: TRUBNIKOV-MNT changed: core(a)didan.org 20110414 source: RIPE person: Trubnikov Andrey address: 86156, Ukraine, Makiyivka, Zeleniy, 60/182, phone: +380623277657 nic-hdl: TA3197-RIPE changed: core(a)didan.org 20110331 mnt-by: TRUBNIKOV-MNT source: RIPE % Information related to '31.133.48.0/20AS52091' route: 31.133.48.0/20 descr: FOP Trubnikov Valeriy Muhaylovich origin: AS52091 mnt-by: TRUBNIKOV-MNT changed: core(a)didan.org 20110423 source: RIPE % Information related to 'AS52091' aut-num: AS52091 as-name: TRUBNIKOV-AS descr: FOP Trubnikov Valeriy Muhaylovich org: ORG-FTVM1-RIPE import: from AS47694 accept ANY export: to AS47694 announce AS52091 import: from AS21189 accept ANY export: to AS21189 announce AS52091 admin-c: TA3197-RIPE tech-c: TA3197-RIPE mnt-by: RIPE-NCC-END-MNT mnt-by: TRUBNIKOV-MNT mnt-routes: TRUBNIKOV-MNT changed: hostmaster(a)ripe.net 20110415 source: RIPE organisation: ORG-FTVM1-RIPE org-name: FOP Trubnikov Valeriy Muhaylovich org-type: other address: 86156, Ukraine, Makiyivka, Zeleniy, 60/182 admin-c: TV1866-RIPE tech-c: TA3197-RIPE e-mail: core(a)didan.org mnt-ref: TRUBNIKOV-MNT mnt-by: TRUBNIKOV-MNT changed: core(a)didan.org 20110414 source: RIPE person: Trubnikov Andrey address: 86156, Ukraine, Makiyivka, Zeleniy, 60/182, phone: +380623277657 nic-hdl: TA3197-RIPE changed: core(a)didan.org 20110331 mnt-by: TRUBNIKOV-MNT source: RIPE whois -B TRUBNIKOV-MNT % This is the RIPE Database query service. % The objects are in RPSL format. % % The RIPE Database is subject to Terms and Conditions. % See http://www.ripe.net/db/support/db-terms-conditions.pdf % Information related to 'TRUBNIKOV-MNT' mntner: TRUBNIKOV-MNT descr: FOP Trubnikov Valeriy Muhaylovich admin-c: TA3197-RIPE upd-to: core(a)didan.org auth: MD5-PW # Filtered mnt-by: TRUBNIKOV-MNT referral-by: TRUBNIKOV-MNT changed: core(a)didan.org 20110418 source: RIPE # Filtered person: Trubnikov Andrey address: 86156, Ukraine, Makiyivka, Zeleniy, 60/182, phone: +380623277657 nic-hdl: TA3197-RIPE changed: core(a)didan.org 20110331 mnt-by: TRUBNIKOV-MNT source: RIPE Is it a bug or a feature, and why?

5 14

Soft launch of "abuse-c:", ripe-563: Improving Abuse Contact Information
by Kaveh Ranjbar 19 Mar '13

19 Mar '13

[Apologies for duplicate emails] Dear Colleagues, RIPE NCC has initiated a soft launch of phase one of RIPE Policy 563: "Abuse Contact Management in the RIPE Database". All facilities to provide abuse contact information for organisations registered in the RIPE Database are be in production state. Please see the announcement sent earlier to the RIPE Anti-Abuse Working Group mailing list (http://www.ripe.net/ripe/mail/archives/anti-abuse-wg/2013-March/002226.html). Kind Regards, Kaveh Ranjbar Database Group Manager RIPE NCC

1 0

Redevelopment of the RIPE Database software is complete
by Kaveh Ranjbar 25 Feb '13

25 Feb '13

Dear Colleagues, The RIPE NCC is proud to announce that our redevelopment of the RIPE Database software is complete. At RIPE 63, the RIPE NCC announced that we were starting a project to replace the 11-year old legacy RIPE Database software with a new system. At RIPE 64, we were able to report completion of the new query software and the start of the update project. The new query software has been handling all queries for almost a year, and the new update software is now in place and handling all updates. This new software is able to scale the RIPE Database to meet the expanding requirements of the members, the RIPE community and the RIPE NCC. It will also improve our response times on bug fixes, changes and major feature requests. With near complete unit test coverage of the code, higher level integration tests and extensive end to end 'use case' testing we have greatly improved the maintainability and reliability of the entire RIPE Database infrastructure. The new systems were rolled out incrementally, so the process was almost invisible to users of the service. The new software is mostly backwards compatible, although we have introduced some long-anticipated features and requirements. On our website, we are publishing software release notes that are updated for each new release. These can be found here: http://www.ripe.net/data-tools/db/release-notes With the new software now in place, we have a lot of opportunity for future expansion of the service and to implement features desired by the members and the wider community. During the redevelopment process we also had the chance to improve the behaviour of our system. Some of the areas improved are: - Some community-approved policies which were not fully enforced within the RIPE Database are now enforced by the new software for all RIPE address space. - A lot of issues in the authorisation are resolved. Resource holders now have full control over the resources they maintain and situations where they might lose control of their resources by mistake are no longer possible. - All resources allocated or assigned by the RIPE NCC have their entries in the RIPE Database protected against accidental deletion by users. - The new system accepts IPv6 network addresses in different formats, but always normalises the input to a single format which is used for storing, searching and displaying the records. If you query, create, modify or delete an object, no matter what format you specify the address in, it will first be normalised, then the operation will be actioned. In general, the new system is much smarter in the interpretation of IPv4 and IPv6 addresses on input and can handle different notations. - New error messages are more comprehensive and the system provides more specific error messages and logging and in many different situations. - The new system is about four times faster on queries and three times faster on processing updates. We have published an article on RIPE Labs with details about the features in the new system: https://labs.ripe.net/ripe-database/changes-to-the-processing-of-ripe-datab… Internally, the new system has a lot of benefits as well: - The new software has been designed from scratch with distribution in mind. Additional query and update nodes can easily be added for redundancy and performance. - Hot deployments of new software are now possible with no interruption to the service. - Some additional security measures have been introduced to improve the overall security of the system. - Logging and debugging facilities have been improved for easier support and maintenance. - The whole system is integrated and self-contained. It doesn't rely on external scripts for any task (such as the import of external data for mirroring or automatic data clean-up of unreferenced objects) and so is much easier to maintain. - Internal management and monitoring facilities have been improved and integrated into the system, so support and maintenance can provide users with direct feedback from the production system. - The memory footprint of the new system is much smaller. - The code base is much cleaner, making it easier to understand and maintain. Java is now the single language, replacing a mix of C, Perl and Java code. So, what's next? - Currently we are working on a new set of RIPE Database documentation that will be clean, concise and linked directly to each release, including a further review of error messages and all information provided when interacting with RIPE Database. - Investigate with the community the integration of RIPE Database authentication with the single sign-on (SSO) service for RIPE NCC services. - Implementing the IETF WEIRDS RDAP Protocol, fully integrated into the RIPE Database API, as a cooperative software development project with APNIC. - Using business rules to 'fix' parts of resource objects that users should not change, then allowing all objects to be updated by normal methods rather than special editors in the LIR Portal. - Providing the new software to the public and making the project open source and accessible for third parties. - Liaise with the community on new features and data clean-up as announced at the last RIPE Meeting. As always, you can contact us at <ripe-dbm(a)ripe.net> for any issues, feature requests or suggestions. Kind regards, Kaveh Ranjbar Database Group Manager RIPE NCC

2 1

[training] RIPE NCC Training Courses April-June 2013 - BA UK AE DE NL PL LI RU FR ES HU FI
by Training Mailbox 25 Feb '13

25 Feb '13

Dear Colleagues, Our training team travels the RIPE NCC service region to deliver training courses to our members without any additional cost. Over the next few months, we'll be in Sarajevo, Leeds, Dubai, Berlin, Amsterdam, Warsaw, Vaduz, St. Petersburg, Marseilles, Madrid, Budapest, Helsinki. Visit the following page to register and to check which training courses we are giving in your area: https://lirportal.ripe.net/training/courses The RIPE NCC delivers the following training courses: - LIR Training Course - Database Training Course (new) - IPv6 for LIRs Training Course - Routing Security Training Course For more information visit: http://www.ripe.net/lir-services/training/courses With kind regards, Rumy Spratley-Kanis Training Services Manager

1 1

bug? accepting larger than 32 bit BGP Communities in RPSL
by Job Snijders 07 Jan '13

07 Jan '13

Dear RIPE DB group, A small snippet from the current AS15562 object as stored in the RIPE DB. remarks: Larger than 32bit BGP Community test remarks: 64 bit export: to AS199036 action community .= { 0000199036:0000199036 }; announce AS-SNIJDERS The value "0000199036:0000199036" is a 64 bit integer: (199036<<32)+199036 = 854853110925692 854853110925692 Is far larger than what RFC 2622 [1] allows in section 7.1: typedef: # a community value in RPSL is either # - a 4 byte integer (ok to use 3561:70 notation) # - internet, no_export, no_advertise (see RFC-1997 ) community_elm union integer[1, 4294967295], enum[internet, no_export, no_advertise], So my main question is, why is the RIPE DB allowing such values to be stored? This is not conforming to any specification that I am aware of. Currently there is no consensus on what the future of BGP communities will look like within the IETF. Kind regards, Job [1] http://tools.ietf.org/html/rfc2622

6 8

Severe performance problems with RIPE whois server
by Nick Hilliard 27 Dec '12

27 Dec '12

So the new kid on the block for issuing IRR querys is bgpq3: http://snar.spb.ru/prog/bgpq3/ (Thanks to Job Snijders for pointing it out). Here are some comparative results for bgpq3 and irrtoolset using a random large-ish as-set with ~5000 ASNs: > ubuntu1:/home/nick/bgpq3-0.1.16> time bgpq3 AS-NEOT > /dev/null > 0.144u 0.032s 0:04.26 3.9% 0+0k 0+0io 0pf+0w > ubuntu1:/home/nick/trunk/src/peval> time ./peval -h whois.radb.net -protocol irrd AS-NEOT > /dev/null > 8.600u 0.516s 10:14.14 1.4% 0+0k 0+0io 0pf+0w > ubuntu1:/home/nick/trunk/src/peval> time ./peval -h whois.ripe.net -protocol ripe AS-NEOT > /dev/null > 2.900u 0.304s 12:21.72 0.4% 0+0k 0+0io 0pf+0w Due to its use of pipelining it can return IRRD query results ridiculously quickly, and it's pretty clear that irrtoolset is being completely trashed by comparison because it doesn't support pipelining. The difference between the RIPE and RADB whois server is related to the fact that the RADB server supports recursive as-set expansion. In fact, a more interesting question is not why bgpq3 is 175 times faster than irrtoolset, but rather why it takes as long as 4.26 seconds to complete an inverse query for the as-set in question. If I did ran a SQL query on a single term with indexed lookups and which returned 50k results, I'd wonder why it was so slow. In this case, it turns out that the answer mostly relates to the amount of data returned (~750k), and the rtt to the whois server from my query box (~150ms), rather than as a result of the query latency. Anyway, it should be clear that for large query sets, the RIPE server is extremely slow because of its lack of command pipelining and lack of support for recursive expansion of certain types of objects. This directly affects INEX because we run a full rebuild of our route server configuration three times daily. Our route server as-set (AS-SET-INEX-RS) contains about 5400 ASNs and the consequent serialisation delays are currently causing our rebuild scripts to take slightly more than 30 minutes to complete. This varies depending on client policies: earlier this year, rebuilds were taking about 2 hours. In fact for performance reasons, we need to query whois.radb.net instead of RIPE for the larger AS-SETs. This is because of the recursive as-set performance issue which affects queries to whois.ripe.net In order to solve this problem, what we'd really love to see is: 1. support for command pipelining 2. support for recursive expansion of as-set to a list of ASNs. 3. support for recursive inverse key expansion of an as-set to a list of prefixes These would mean a massive performance gain for INEX's use case (and a bunch of other IXPs around europe) and #3 would mean that the RIPE whois server could leapfrog IRRD in terms of performance. Would it be possible for the DB people to look into this? Nick

4 4

RIPE NCC Webinars - new dates
by Training Mailbox 21 Dec '12

21 Dec '12

[Apologies for duplicate e-mails] Dear colleagues, We are pleased to announce the launch of new dates for our Webinars for LIRs. The RIPE NCC Webinars are live, one hour online training courses that allow participants to interact with our trainers without leaving their desks. We focus on the topics and issues most important for LIRs. Register now at https://www.ripe.net/lir-services/training/e-learning/webinars Participation is limited to 20 people, so don't hesitate if you want to take part! If you have questions, please email <e-learning(a)ripe.net>. We look forward to seeing you online. Kind regards, RIPE NCC Training Services

1 1

Re: [db-wg] [anti-abuse-wg] RIPE NCC's proposed implementation of Abuse Contact Management in the RIPE Database
by Denis Walker 13 Dec '12

13 Dec '12

Dear Gert and other colleagues, After some off list clarification we see the point you are concerned with. The simplest case where an organisation manages all the abuse for all customers is easy to understand. In the real world this organisation has one or more internet resources and within the organisation there is a function (or role) that handles abuse complaints. So a real world picture is clear. Taking a member organisation as an example: member organisation / \ / \ internet abuse resources role This maps exactly to the new RIPE Database model: member ORGANISATION object / \ / \ INET(6)NUM member abuse objects ROLE object It is easy to see how to find the abuse handling role for any of these internet resources. The problem appears to be with the vision of what happens lower down in a network. After years of modifications the data model of the RIPE Database no longer reflects the real world. The relationships between objects are less than optimal. If you try to think of anything in terms of RIPE Database objects and relationships, it is not always obvious or clear. The implementation of the abuse-c proposal takes one small step to bringing the RIPE Database model closer to reflecting the real world, as outlined in the Database Groups presentation at RIPE 65. When a customer wants to handle abuse for their part of a network, they are taking over part of the management of that internet resource. In reality the customer's situation is the same as the member's. member ORGANISATION object / \ / \ INET(6)NUM abuse objects ROLE object / \ / \ / \ customer ORGANISATION object / \ / \ / \ / \ other INET(6)NUM customer abuse customers object ROLE object INET(6)NUM objects By including the customers ORGANISATION object the same structure is modelled at any level. This is the basic structure of any organisation that manages internet resources. So now it is easy to see how to find this customers abuse handling role and also how to find the abuse handling role for any of the other customers. This structure can be repeated as many times as necessary at any point in a network. We don't have abuse contacts hanging off different object types. The principle of finding the abuse contact is always the same. That is what makes the model simple in all cases. Internet resources includes AUT-NUM objects as well, and an organisation can have multiple top level INET(6)NUM objects, but to keep the diagram simple I ignored them. I hope this helps to explain the reasoning behind this model. To help with practical administration the RIPE NCC will provide some easy to use tools initially so a customers details only need to be entered once and appropriate objects will be created and the references made. To remove this setup for a specific customer just delete the reference to the ORGANISATION object. The extended garbage cleanup process will delete any cluster of ORGANISATION, MNTNER, ROLE, PERSON objects that are unused. We will also provide more extensive tools for managing changes to contact data at any level across multiple objects. Regards, Denis Walker Business Analyst RIPE NCC Database Group On 10/12/2012 14:42, Gert Doering wrote: > Hi, > > On Wed, Dec 05, 2012 at 11:10:09AM +0100, Denis Walker wrote: >> If at some point a customer chooses to do their own management of abuse >> handling for their assigned address space you simply create the >> ORGANISATION and abuse ROLE objects for this customer and add the "org:" >> reference to this customers assignment. > > I'm not sure if I find this "simple", to be honest. > > Being able to specify an abuse-c: in the ASSIGNED inet(6)num: object, > and having the RIPE DB software honour that on queries - that is: not > go to the allocation's org->abuse-c - would require less objects to be > created, and thus, less effort and less opportunity for confusion. > > Gert Doering > -- NetMaster >

2 2

RIPE NCC's proposed implementation of Abuse Contact Management in the RIPE Database
by Kaveh Ranjbar 07 Dec '12

07 Dec '12

Dear Colleagues, The RIPE NCC's proposed implementation plan of RIPE policy 563 (policy proposal 2011-06) titled "Abuse Contact Management in the RIPE Database" is as follows: =========== Introduction =========== Abuse Contact Management in the RIPE Database (as defined in RIPE policy document 563) mandates the RIPE NCC to gather and publish abuse contact information for Internet resources provisioned by the RIPE NCC. =========== Approach =========== Abuse handling is generally a role within an organisation and the RIPE NCC is planning to model this relation in the RIPE Database. So the "abuse-c:" attribute will be available in the ORGANISATION object and should reference a ROLE object. When a ROLE object is referenced from an ORGANISATION object by the "abuse-c:" attribute, it must have an "abuse-mailbox:" attribute. All resources allocated or assigned by the RIPE NCC are required to have a reference to an ORGANISATION object in the RIPE Database. By adding a single "abuse-c:" attribute to their ORGANISATION object, a resource holder can quickly cover all their resources. All resources that are allocated or assigned by the RIPE NCC and which are subject to and compliant with current resource policies will then either directly reference an abuse contact or will inherit one from the parent resource. The detailed changes to RIPE Database objects and business rules, fine tuning, search options and facilities for quick data entry by RIPE NCC members using the LIR Portal are all described in an article on RIPE Labs: https://labs.ripe.net/Members/kranjbar/implementation-details-of-policy-201… ============ Timeline ============ * Phase 0 - Q1 2013 The RIPE NCC will make internal changes to the RIPE Database and LIR Portal, this should be done by end of Q1 2013. * Phase 1 - Q2 and Q3 2013 The RIPE NCC will ask members to add "abuse-c:" to their member ORGANISATION object at least and will conduct a follow-up on that. RIPE NCC Members will be required to have abuse contact data for their resources when working with our Registration Services department. If a member has not added abuse-c:" to their ORGANISATION object by the end of this phase, the RIPE NCC will automatically add the member's published contact email address as the member's abuse contact. Members can edit this information any time through the LIR Portal. At the end of this phase, all PA allocations and their more specifics will have an abuse contact. * Phase 2 - Q4 2013 to Q3 2014 The RIPE NCC will ask AS Number and PI Resource holders to add abuse contact information to their ORGANISATION objects. If some resource holders have not added their abuse contact data by the end of this phase, the RIPE NCC will add the published abuse contact data for the responsible LIR to the ORGANISATION object listed in the resource. Any resource holder with access to the related object can always edit this information. At this point, ALL resources in the RIPE Database which are subject to and compliant with RIPE policies will have accessible abuse contact information. Please let us know if you like this proposal or have any feedback or ideas for improvement of this implementation proposal. Regards, Kaveh Ranjbar, RIPE NCC Database Group Manager

6 8

Implementation of 2011-06
by Brian Nisbet 05 Dec '12

05 Dec '12

Colleagues, Hopefully by now you have all read the mail from the NCC Database Group regarding the implementation plan for Policy 2011-06 sent on the 15th of November 2012. The Co-Chairs of the Anti-Abuse and Database WGs have conferred and we feel that while there have been a few questions regarding this plan, these have been answered and there were no objections so consensus has been reached in this regard. With this support from the community in mind, the Co-Chairs have asked the NCC to begin work on this implementation and to keep both WGs updated on their progress. As always, if you have any questions, please do raise them on the mailing lists. Brian, On behalf of the Co-Chairs of the AA-WG & DB-WG

1 0