- db-wg - mailman.ripe.net

Re: [db-wg] [opensource-wg] Question for RPSL Parser
by Stavros Konstantaras 20 Aug '15

20 Aug '15

Shane is right, I don’t have the intention to shuffle the project with Bison and Flex because 1) I don’t have the knowledge of these technologies 2) I don’t have the time to learn them 3) I want to Pythonize as much as I can and be independent as much as possible (OK maybe a little bit dependent from other Python Libraries but this is the boundary). I am sure that you guys made a great job 14 years before but now or soon RPSL will be replaced with something else so I still want my toolset to be alive by easily and quickly replacing 1-2 Python libraries. What my prototype does now is extract the attributes and values by using Regular Expressions, a solution that has already been proposed in this discussion and I think this is the way to go. In theory, it can be an easy parser based on Python’s RE but complexity starts when -for example- RPSL says “accept as-set BLABLA” instead of “accept 10.100.200.0/16" and that as-set needs to be resolved (another query to RIPE DB). But anyway, thanks for your ideas. Crazy or not, I don’t mind everyone is welcome. If I still need to reply to Tomas last e-mail but I am quite busy today. Kind Regards Stavros > On 20 Aug 2015, at 13:05, Shane Kerr <shane(a)time-travellers.org> wrote: > > Denis, > > There are two problems with this: > > 1. Lex and YACC (actually flex and bison for the RIPE Database) are > tightly integrated into C (or C++ if you are feeling brave). > > 2. Stavros does state a goal of avoiding reuse of past or legacy > software, which surely means the RIPE Database code. (Okay, I think > it was only 12 or 14 years ago when we wrote these parts, but that > surely counts as "past"...) > > :) > > Cheers, > > -- > Shane > > On Thu, 20 Aug 2015 10:56:20 +0000 (UTC) > denis walker <dw100uk(a)yahoo.co.uk> wrote: > >> HI >> The RIPE Database I believe still uses LEX and YACC to parse the routing policy information from the RPSL attribute/value pairs. Is it possible to just 'drop' this into a python script and extend it to do any extra parsing you want? (I know nothing about python so I may be way off the mark here :) but it is just a thought) >> cheersdenis >> From: Shane Kerr <shane(a)time-travellers.org> >> To: Stavros Konstantaras <stavros(a)nlnetlabs.nl> >> Cc: db-wg(a)ripe.net >> Sent: Thursday, 20 August 2015, 5:21 >> Subject: Re: [db-wg] [opensource-wg] Question for RPSL Parser >> >> Stavros, >> >> On Wed, 19 Aug 2015 16:00:59 +0200 >> Stavros Konstantaras <stavros(a)nlnetlabs.nl> wrote: >> >>> To make things more clear, our project at NLnet Labs is the >>> development of "a modern IRRtoolset" written in Python and targeting >>> any operator, not just another homemade tool for our own needs. Or to >>> say in more detail the creation of a tool that is able to configure >>> BGP routers directly+automatically by extracting the policies from >>> RIPE DB. This means that I am struggling to avoid the (re)use of any >>> past or legacy software that exists, otherwise we loose independency >>> and inherit restrictions from the past. >> >> I don't know of any Python RPSL parser you can use. >> >> Your goals make a certain amount of sense. Unfortunately RPSL is an >> ugly language... it's actually more like several languages in one, all >> bad. :P >> >> * Start with simple attribute/value, one per line >> >> * Oh, but then "RPSL-ize", which allows line continuation and >> end-of-line comments >> >> * Oh, actually we want to make a separate grammar for each attribute >> >> * But be sure to make all of this "extensible", to insure maximum >> confusion! >> >> My approach in the past has been to start with a simple generic parser >> that split text into objects, objects into attributes, extracts >> attribute name & values (handling line continuation and end-of-line >> comments). You can do all of this with Python regular expressions, >> something like r"\n(?![ \t+])" to split objects into attributes; >> cleaning end-of-line comments & line continuation characters is >> straightforward. Then you can deal with the grammar for the attributes >> you know, and ignore the rest. You can extend the parsing to be more >> and more comprehensive until you are able to parse the data set you >> worry about. >> >> I'm not sure this is worth doing... but if there was a good, >> easy-to-use Python library then maybe interesting things would result? >> >> >> >> Cheers, >> >> -- >> Shane >> >> >> >> >

2 1

Re: [db-wg] [opensource-wg] Question for RPSL Parser
by Stavros Konstantaras 20 Aug '15

20 Aug '15

Hi Thomas and others, thanks for your quick replies. To make things more clear, our project at NLnet Labs is the development of "a modern IRRtoolset" written in Python and targeting any operator, not just another homemade tool for our own needs. Or to say in more detail the creation of a tool that is able to configure BGP routers directly+automatically by extracting the policies from RIPE DB. This means that I am struggling to avoid the (re)use of any past or legacy software that exists, otherwise we loose independency and inherit restrictions from the past. Currently, I do have a working proof of concept with a simple RPSL parser written in Python which is able to parse simple policies with (v6)import(s)-(v6)export(s)-accept-announce-action-pref-med-coomunity attributes. Convenient I would say for many existing policies like KPN’s, RIPE’s etc. but definitely it will crash in complex policy files like GRNET’s. My current solution uses also BGPq3 to translate AS, AS-SET and ROUTE-SET and this is also something that I want to ged rid of. So I was looking for the existence of a Python Library which can do a complete job and translate any existing policy in an intermediate data structure like XML (this is what I do now). I was hoping to find one, adopt it and present our (completed) work in the next RIPE meeting (yes I know I am very ambitious, I need to slow down). * Masimo, I already checked rpsltool and discarded it. But thanks for your super-fast reply. * Tomas, thanks for pointing me to your library. I will have a look at it and check if it is convenient for our project. * George, yes you are right. This should be the #1 step, take corresponding RPSL RFCs and implement them in Python. However, that would take too much time and will not allow us to work on other missing functionalities of the prototype. And we would like to have something ready at the beginning of November. However, I will check your solution as well. I hope now that I made myself more clear. Big thanks to everyone, every help, info or idea is kindly appreciated. Kind Regards Stavros ---- Stavros Konstantaras Internet Research Engineer, NLnet Labs Science Park 400, 1098 XH, Amsterdam Fingerprint: 4502 7D16 2DF8 ADB0 4AA6 21A9 BF9E EFFF 2744 FE5D > On 19 Aug 2015, at 14:39, Tomas Hlavacek <tomas.hlavacek(a)nic.cz> wrote: > > Hi Stavros! > > On 08/19/2015 02:15 PM, Ondrej Filip wrote: >> >> Vis o necem? >> >> -------- Forwarded Message -------- >> Subject: [opensource-wg] Question for RPSL Parser >> Date: Wed, 19 Aug 2015 14:13:58 +0200 >> From: Stavros Konstantaras <stavros(a)nlnetlabs.nl> >> To: db-wg(a)ripe.net >> CC: opensource-wg(a)ripe.net >> >> >> >> >> Hi all WG members, >> >> For my project at NLnet Labs I would like to ask you if you know any >> (relatively) complete RPSL parser written in *Python*. I searched >> around and I didnât find anything. Does someone know any repository >> where I can get one or maybe contact someone who has written one? > > It depends on what you need... I wrote one - > https://github.com/tmshlvck/bgpcrunch , but the aim was rather data > analysis than "authoritative" RPSL translation. When it comes to filter > interpretation my approach is quite "heuristic" and it is focused on > speed and ability to take short-cuts which wouldn't be possible with > full parsing the RPSL objects according to EBNF metasyntax. > > Have you considered creating simple Python interface for peval from > IRRToolSet? It might do the job if you don't need any special features. > > Cheers, > Tomas

4 4

Question for RPSL Parser
by Stavros Konstantaras 19 Aug '15

19 Aug '15

Hi all WG members, For my project at NLnet Labs I would like to ask you if you know any (relatively) complete RPSL parser written in Python. I searched around and I didn’t find anything. Does someone know any repository where I can get one or maybe contact someone who has written one? Thanks in advance. Kind Regards Stavros ---- Stavros Konstantaras Internet Research Engineer, NLnet Labs Science Park 400, 1098 XH, Amsterdam Fingerprint: 4502 7D16 2DF8 ADB0 4AA6 21A9 BF9E EFFF 2744 FE5D

3 2

NCC still enforcing descr: content
by Jan Saner 15 Aug '15

15 Aug '15

Hello Since the NCC still seems to heavy-handedly revert changes to descr: in AUT-NUM, INETNUM and INETNUM6 I’m wondering what the current status of the implementation mentioned by Tim in a post to this list from February is: https://www.ripe.net/ripe/mail/archives/db-wg/2015-February/004496.html Quote: ----------------------------------------------------------- "> When will this change go into effect? We were thinking of mid April, for two reasons: 1) This gives the working group enough time to comment. 2) While the RIPE NCC has finished the effort to ensure that the org-id (and name) are set up correctly for all INETNUM and INET6NUM objects, we are still finalising this work for AUT-NUM objects. We expect that we will finish this effort in about 6 weeks. For consistency it would be easiest to change this for all resource types simultaneously." ----------------------------------------------------------- I personally know a case where the NCC went as far as to ignore German “umlauts” in the name and deliberately spell it wrong in the descr: field of the guy’s AUT-NUM object. When he tried to set it right (ae (ä) instead of a), the NCC reverted the change. Then there’s a case where someone tried to remove his middle name from the descr: field —> reverted. Another post on the same subject: https://www.ripe.net/ripe/mail/archives/members-discuss/2015-June/001737.ht… Looks like there’s still a lot of energy wasted in those descr fields now that the org pointer is mandatory. Cheers

10 17

Upcoming work by the RIPE NCC Database team
by Tim Bruijnzeels 14 Aug '15

14 Aug '15

Dear working group, Following working group discussions in different threads we would like to give you a quick overview of current and planned work. 1= SSO integration in Webupdates As you may know we have been working on this in the background for a while. The objective of this effort is to improve the integration of SSO with Webupdates. Where previously we were asking users to provide plain text (MD5) passwords, and storing them in the session, we now want to encourage everyone to use their SSO account instead. We expect that we will finish this work 2-3 weeks from now, and plan to deploy a new release 1.81 to the RC environment then. 2= Changed phase 3 - deprecating "changed:" We plan to work on this as soon as the SSO release is deployed to the RC environment. We we expect that we can have a release 1.82 for this ready for testing in the RC environment by the end of September. As mentioned before, because the impact of this change can be bigger than usual, we plan to have this release in RC for testing for 6 weeks before deploying it to production. 3= Descr We don't foresee major technical issues with the options being discussed currently, but can only present a definite implementation plan when consensus is reached on this. If consensus is reached in time, it may be an option to postpone the release of 1.82 above and have a bigger release that also includes this work - to avoid delays because of the long RC period required for that release. 4= Personalized Authorization We are very happy to have a mandate from the chairs to work on a proof of concept for personalised authentication. We look forward to implementing this as soon as we have the opportunity and working closely with the working group using the RC environment. In addition to core whois changes we will also deploy improvements to Webupdates in the RC environment when we work on this - to exemplify how we can make use of this to improve usability. Kind regards Tim Bruijnzeels Assistant Manager Software Engineering RIPE NCC

1 0

Proposal about personalised authorisation
by Tim Bruijnzeels 12 Aug '15

12 Aug '15

The RIPE NCC has discussed the concept of personalised authorisation on various occasions, most recently at the DB WG session at RIPE 69. Following discussions and input from the working group we would now like to propose the following additions to the RIPE Database: = Extend the person object template with "auth:" as an optional, multiple attribute, with all current authentication methods. = Extend the mntner object "auth:" attribute with a new method that allows a reference to a person object that has at least one "auth:" attribute. There is no requirement to use these extensions, but users that choose to do so can benefit from the following advantages: = Currently it is is not always clear who uses the credentials in each "auth:" attribute on a mntner. There are two main problems with this. First of all this can result in authorisation being given to the wrong users, and second this makes it difficult to keep these objects up to date. For example it is not easy to remove the password for a specific user from a mntner without knowing, or deducing, which password hash is theirs. Having explicit references to person objects will make this clear. = Users who are authorised for multiple mntners will be able to maintain their own credentials in one place. = Audit logging can be improved. It will become possible to show to authorised users of the database which person made which change to their object. = The security of regaining lost access to a maintainer can be improved by letting users of the database reset the credentials on their own person object, rather than on a maintainer. Especially when the person object is linked with a Single Sign-On account. Allowing "auth:" attributes on person objects also allows us to make it easier for users to manage their person object in the RIPE Database in combination with their Single Sign-On (SSO) account on RIPE NCC Access as a single identity. To facilitate this we propose the following: = Being they are part of the same identity, person objects can only be associated with one SSO account, and vice versa. = Associated person objects and SSO accounts are automatically kept in sync. = SSO accounts can associated with new person objects, or existing person objects, provided that the user has access to both the SSO account and at least one maintainer of the person object. = Users will be able to decouple their own SSO account and person object.

4 4

Help needed to delete an object via email
by Hank Nussbacher 07 Aug '15

07 Aug '15

I believe I have encountered a bug with email processing of updates via auto-dbm@ripe and have not been able to engage the people there via ticket NCC#2015080034. I reported it and the ticket was quickly closed without being solved. I therefore ask for help from the hive-mind. The recently added "remarks:" line is causing parsing problems. The line looks like this: remarks: For information on "status:" attribute read https://www.ripe.net/data-tools/db/faq/faq-status-values-legacy-resources First I tried editing an object but the parser kept chopping the line as follows: >remarks: For information on "status:" attribute read >https: >//www.ripe.net/data-tools/db/faq/faq-status-values-legacy-resources >***Error: "https" is not a known RPSL attribute So I just deleted the line and was able to update the object. But now I want to delete an intenum object with the "remarks" line. When I do that I get: >remarks: For information on "status:" attribute read >https: >//www.ripe.net/data-tools/db/faq/faq-status-values-legacy-resources >... >created: 1970-01-01T00:00:00Z >last-modified: 2014-05-27T13:18:49Z >source: RIPE > >***Error: Object [inetnum] 192.114.48.0 - 192.114.63.255 doesn't match >version > in database The parser again chops the line at https and auto-starts a new line and therefore flags the entry as not macthing the version in the DB. Is there a workaround for this? Some escape character that will cause it not to break the line? Thanks, Hank

5 11

RIPE DB syntax check for mp-import
by Tomas Hlavacek 06 Aug '15

06 Aug '15

Hello! I just noticed that aut-num object for AS2852 in RIPE DB contains line mp-import: afi ipv6.unicastfrom AS39790 with 'afi ipv6.unicast' and 'from' keywords merged, which causs some confusion of my parsing scripts. Maybe I am missing something but I think this is not syntactically valid, is it? And if it is not, how could that get into RIPE DB? Tomas

3 2

ad-hoc RIPE db-wg meeting at IETF93
by Job Snijders 27 Jul '15

27 Jul '15

Dear best-db-working-group-in-the-world! I was looking through the IETF93 attendees list and noticed quite a few familiar names are attending the IETF meeting in Prague this week! :-) I'd love to catch up with you and discuss some of the current db-wg topics face to face, possibly while holding a sparkling beverage. Let's gather Tuesday July 21st somewhere in the Hilton from 18:00 to 19:00. I'll arrange a room or spot and follow-up with details. Let me know if you are interested in attending this ad-hoc db-wg meeting. No worries if you aren't around this time, we'll see each other at RIPE71 in Budapest. Kind regards, Job

6 13

Release 1.80.1 deployed: warning users on failed update/delete, if maintainers had passwords removed
by Tim Bruijnzeels 16 Jul '15

16 Jul '15

Dear working group, We have just deployed whois release 1.80.1 to production. We have seen some cases where users were not aware that passwords had been removed from their maintainer, and they would only get an "authorisation failed" error. To help these users debug this issue better we now also include the following warning in case one or more of the maintainers had any passwords removed: "MD5 passwords older than November 2011 were removed for one or more maintainers of this object, see: https://www.ripe.net/removed2011pw" Example case (used by us for testing): > The following object(s) were found to have ERRORS: > > --- > Create FAILED: [person] AP1-TEST Another Person > > person: Another Person > address: Some Address > phone: +12 345 678 90 > nic-hdl: AP1-TEST > mnt-by: OWNER-MNT > source: TEST > > ***Error: Authorisation for [person] AP1-TEST failed > using "mnt-by:" > not authenticated by: OWNER-MNT > > ***Warning: MD5 passwords older than November 2011 were removed for one or more > maintainers of this object, see: https://www.ripe.net/removed2011pw Kind regards, Tim Bruijnzeels Assistant Manager Software Engineering RIPE NCC Database Team

1 0