Re: [db-wg] [opensource-wg] Question for RPSL Parser
Hi Thomas and others, thanks for your quick replies. To make things more clear, our project at NLnet Labs is the development of "a modern IRRtoolset" written in Python and targeting any operator, not just another homemade tool for our own needs. Or to say in more detail the creation of a tool that is able to configure BGP routers directly+automatically by extracting the policies from RIPE DB. This means that I am struggling to avoid the (re)use of any past or legacy software that exists, otherwise we loose independency and inherit restrictions from the past. Currently, I do have a working proof of concept with a simple RPSL parser written in Python which is able to parse simple policies with (v6)import(s)-(v6)export(s)-accept-announce-action-pref-med-coomunity attributes. Convenient I would say for many existing policies like KPN’s, RIPE’s etc. but definitely it will crash in complex policy files like GRNET’s. My current solution uses also BGPq3 to translate AS, AS-SET and ROUTE-SET and this is also something that I want to ged rid of. So I was looking for the existence of a Python Library which can do a complete job and translate any existing policy in an intermediate data structure like XML (this is what I do now). I was hoping to find one, adopt it and present our (completed) work in the next RIPE meeting (yes I know I am very ambitious, I need to slow down). * Masimo, I already checked rpsltool and discarded it. But thanks for your super-fast reply. * Tomas, thanks for pointing me to your library. I will have a look at it and check if it is convenient for our project. * George, yes you are right. This should be the #1 step, take corresponding RPSL RFCs and implement them in Python. However, that would take too much time and will not allow us to work on other missing functionalities of the prototype. And we would like to have something ready at the beginning of November. However, I will check your solution as well. I hope now that I made myself more clear. Big thanks to everyone, every help, info or idea is kindly appreciated. Kind Regards Stavros ---- Stavros Konstantaras Internet Research Engineer, NLnet Labs Science Park 400, 1098 XH, Amsterdam Fingerprint: 4502 7D16 2DF8 ADB0 4AA6 21A9 BF9E EFFF 2744 FE5D
On 19 Aug 2015, at 14:39, Tomas Hlavacek <tomas.hlavacek@nic.cz> wrote:
Hi Stavros!
On 08/19/2015 02:15 PM, Ondrej Filip wrote:
Vis o necem?
-------- Forwarded Message -------- Subject: [opensource-wg] Question for RPSL Parser Date: Wed, 19 Aug 2015 14:13:58 +0200 From: Stavros Konstantaras <stavros@nlnetlabs.nl> To: db-wg@ripe.net CC: opensource-wg@ripe.net
Hi all WG members,
For my project at NLnet Labs I would like to ask you if you know any (relatively) complete RPSL parser written in *Python*. I searched around and I didnât find anything. Does someone know any repository where I can get one or maybe contact someone who has written one?
It depends on what you need... I wrote one - https://github.com/tmshlvck/bgpcrunch , but the aim was rather data analysis than "authoritative" RPSL translation. When it comes to filter interpretation my approach is quite "heuristic" and it is focused on speed and ability to take short-cuts which wouldn't be possible with full parsing the RPSL objects according to EBNF metasyntax.
Have you considered creating simple Python interface for peval from IRRToolSet? It might do the job if you don't need any special features.
Cheers, Tomas
Hi Stavros! On 2015-08-19 16:00, Stavros Konstantaras wrote: [...]
* George, yes you are right. This should be the #1 step, take corresponding RPSL RFCs and implement them in Python.
A word of caution here... While this suggestion is probably a sound one, formally, everyone needs to be aware that the set of RFCs out there is ancient. Not all of the provisions in the documents have been implemented (in some or all IRRs) and, ototh, some IRRs did implement some (minor?) extensions which never made it back into updated documents. But I guess you are aware of that. FWIW, Wilfried
However, that would take too much time and will not allow us to work on other missing functionalities of the prototype. And we would like to have something ready at the beginning of November. However, I will check your solution as well.
I hope now that I made myself more clear. Big thanks to everyone, every help, info or idea is kindly appreciated.
Kind Regards Stavros
---- Stavros Konstantaras Internet Research Engineer, NLnet Labs Science Park 400, 1098 XH, Amsterdam
Fingerprint: 4502 7D16 2DF8 ADB0 4AA6 21A9 BF9E EFFF 2744 FE5D
On 19 Aug 2015, at 14:39, Tomas Hlavacek <tomas.hlavacek@nic.cz> wrote:
Hi Stavros!
On 08/19/2015 02:15 PM, Ondrej Filip wrote:
Vis o necem?
-------- Forwarded Message -------- Subject: [opensource-wg] Question for RPSL Parser Date: Wed, 19 Aug 2015 14:13:58 +0200 From: Stavros Konstantaras <stavros@nlnetlabs.nl> To: db-wg@ripe.net CC: opensource-wg@ripe.net
Hi all WG members,
For my project at NLnet Labs I would like to ask you if you know any (relatively) complete RPSL parser written in *Python*. I searched around and I didnât find anything. Does someone know any repository where I can get one or maybe contact someone who has written one?
It depends on what you need... I wrote one - https://github.com/tmshlvck/bgpcrunch , but the aim was rather data analysis than "authoritative" RPSL translation. When it comes to filter interpretation my approach is quite "heuristic" and it is focused on speed and ability to take short-cuts which wouldn't be possible with full parsing the RPSL objects according to EBNF metasyntax.
Have you considered creating simple Python interface for peval from IRRToolSet? It might do the job if you don't need any special features.
Cheers, Tomas
Hi Wilfried. Yes I know that related RFCs are quite old but I wasn’t aware of this issue that you mention. Thanks for pointing it out. The oldness of RPSL is something that is already known to me and next year we will focus on that part. Kind Regards Stavros
On 19 Aug 2015, at 16:27, Wilfried Woeber <woeber@cc.univie.ac.at> wrote:
Hi Stavros!
On 2015-08-19 16:00, Stavros Konstantaras wrote: [...]
* George, yes you are right. This should be the #1 step, take corresponding RPSL RFCs and implement them in Python.
A word of caution here...
While this suggestion is probably a sound one, formally, everyone needs to be aware that the set of RFCs out there is ancient. Not all of the provisions in the documents have been implemented (in some or all IRRs) and, ototh, some IRRs did implement some (minor?) extensions which never made it back into updated documents.
But I guess you are aware of that.
FWIW, Wilfried
However, that would take too much time and will not allow us to work on other missing functionalities of the prototype. And we would like to have something ready at the beginning of November. However, I will check your solution as well.
I hope now that I made myself more clear. Big thanks to everyone, every help, info or idea is kindly appreciated.
Kind Regards Stavros
---- Stavros Konstantaras Internet Research Engineer, NLnet Labs Science Park 400, 1098 XH, Amsterdam
Fingerprint: 4502 7D16 2DF8 ADB0 4AA6 21A9 BF9E EFFF 2744 FE5D
On 19 Aug 2015, at 14:39, Tomas Hlavacek <tomas.hlavacek@nic.cz> wrote:
Hi Stavros!
On 08/19/2015 02:15 PM, Ondrej Filip wrote:
Vis o necem?
-------- Forwarded Message -------- Subject: [opensource-wg] Question for RPSL Parser Date: Wed, 19 Aug 2015 14:13:58 +0200 From: Stavros Konstantaras <stavros@nlnetlabs.nl> To: db-wg@ripe.net CC: opensource-wg@ripe.net
Hi all WG members,
For my project at NLnet Labs I would like to ask you if you know any (relatively) complete RPSL parser written in *Python*. I searched around and I didnât find anything. Does someone know any repository where I can get one or maybe contact someone who has written one?
It depends on what you need... I wrote one - https://github.com/tmshlvck/bgpcrunch , but the aim was rather data analysis than "authoritative" RPSL translation. When it comes to filter interpretation my approach is quite "heuristic" and it is focused on speed and ability to take short-cuts which wouldn't be possible with full parsing the RPSL objects according to EBNF metasyntax.
Have you considered creating simple Python interface for peval from IRRToolSet? It might do the job if you don't need any special features.
Cheers, Tomas
Hi Stavros! On 08/19/2015 04:00 PM, Stavros Konstantaras wrote:
Hi Thomas and others, thanks for your quick replies.
To make things more clear, our project at NLnet Labs is the development of "a modern IRRtoolset" written in Python and targeting any operator, not just another homemade tool for our own needs. Or to say in more detail the creation of a tool that is able to configure BGP routers directly+automatically by extracting the policies from RIPE DB. This means that I am struggling to avoid the (re)use of any past or legacy software that exists, otherwise we loose independency and inherit restrictions from the past.
I think that there is a legitimate question whether it make sense to re-create IRRToolSet in Python? (I tried something alike in Perl few years ago - http://sourceforge.net/projects/bgflib/ and the project is abandoned now... For multiple reasons. ) Remember Nick's presentation at RIPE61 about RPSL meaningfulness/meaninglessness? - http://ripe61.ripe.net/presentations/231-228-inex-ripe-rome-routingwg-whithe... This summer I tried to conduct an analysis of data in RIPE DB and match the routing policies to actual BGP routes in DFZ. The aim was to check per-hop routing policies in ASes that are/should be covered by aut-num objects in RIPE DB. I am going to publish preliminary results in a few days. But I think that from what I have seen so far it is obvious that the divergence among RPSL routing policies and BGP paths in DFZ is a serious issue even here in Europe. And we are supposed to have our routing policy records in better shape than the rest of the world, right? :-)
Currently, I do have a working proof of concept with a simple RPSL parser written in Python which is able to parse simple policies with (v6)import(s)-(v6)export(s)-accept-announce-action-pref-med-coomunity attributes. Convenient I would say for many existing policies like KPN’s, RIPE’s etc. but definitely it will crash in complex policy files like GRNET’s. My current solution uses also BGPq3 to translate AS, AS-SET and ROUTE-SET and this is also something that I want to ged rid of. So I was looking for the existence of a Python Library which can do a complete job and translate any existing policy in an intermediate data structure like XML (this is what I do now). I was hoping to find one, adopt it and present our (completed) work in the next RIPE meeting (yes I know I am very ambitious, I need to slow down).
* Masimo, I already checked rpsltool and discarded it. But thanks for your super-fast reply.
* Tomas, thanks for pointing me to your library. I will have a look at it and check if it is convenient for our project.
I don't think it is. The https://github.com/tmshlvck/bgpcrunch is just an analysis tool, not a serious RPSL parser. But I am willing to collaborate on a new library if you really mean it. But still I think that we should try to open the discussion about RPSL future that actually didn't happen after Nick's presentation (or I am not aware of it). Cheers, Tomas
Stavros, On Wed, 19 Aug 2015 16:00:59 +0200 Stavros Konstantaras <stavros@nlnetlabs.nl> wrote:
To make things more clear, our project at NLnet Labs is the development of "a modern IRRtoolset" written in Python and targeting any operator, not just another homemade tool for our own needs. Or to say in more detail the creation of a tool that is able to configure BGP routers directly+automatically by extracting the policies from RIPE DB. This means that I am struggling to avoid the (re)use of any past or legacy software that exists, otherwise we loose independency and inherit restrictions from the past.
I don't know of any Python RPSL parser you can use. Your goals make a certain amount of sense. Unfortunately RPSL is an ugly language... it's actually more like several languages in one, all bad. :P * Start with simple attribute/value, one per line * Oh, but then "RPSL-ize", which allows line continuation and end-of-line comments * Oh, actually we want to make a separate grammar for each attribute * But be sure to make all of this "extensible", to insure maximum confusion! My approach in the past has been to start with a simple generic parser that split text into objects, objects into attributes, extracts attribute name & values (handling line continuation and end-of-line comments). You can do all of this with Python regular expressions, something like r"\n(?![ \t+])" to split objects into attributes; cleaning end-of-line comments & line continuation characters is straightforward. Then you can deal with the grammar for the attributes you know, and ignore the rest. You can extend the parsing to be more and more comprehensive until you are able to parse the data set you worry about. I'm not sure this is worth doing... but if there was a good, easy-to-use Python library then maybe interesting things would result? Cheers, -- Shane
participants (4)
-
Shane Kerr -
Stavros Konstantaras -
Tomas Hlavacek -
Wilfried Woeber