some thoughts on add-path and bmp at ripe/ris
i have some small thoughts on add-path and BMP in RIPE/RIS. this may hinder more than help, unfortunately. as a researcher, of course i would love a wider view into the AS graph and the views of as many ASs as possible. nom nom. but ... add-path, if not done very carefully, and documented in detail for each peer, will produce views i will not understand. it is analogous to not knowing what view a BGP peer gives RIS today. did it give me a full Nth view into peer X? is the path that of a peer, a customer, a provider? and add-path is worse because it prunes in configuration and vendor- dependent and otherwise unpredictable ways. and, as an operator if P is a RIS peer, and they give RIS BMP showing their input from their peers, upstreams, and customers, those peers, upstreams, and customers have not given thir consent to have their internals published! if they want to have RIS publish their internals, they would peer with RIS and choose what view they present. and, of course, the researcher does not know what view, peer, upstream, or customer P's BMP peer is giving them. same problem as above under add-path. randy
Dear Randy, I'm glad you brought ADDPATH topic up! Up to 2019 I was running the now-defunct-Isolario project with my colleague Luca at IIT-CNR, and we did set up an ADDPATH-capable route collector. First, my 2 cents on your operator consideration on the consent from original data sources. I'm far from being a legal, but I don't think ADDPATH differs much from collecting regular BGP sessions. After all, at the moment a BGP peer connected with RIS shares pieces of tables (sometimes close to a full table)... if it was sharing data with ADDPATH it would share all the pieces of the tables. I believe that the very same consideration/concern could apply on both case. Here a few random considerations on our Isolario experience with ADDPATH. Sorry in advance for the long post! 1. Overall, we had about 25-30 peers in Isolario sharing data in ADDPATH. One major peer was sharing ~80 different full routing tables coming from different ASes and was particularly interesting for us. Most of the peers using ADDPATH were using Bird to share their data, and the configuration was trivial 2. The size of data collected - RIB snapshots in particular - was very large wrt regular collectors. I recall having RIB snapshots compressed in bz2 of ~1GB and 20MB in average of UPDATE files. It gets of uttermost importance to have high-perfomance MRT data readers to parse these data. Sometimes it is simply not possible to keep the number of peers low on each collector (see the 80 full routes from one single peer case). This is why we deployed - and presented at one of the RIPE meetings - bgpscanner with Lorenzo Cogotti back in time. As far as I know, Lorenzo improved bgpscanner furthermore developing Micro BGP suite (https://labs.ripe.net/author/lorenzo_cogotti/micro-bgp-suite-the-swiss-army-...). 3. Data was often redundant, but never fully redundant. That was a reasonable price to pay to have more data. 4. ADDPATH hides to the collector the possibility to see the full routing table of the direct peer. You have all the tables that contribute in building the table, but you don't have the result of the BGP decision process of the peer. Depending on the kind of application and monitoring done, this could be a problem. In Isolario, we asked to set up a regular BGP session + an ADDPATH session from each peer that shared data with ADDPATH. Of course, this is not an ideal solution and there was nothing that could enforce the binding between the two sessions 5. Back in 2018, there were two separate way of interpreting ADDPATH - both compliant to RFCs. The first - implemented by Bird - consisted in keeping a single path ID for each table shared. Hence it was trivial to map the table to one single source and re-create the table from a reader perspective (e.g. 1.0.0.0/24 1 from table 1, 1.0.0.0/24 2 from table 2, 1.0.1.0/24 1 from table 1). The second - implemented by FRR - consisted in using many different path IDs for each table - still keeping the possibility to distinguish whether the same network was belonging to a different table (e.g. 1.0.0.0/24 1 from table 1, 1.0.0.0/24 2 from table 2, 1.0.1.0/24 3 from table 1). See https://github.com/FRRouting/frr/issues/1743. Of course, this latter approach was creating big headaches to us in matching tables, path IDs and "original peer"... hence in Isolario allowed only ADDPATH from Bird-like data sources. 6. If I recall well - and I may be wrong here - the only way to recognize that the BGP session between the BGP peer and one its peers/providers/customers went down was to search for a storm of withdrawn messages I hope this may be helpful to anyone working around this topic! Best Regards, Alessandro Improta Engineering manager p. +393488077654 e. aimprota@catchpoint.com<mailto:aimprota@catchpoint.com> a. Via Oberdan 53, Pietrasanta (LU) [cid:dc32a76e-0073-4695-9722-21fe3edb8e0a] Learn more about Catchpoint → Watch this 2-minute video!<https://www.catchpoint.com/explainer> [linkedin]<https://www.linkedin.com/company/catchpoint-systems-inc> [twitter] <https://twitter.com/Catchpoint> [facebook] <https://www.facebook.com/catchpoint/> [youtube] <https://www.youtube.com/c/Catchpoint/> ________________________________ From: mat-wg <mat-wg-bounces@ripe.net> on behalf of Randy Bush <randy@psg.com> Sent: Monday, January 23, 2023 1:23 PM To: Measurement Analysis and Tools Working Group <mat-wg@ripe.net> Subject: [mat-wg] some thoughts on add-path and bmp at ripe/ris i have some small thoughts on add-path and BMP in RIPE/RIS. this may hinder more than help, unfortunately. as a researcher, of course i would love a wider view into the AS graph and the views of as many ASs as possible. nom nom. but ... add-path, if not done very carefully, and documented in detail for each peer, will produce views i will not understand. it is analogous to not knowing what view a BGP peer gives RIS today. did it give me a full Nth view into peer X? is the path that of a peer, a customer, a provider? and add-path is worse because it prunes in configuration and vendor- dependent and otherwise unpredictable ways. and, as an operator if P is a RIS peer, and they give RIS BMP showing their input from their peers, upstreams, and customers, those peers, upstreams, and customers have not given thir consent to have their internals published! if they want to have RIS publish their internals, they would peer with RIS and choose what view they present. and, of course, the researcher does not know what view, peer, upstream, or customer P's BMP peer is giving them. same problem as above under add-path. randy -- To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ripe...
alessandro, real data and experience deeply appreciated. a few questions. o once a collector commits to ADD-PATH, it's a promise to keep the data forever; well, it's been 26 years of RV and a few less for RIS. - Did operators find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did researchers find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did folk develop special tools to take advantage of ADD-PATH data? o On the problem of stream/peer identification. You describe the divergence of BIRD and FRR. What about commercial router vendors? o You say data were often redundant, though not fully. Did you investigate mechanisms to reduce the storage, or did you see that as a path to complexity and fragility merely to save some spinning rust? o You mention the withdraw storm between your peer and their peer (and later an announcment storm, I presume). For peers with large out-degree, this could be likely. Were these data interesting in any way, or just more storage? again, thanks so much for real experience. gives me a bit more clue. randy
Let's see...
o once a collector commits to ADD-PATH, it's a promise to keep the data forever; well, it's been 26 years of RV and a few less for RIS. - Did operators find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did researchers find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did folk develop special tools to take advantage of ADD-PATH data?
This is hard to answer... I know for sure that a couple of research groups used Isolario data for their studies, in particular researchers at Max Planck Institute where studying ADD-PATH data we collected, but I'm not sure on the final outcome of their research. Also, data was used by Hurricane Electric for their bgp.he.net tool. Besides that, I am not aware of operators that used the data, but I never focused my attention on access logs back in the days.
o On the problem of stream/peer identification. You describe the divergence of BIRD and FRR. What about commercial router vendors?
This may be totally wrong, but I remember that Cisco was not allowing ADDPATH for eBGP. Not sure about other vendors though.
o You say data were often redundant, though not fully. Did you investigate mechanisms to reduce the storage, or did you see that as a path to complexity and fragility merely to save some spinning rust?
We did investigate that. Memory-wise, we studied methodologies to exploit compressing technique such as LZW (see Interactive Collector Engine (ripe.net)<https://ripe75.ripe.net/presentations/21-pres.pdf>). However, the routing engine we devised for route collecting + the amount of RAM we had in the collectors were enough and never required us to turn that feature on. Disk-wise, we found out that xz was a great candidate to replace bz2 and gz as the next compressing methodology for route collecting. I don't remember the results to be honest, but that should be easy to reproduce. However, we decided to not compress data in xz because only bgpscanner was able to read xz data - hence anyone that wanted to use other tools such as bgpdump would not have been able to read them.
o You mention the withdraw storm between your peer and their peer (and later an announcment storm, I presume). For peers with large out-degree, this could be likely. Were these data interesting in any way, or just more storage?
Yes, you presume correct. I honestly didn't find that really useful... A simple "Peer down" message would have been sufficient in those cases to replace the withdrawn storm, while the announcement storm caused by the RIB transfer of peers was inevitable.
again, thanks so much for real experience. gives me a bit more clue.
You're most welcome! Best Regards, Alessandro Improta Engineering manager p. +393488077654 e. aimprota@catchpoint.com<mailto:aimprota@catchpoint.com> a. Via Oberdan 53, Pietrasanta (LU) [cid:64b79133-76ac-486c-8aea-c2984ec6f4ab] Learn more about Catchpoint → Watch this 2-minute video!<https://www.catchpoint.com/explainer> [linkedin]<https://www.linkedin.com/company/catchpoint-systems-inc> [twitter] <https://twitter.com/Catchpoint> [facebook] <https://www.facebook.com/catchpoint/> [youtube] <https://www.youtube.com/c/Catchpoint/> ________________________________ From: Randy Bush <randy@psg.com> Sent: Friday, March 17, 2023 5:15 PM To: Alessandro Improta <aimprota@catchpoint.com> Cc: Measurement Analysis and Tools Working Group <mat-wg@ripe.net> Subject: Re: [mat-wg] some thoughts on add-path and bmp at ripe/ris alessandro, real data and experience deeply appreciated. a few questions. o once a collector commits to ADD-PATH, it's a promise to keep the data forever; well, it's been 26 years of RV and a few less for RIS. - Did operators find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did researchers find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did folk develop special tools to take advantage of ADD-PATH data? o On the problem of stream/peer identification. You describe the divergence of BIRD and FRR. What about commercial router vendors? o You say data were often redundant, though not fully. Did you investigate mechanisms to reduce the storage, or did you see that as a path to complexity and fragility merely to save some spinning rust? o You mention the withdraw storm between your peer and their peer (and later an announcment storm, I presume). For peers with large out-degree, this could be likely. Were these data interesting in any way, or just more storage? again, thanks so much for real experience. gives me a bit more clue. randy
Hi all, hi Randy and Alessandro, and thank you for sharing your experience with BGP data collection and ADD-PATH/BMP. I wanted to reopen this thread after a couple of years because we’ve just added BMP support to bgproutes.io <http://bgproutes.io/>. Everything is automated: operators can connect their routers to us via BMP, and the data will be available to the public—either in an MRT archive, through an API, or via a live WebSocket stream. We’ve written a blog post with more details if you’re interested: https://bgproutes.io/blog-bmp. Below is our experience with BMP and our feedback with respect to the points raised by Randy and Alessandro in the previous emails: BMP/ADD-PATH must be used carefully. Users need to know from which peer the data comes, and—if using BMP—at which stage in the router the data is exported (Adj-RIB-In pre/post, Loc-RIB, etc.). The good news is that with BMP we can obtain this information cleanly (it’s more difficult with ADD-PATH). In bgproutes.io <http://bgproutes.io/>, we updated our API to give users fine-grained control over the data they collect: for a BMP session, they can filter on specific peer IPs or the BMP stage (e.g., Adj-RIB-In pre, Loc-RIB, etc.). More generally, this is why we chose BMP over ADD-PATH: BMP provides precise, fine-grained context about what is being exported, whereas, afaik, with ADD-PATH you may see additional/backup paths, but it’s harder to map those paths back to the peer that actually sent them. Another advantage of BMP is access to the Loc-RIB—the output of the BGP decision process—which, afaik, ADD-PATH cannot provide precisely. This can be very helpful; however, operators should be aware that if they export both Adj-RIB-In and Loc-RIB, users may be able to infer routing policies. That may or may not be desirable. On data privacy: I’m not a legal expert, but I agree with Alessandro that what BMP/ADD-PATH export compared to a standard BGP session is essentially a superset; any updates in that superset can end up being the subset actually announced on the BGP session. From that perspective, it seems acceptable. That said, BMP can introduce a practical issue: some routers do not allow per-peer export selection (it’s “all peers or none”). That’s problematic if an operator only wants to share updates from a subset of peers. In bgproutes.io <http://bgproutes.io/>, when operators fill out the onboarding form, they can tell us which peers we should not collect. The data will still reach us, but we exclude and discard those peers immediately. Interoperability challenges: most routers support BMP, but not all—and even among commercial vendors, each tends to do BMP with its own flavor. We couldn’t just implement the RFC literally; we added checks to ensure compatibility across devices. We also observe that many routers cannot export every stage (Adj-RIB-In pre/post, Loc-RIB, Adj-RIB-Out pre/post). Fortunately, it’s usually Adj-RIB-Out pre/post that’s weakly supported—which is less useful for a data-collection platform like bgproutes.io <http://bgproutes.io/> (or RIS/RouteViews). What matters most—Adj-RIB-In pre/post and Loc-RIB—is generally well supported. Detecting feed changes: another challenge with BMP is knowing when an operator changes the exported stages after the session is established. For example, if an operator starts with Adj-RIB-In pre and later switches to Adj-RIB-In post, we’ll correctly receive the new stage, but we won’t automatically know that Adj-RIB-In pre was deactivated. We don’t have a solid solution for this yet; we’ll explore options as we gain more operational experience. Since we’ve just launched this feature, it’s too early to say how people will use the data and how many operators will connect via BMP. We’ll be happy to share a recap once we have more data points. To conclude: if some of you are interested, we’re going to present this at the NLNOG Day (Sept. 30). For now, we’re connected with Randy and Alarig Le Lay (thanks to both!), but if you like this approach and would like to connect with us via BMP, you’re very welcome—all data is publicly available and onboarding is super simple. Cheers, Thomas
On 17 Mar 2023, at 17:51, Alessandro Improta via mat-wg <mat-wg@ripe.net> wrote:
Let's see...
o once a collector commits to ADD-PATH, it's a promise to keep the data forever; well, it's been 26 years of RV and a few less for RIS. - Did operators find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did researchers find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did folk develop special tools to take advantage of ADD-PATH data?
This is hard to answer... I know for sure that a couple of research groups used Isolario data for their studies, in particular researchers at Max Planck Institute where studying ADD-PATH data we collected, but I'm not sure on the final outcome of their research. Also, data was used by Hurricane Electric for their bgp.he.net <http://bgp.he.net/> tool. Besides that, I am not aware of operators that used the data, but I never focused my attention on access logs back in the days.
o On the problem of stream/peer identification. You describe the divergence of BIRD and FRR. What about commercial router vendors?
This may be totally wrong, but I remember that Cisco was not allowing ADDPATH for eBGP. Not sure about other vendors though.
o You say data were often redundant, though not fully. Did you investigate mechanisms to reduce the storage, or did you see that as a path to complexity and fragility merely to save some spinning rust?
We did investigate that. Memory-wise, we studied methodologies to exploit compressing technique such as LZW (see Interactive Collector Engine (ripe.net) <https://ripe75.ripe.net/presentations/21-pres.pdf>). However, the routing engine we devised for route collecting + the amount of RAM we had in the collectors were enough and never required us to turn that feature on. Disk-wise, we found out that xz was a great candidate to replace bz2 and gz as the next compressing methodology for route collecting. I don't remember the results to be honest, but that should be easy to reproduce. However, we decided to not compress data in xz because only bgpscanner was able to read xz data - hence anyone that wanted to use other tools such as bgpdump would not have been able to read them.
o You mention the withdraw storm between your peer and their peer (and later an announcment storm, I presume). For peers with large out-degree, this could be likely. Were these data interesting in any way, or just more storage?
Yes, you presume correct. I honestly didn't find that really useful... A simple "Peer down" message would have been sufficient in those cases to replace the withdrawn storm, while the announcement storm caused by the RIB transfer of peers was inevitable.
again, thanks so much for real experience. gives me a bit more clue.
You're most welcome!
Best Regards,
Alessandro Improta Engineering manager p. +393488077654 e. aimprota@catchpoint.com <mailto:aimprota@catchpoint.com> a. Via Oberdan 53, Pietrasanta (LU) <Outlook-ajnjxisv.png> Learn more about Catchpoint → Watch this 2-minute video! <https://www.catchpoint.com/explainer> <https://www.linkedin.com/company/catchpoint-systems-inc> <https://twitter.com/Catchpoint> <https://www.facebook.com/catchpoint/> <https://www.youtube.com/c/Catchpoint/>
From: Randy Bush <randy@psg.com> Sent: Friday, March 17, 2023 5:15 PM To: Alessandro Improta <aimprota@catchpoint.com> Cc: Measurement Analysis and Tools Working Group <mat-wg@ripe.net> Subject: Re: [mat-wg] some thoughts on add-path and bmp at ripe/ris
alessandro,
real data and experience deeply appreciated. a few questions.
o once a collector commits to ADD-PATH, it's a promise to keep the data forever; well, it's been 26 years of RV and a few less for RIS.
- Did operators find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often?
- Did researchers find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often?
- Did folk develop special tools to take advantage of ADD-PATH data?
o On the problem of stream/peer identification. You describe the divergence of BIRD and FRR. What about commercial router vendors?
o You say data were often redundant, though not fully. Did you investigate mechanisms to reduce the storage, or did you see that as a path to complexity and fragility merely to save some spinning rust?
o You mention the withdraw storm between your peer and their peer (and later an announcment storm, I presume). For peers with large out-degree, this could be likely. Were these data interesting in any way, or just more storage?
again, thanks so much for real experience. gives me a bit more clue.
randy --
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/mat-wg
Hi Thomas, On Thu 25 Sep 2025, 00:11, Thomas Holterbach wrote:
Detecting feed changes: another challenge with BMP is knowing when an operator changes the exported stages after the session is established. For example, if an operator starts with Adj-RIB-In pre and later switches to Adj-RIB-In post, we’ll correctly receive the new stage, but we won’t automatically know that Adj-RIB-In pre was deactivated. We don’t have a solid solution for this yet; we’ll explore options as we gain more operational experience.
Perhaps you are already aware, but this is one of the use cases for the newly proposed Generic Event Notification, https://datatracker.ietf.org/doc/draft-sp-grow-bmp-gen/ Specifically, Sec 4.4.1 in version -01: https://datatracker.ietf.org/doc/html/draft-sp-grow-bmp-gen-01#name-example-... Cheers, luuk
On 17 Mar 2023, at 17:51, Alessandro Improta via mat-wg <mat-wg@ripe.net> wrote:
Let's see...
o once a collector commits to ADD-PATH, it's a promise to keep the data forever; well, it's been 26 years of RV and a few less for RIS. - Did operators find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did researchers find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often? - Did folk develop special tools to take advantage of ADD-PATH data?
This is hard to answer... I know for sure that a couple of research groups used Isolario data for their studies, in particular researchers at Max Planck Institute where studying ADD-PATH data we collected, but I'm not sure on the final outcome of their research. Also, data was used by Hurricane Electric for their bgp.he.net <http://bgp.he.net/> tool. Besides that, I am not aware of operators that used the data, but I never focused my attention on access logs back in the days.
o On the problem of stream/peer identification. You describe the divergence of BIRD and FRR. What about commercial router vendors?
This may be totally wrong, but I remember that Cisco was not allowing ADDPATH for eBGP. Not sure about other vendors though.
o You say data were often redundant, though not fully. Did you investigate mechanisms to reduce the storage, or did you see that as a path to complexity and fragility merely to save some spinning rust?
We did investigate that. Memory-wise, we studied methodologies to exploit compressing technique such as LZW (see Interactive Collector Engine (ripe.net) <https://ripe75.ripe.net/presentations/21-pres.pdf>). However, the routing engine we devised for route collecting + the amount of RAM we had in the collectors were enough and never required us to turn that feature on. Disk-wise, we found out that xz was a great candidate to replace bz2 and gz as the next compressing methodology for route collecting. I don't remember the results to be honest, but that should be easy to reproduce. However, we decided to not compress data in xz because only bgpscanner was able to read xz data - hence anyone that wanted to use other tools such as bgpdump would not have been able to read them.
o You mention the withdraw storm between your peer and their peer (and later an announcment storm, I presume). For peers with large out-degree, this could be likely. Were these data interesting in any way, or just more storage?
Yes, you presume correct. I honestly didn't find that really useful... A simple "Peer down" message would have been sufficient in those cases to replace the withdrawn storm, while the announcement storm caused by the RIB transfer of peers was inevitable.
again, thanks so much for real experience. gives me a bit more clue.
You're most welcome!
Best Regards,
Alessandro Improta Engineering manager p. +393488077654 e. aimprota@catchpoint.com <mailto:aimprota@catchpoint.com> a. Via Oberdan 53, Pietrasanta (LU) <Outlook-ajnjxisv.png> Learn more about Catchpoint → Watch this 2-minute video! <https://www.catchpoint.com/explainer> <https://www.linkedin.com/company/catchpoint-systems-inc> <https://twitter.com/Catchpoint> <https://www.facebook.com/catchpoint/> <https://www.youtube.com/c/Catchpoint/>
From: Randy Bush <randy@psg.com> Sent: Friday, March 17, 2023 5:15 PM To: Alessandro Improta <aimprota@catchpoint.com> Cc: Measurement Analysis and Tools Working Group <mat-wg@ripe.net> Subject: Re: [mat-wg] some thoughts on add-path and bmp at ripe/ris
alessandro,
real data and experience deeply appreciated. a few questions.
o once a collector commits to ADD-PATH, it's a promise to keep the data forever; well, it's been 26 years of RV and a few less for RIS.
- Did operators find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often?
- Did researchers find the ADD-PATH data sufficiently useful to be worth the cost? How did they use the ADD-PATH data and how often?
- Did folk develop special tools to take advantage of ADD-PATH data?
o On the problem of stream/peer identification. You describe the divergence of BIRD and FRR. What about commercial router vendors?
o You say data were often redundant, though not fully. Did you investigate mechanisms to reduce the storage, or did you see that as a path to complexity and fragility merely to save some spinning rust?
o You mention the withdraw storm between your peer and their peer (and later an announcment storm, I presume). For peers with large out-degree, this could be likely. Were these data interesting in any way, or just more storage?
again, thanks so much for real experience. gives me a bit more clue.
randy --
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/mat-wg
----- To unsubscribe from this mailing list or change your subscription options, please visit: https://mailman.ripe.net/mailman3/lists/mat-wg.ripe.net/ As we have migrated to Mailman 3, you will need to create an account with the email matching your subscription before you can change your settings. More details at: https://www.ripe.net/membership/mail/mailman-3-migration/
participants (4)
-
Alessandro Improta -
Luuk Hendriks -
Randy Bush -
Thomas Holterbach