Good morning, Natanlie pointed us to https://www.ripe.net/manage-ips-and-asns/resource-management/rpki/rpki-plann... a while ago. Among other things this says: “In preparation for the improved RPKI repository architecture, the distributed nature of the RRDP repository is going to be implemented using containers and krill-sync that pulls data from the centralised on-premise repository. This greatly simplifies smooth transitioning between publication servers without any downtime. NOTE: We are not referring to cloud technologies here, just to our internal deployment technologies.” The silence here worries me. I would like to see some feedback from this group whether this is what you want to see happening. The RIPE Routing WG is the forum for giving guidance to the RIPE NCC about RPKI. I know other channels exist too and that is fine. I also know that individuals here seem to be happy with what is happening. However private channels and conversations are not the way RIPE does this. This group is where the RIPE NCC looks for guidance and where that guidance gets properly archived and responded to. Daniel
To be perfectly clear: I have *no issues* with this plan at all! It is the total silence *on this list* that worries me. On 12 Jul 2021, at 10:23, Daniel Karrenberg wrote:
Good morning,
Natanlie pointed us to https://www.ripe.net/manage-ips-and-asns/resource-management/rpki/rpki-plann... a while ago. Among other things this says:
“In preparation for the improved RPKI repository architecture, the distributed nature of the RRDP repository is going to be implemented using containers and krill-sync that pulls data from the centralised on-premise repository. This greatly simplifies smooth transitioning between publication servers without any downtime.
NOTE: We are not referring to cloud technologies here, just to our internal deployment technologies.”
The silence here worries me.
I would like to see some feedback from this group whether this is what you want to see happening. The RIPE Routing WG is the forum for giving guidance to the RIPE NCC about RPKI. I know other channels exist too and that is fine. I also know that individuals here seem to be happy with what is happening. However private channels and conversations are not the way RIPE does this. This group is where the RIPE NCC looks for guidance and where that guidance gets properly archived and responded to.
Daniel
to be honest, i do not know what krill-sync is. but rather than going directly down the rabbit hole of advertising it, perhaps this is another case of the operational community being brought in late. what is the problem? what is the solution space? what are the criteria? why should we love this approach?
“In preparation for the improved RPKI repository architecture, the distributed nature of the RRDP repository
and rsync is still a first class protocol, i assume. randy --- randy@psg.com `gpg --locate-external-keys --auto-key-locate wkd randy@psg.com` signatures are back, thanks to dmarc header butchery
Randy, It might also be that the operational community has chosen other fora to discuss because this working group is not working. Daniel On 12 Jul 2021, at 19:44, Randy Bush wrote:
to be honest, i do not know what krill-sync is. but rather than going directly down the rabbit hole of advertising it, perhaps this is another case of the operational community being brought in late. what is the problem? what is the solution space? what are the criteria? why should we love this approach?
“In preparation for the improved RPKI repository architecture, the distributed nature of the RRDP repository
and rsync is still a first class protocol, i assume.
randy
--- randy@psg.com `gpg --locate-external-keys --auto-key-locate wkd randy@psg.com` signatures are back, thanks to dmarc header butchery
evenin' daniel,
It might also be that the operational community has chosen other fora to discuss because this working group is not working.
two things not that i am aware and ncc staff is supposed to be part of the operational community. just that part who we support to bear the administrative burden. randy --- randy@psg.com `gpg --locate-external-keys --auto-key-locate wkd randy@psg.com` signatures are back, thanks to dmarc header butchery
On Tue, Jul 13, 2021 at 05:25:11AM +0200, Daniel Karrenberg wrote:
It might also be that the operational community has chosen other fora to discuss because this working group is not working.
What a strange thing to say. Of course there are other fora to discuss RPKI, one of the most important ones is IETF's SIDROPS working group (which is quite active!). As for the road map - RIPE NCC indicated feedback can be shared with the routing-wg@ or with rpki@ripe.net. I myself opted to try the latter route to re-iterate a request for publish dashboards and graphs about the RPKI service which resulted in 'RPKI-2021-#01' being added to the roadmap. The motivation behind RPKI-2021-#01 is that many IXPs offer publicly accessible graphs ala: https://www.ams-ix.net/ams/documentation/total-stats https://portal.linx.net/ https://www.jpnap.net/ix/traffic.html https://www.netnod.se/ix/statistics https://de-cix.net/en/locations/frankfurt/statistics When incidents happen, these graphs enable the IX participants to quickly understand whether 'something is wrong', because humans are really good at pattern recognition. I imagine that developing more insight into the RIPE NCC RPKI service will offer the community similar benefits as what the community gleans from these public IX stats, hence the ask for RPKI-2021-#01. Kind regards, Job
On 13/07/2021 14:08, Job Snijders via routing-wg wrote:
On Tue, Jul 13, 2021 at 05:25:11AM +0200, Daniel Karrenberg wrote:
It might also be that the operational community has chosen other fora to discuss because this working group is not working.
What a strange thing to say. Of course there are other fora to discuss RPKI, one of the most important ones is IETF's SIDROPS working group (which is quite active!).
As for the road map - RIPE NCC indicated feedback can be shared with the routing-wg@ or with rpki@ripe.net. I myself opted to try the latter route to re-iterate a request for publish dashboards and graphs about the RPKI service which resulted in 'RPKI-2021-#01' being added to the roadmap.
The motivation behind RPKI-2021-#01 is that many IXPs offer publicly accessible graphs ala:
https://www.ams-ix.net/ams/documentation/total-stats https://portal.linx.net/ https://www.jpnap.net/ix/traffic.html https://www.netnod.se/ix/statistics https://de-cix.net/en/locations/frankfurt/statistics
When incidents happen, these graphs enable the IX participants to quickly understand whether 'something is wrong', because humans are really good at pattern recognition.
I imagine that developing more insight into the RIPE NCC RPKI service will offer the community similar benefits as what the community gleans from these public IX stats, hence the ask for RPKI-2021-#01.
Kind regards,
Job
Tool missing that I would like to see: Breakdown by country - unknowns + invalid ROAs. Listed as a table of prefix + ASN. Best I have found is https://stats.labs.apnic.net/roas which lists totals but doesn't provide a breakdown by specific prefix and ASN per country. Maybe I'm missing it. Regards, Hank
Hi Hank,
Op 13 jul. 2021, om 14:46 heeft Hank Nussbacher <hank@interall.co.il> het volgende geschreven:
On 13/07/2021 14:08, Job Snijders via routing-wg wrote:
It might also be that the operational community has chosen other fora to discuss because this working group is not working. What a strange thing to say. Of course there are other fora to discuss RPKI, one of the most important ones is IETF's SIDROPS working group (which is quite active!). As for the road map - RIPE NCC indicated feedback can be shared with the routing-wg@ or with rpki@ripe.net. I myself opted to try the latter route to re-iterate a request for publish dashboards and graphs about
On Tue, Jul 13, 2021 at 05:25:11AM +0200, Daniel Karrenberg wrote: the RPKI service which resulted in 'RPKI-2021-#01' being added to the roadmap. The motivation behind RPKI-2021-#01 is that many IXPs offer publicly accessible graphs ala: https://www.ams-ix.net/ams/documentation/total-stats https://portal.linx.net/ https://www.jpnap.net/ix/traffic.html https://www.netnod.se/ix/statistics https://de-cix.net/en/locations/frankfurt/statistics When incidents happen, these graphs enable the IX participants to quickly understand whether 'something is wrong', because humans are really good at pattern recognition. I imagine that developing more insight into the RIPE NCC RPKI service will offer the community similar benefits as what the community gleans from these public IX stats, hence the ask for RPKI-2021-#01. Kind regards, Job
Tool missing that I would like to see:
Breakdown by country - unknowns + invalid ROAs. Listed as a table of prefix + ASN. Best I have found is https://stats.labs.apnic.net/roas which lists totals but doesn't provide a breakdown by specific prefix and ASN per country. Maybe I'm missing it.
Regards, Hank
Is this of any help? Romain Fontugne launched this tool this week: https://ihr.iijlab.net/ihr/en-us/rov <https://ihr.iijlab.net/ihr/en-us/rov> (worldwide) https://ihr.iijlab.net/ihr/en-us/countries/FR <https://ihr.iijlab.net/ihr/en-us/countries/FR> (per country) https://ihr.iijlab.net/ihr/en-us/networks/AS7922 <https://ihr.iijlab.net/ihr/en-us/networks/AS7922> (per ASN) Kind regards, Nathalie Trenaman RIPE NCC
On 13/07/2021 16:33, Nathalie Trenaman wrote:
Hi Hank,
Op 13 jul. 2021, om 14:46 heeft Hank Nussbacher <hank@interall.co.il <mailto:hank@interall.co.il>> het volgende geschreven:
On 13/07/2021 14:08, Job Snijders via routing-wg wrote:
It might also be that the operational community has chosen other fora to discuss because this working group is not working. What a strange thing to say. Of course there are other fora to discuss RPKI, one of the most important ones is IETF's SIDROPS working group (which is quite active!). As for the road map - RIPE NCC indicated feedback can be shared with the routing-wg@ or with rpki@ripe.net <mailto:rpki@ripe.net>. I myself opted to try the latter route to re-iterate a request for publish dashboards and graphs about
On Tue, Jul 13, 2021 at 05:25:11AM +0200, Daniel Karrenberg wrote: the RPKI service which resulted in 'RPKI-2021-#01' being added to the roadmap. The motivation behind RPKI-2021-#01 is that many IXPs offer publicly accessible graphs ala: https://www.ams-ix.net/ams/documentation/total-stats <https://www.ams-ix.net/ams/documentation/total-stats> https://portal.linx.net/ <https://portal.linx.net/> https://www.jpnap.net/ix/traffic.html <https://www.jpnap.net/ix/traffic.html> https://www.netnod.se/ix/statistics <https://www.netnod.se/ix/statistics> https://de-cix.net/en/locations/frankfurt/statistics <https://de-cix.net/en/locations/frankfurt/statistics> When incidents happen, these graphs enable the IX participants to quickly understand whether 'something is wrong', because humans are really good at pattern recognition. I imagine that developing more insight into the RIPE NCC RPKI service will offer the community similar benefits as what the community gleans from these public IX stats, hence the ask for RPKI-2021-#01. Kind regards, Job
Tool missing that I would like to see:
Breakdown by country - unknowns + invalid ROAs. Listed as a table of prefix + ASN. Best I have found is https://stats.labs.apnic.net/roas <https://stats.labs.apnic.net/roas> which lists totals but doesn't provide a breakdown by specific prefix and ASN per country. Maybe I'm missing it.
Regards, Hank
Is this of any help? Romain Fontugne launched this tool this week: https://ihr.iijlab.net/ihr/en-us/rov <https://ihr.iijlab.net/ihr/en-us/rov> (worldwide) https://ihr.iijlab.net/ihr/en-us/countries/FR <https://ihr.iijlab.net/ihr/en-us/countries/FR> (per country)
Perfect! Thanks, Hank
https://ihr.iijlab.net/ihr/en-us/networks/AS7922 <https://ihr.iijlab.net/ihr/en-us/networks/AS7922> (per ASN)
Kind regards, Nathalie Trenaman RIPE NCC
On 13 Jul 2021, at 10:46 pm, Hank Nussbacher <hank@interall.co.il> wrote:
On 13/07/2021 14:08, Job Snijders via routing-wg wrote:
It might also be that the operational community has chosen other fora to discuss because this working group is not working. What a strange thing to say. Of course there are other fora to discuss RPKI, one of the most important ones is IETF's SIDROPS working group (which is quite active!). As for the road map - RIPE NCC indicated feedback can be shared with the routing-wg@ or with rpki@ripe.net. I myself opted to try the latter route to re-iterate a request for publish dashboards and graphs about
On Tue, Jul 13, 2021 at 05:25:11AM +0200, Daniel Karrenberg wrote: the RPKI service which resulted in 'RPKI-2021-#01' being added to the roadmap. The motivation behind RPKI-2021-#01 is that many IXPs offer publicly accessible graphs ala: https://www.ams-ix.net/ams/documentation/total-stats https://portal.linx.net/ https://www.jpnap.net/ix/traffic.html https://www.netnod.se/ix/statistics https://de-cix.net/en/locations/frankfurt/statistics When incidents happen, these graphs enable the IX participants to quickly understand whether 'something is wrong', because humans are really good at pattern recognition. I imagine that developing more insight into the RIPE NCC RPKI service will offer the community similar benefits as what the community gleans from these public IX stats, hence the ask for RPKI-2021-#01. Kind regards, Job
Tool missing that I would like to see:
Breakdown by country - unknowns + invalid ROAs. Listed as a table of prefix + ASN. Best I have found is https://stats.labs.apnic.net/roas which lists totals but doesn't provide a breakdown by specific prefix and ASN per country. Maybe I'm missing it.
https://stats.labs.apnic.net/roas/IL more generally, https://stats.labs.apnic.net/roas/cc for any country code and https://stats.labs.apnic.net/roas/ASnn for any AS (Maxmind is pretty crappy and there are lots of small scale anomalies in per-country attribution in this report which annoy me A LOT!) regards, Geoff
Hi, On Mon, Jul 12, 2021 at 10:23:20AM +0200, Daniel Karrenberg wrote:
Natanlie pointed us to https://www.ripe.net/manage-ips-and-asns/resource-management/rpki/rpki-plann... a while ago. Among other things this says:
“In preparation for the improved RPKI repository architecture, the distributed nature of the RRDP repository is going to be implemented using containers and krill-sync that pulls data from the centralised on-premise repository. This greatly simplifies smooth transitioning between publication servers without any downtime.
NOTE: We are not referring to cloud technologies here, just to our internal deployment technologies.”
The silence here worries me.
What silence?! Over the last few months there have been quite some mail threads in this working group about RPKI and RPKI outage incidents, and NCC staff have provided updates during the virtual RIPE meetings in the Routing WG slot. To me the roadmap seems to reflect the sentiment that reliability is the key objective at this moment in time.
I would like to see some feedback from this group whether this is what you want to see happening. The RIPE Routing WG is the forum for giving guidance to the RIPE NCC about RPKI. I know other channels exist too and that is fine. I also know that individuals here seem to be happy with what is happening. However private channels and conversations are not the way RIPE does this. This group is where the RIPE NCC looks for guidance and where that guidance gets properly archived and responded to.
To be honest I am not sure what the purpose of krill-sync is. In May 2021 [1] extensive testing was conducted with the help of the NLNOG RING to see if krill-sync could be used to power the RSYNC service, but it turned out there were multiple issues with krill-sync making it a suboptimal choice. I believe RIPE NCC ended up deploying a different solution to serve RSYNC - and my hope is that the recently-achieved stability is here to stay, because the current setup seems to work quite nicely. As for 'hidden RRDP' master, I fail to see what the benefits of krill-sync are compared to say Varnish [2] (or Squid [3]). Or what already is achieved by using a CDN to deliver the RRDP deltas. Maybe the krill-sync reference is an outdated comment? Kind regards, Job [1]: https://www.ripe.net/ripe/mail/archives/routing-wg/2021-May/004345.html [2]: https://varnish-cache.org/ [3]: http://www.squid-cache.org/
Hi Job,
On 13 Jul 2021, at 12:57, Job Snijders via routing-wg <routing-wg@ripe.net> wrote:
Hi,
On Mon, Jul 12, 2021 at 10:23:20AM +0200, Daniel Karrenberg wrote:
Natanlie pointed us to https://www.ripe.net/manage-ips-and-asns/resource-management/rpki/rpki-plann... a while ago. Among other things this says:
“In preparation for the improved RPKI repository architecture, the distributed nature of the RRDP repository is going to be implemented using containers and krill-sync that pulls data from the centralised on-premise repository. This greatly simplifies smooth transitioning between publication servers without any downtime.
NOTE: We are not referring to cloud technologies here, just to our internal deployment technologies.”
The silence here worries me.
What silence?!
Over the last few months there have been quite some mail threads in this working group about RPKI and RPKI outage incidents, and NCC staff have provided updates during the virtual RIPE meetings in the Routing WG slot.
To me the roadmap seems to reflect the sentiment that reliability is the key objective at this moment in time.
I would like to see some feedback from this group whether this is what you want to see happening. The RIPE Routing WG is the forum for giving guidance to the RIPE NCC about RPKI. I know other channels exist too and that is fine. I also know that individuals here seem to be happy with what is happening. However private channels and conversations are not the way RIPE does this. This group is where the RIPE NCC looks for guidance and where that guidance gets properly archived and responded to.
To be honest I am not sure what the purpose of krill-sync is.
In May 2021 [1] extensive testing was conducted with the help of the NLNOG RING to see if krill-sync could be used to power the RSYNC service, but it turned out there were multiple issues with krill-sync making it a suboptimal choice. I believe RIPE NCC ended up deploying a different solution to serve RSYNC - and my hope is that the recently-achieved stability is here to stay, because the current setup seems to work quite nicely.
We are [1] evaluating krill-sync as a tool to build rsync servers that are independent of NFS and can use cached IO. The reason for this is rsync fallback. We see ~139 RPs using the rsync repository (as well as the majority of the NLNOG RING nodes ) and >1600 RPs using the RRDP repository [2]. When rsync fallback happens for many RPs, the current infrastructure will likely not scale, even when each RP starts from the last RRDP state. We are evaluating krill-sync because it allows us to build a rsync repository from RRDP and is available as an open-source project. I recall that while evaluating that krill-sync based environment we found three issues: * Repository versions need to be available for two hours _after they last were the current version_ to give slow clients the chance to retrieve them [3]. * The modification time of objects needs to be the same (between nodes and between copies for a serial) to prevent additional IOs for RPs. * There are very slow outliers reading repositories, but keeping versions available for two hours is long enough in practice. Finding these issues was good: it ensured that they were accounted for in our implementation that writes to NFS. After reporting the relevant issues upstream they have been fixed in krill-sync. The use of NLNOG RING helped verify the current NFS based setup - which I agree is working nicely. Kind regards, Ties [1]: https://www.ripe.net/ripe/mail/archives/routing-wg/2021-June/004351.html [2]: rsync: number of unique IPs reading from /repository yesterday in one hour. hour-to-hour variance is minimal. RRDP: number of unique IPs retrieving notification.xml >24 times/day in early July. [3]: Example: revision 0 gets published at 0h0m, revision 1 at 1h59m, revision 2 at 2h01m (and revision 0 is deleted). The files that clients that connect at 1h58m read get deleted.
participants (7)
-
Daniel Karrenberg -
Geoff Huston -
Hank Nussbacher -
Job Snijders -
Nathalie Trenaman -
Randy Bush -
Ties de Kock