New on RIPE Labs: New Architecture Model for K-root Local Instances
Dear colleagues, We have redesigned the K-root architecture for our hosted nodes, formerly known as K-root local instances. System architecture and network setup have been simplified a lot. This will reduce our management effort for K-root and reduce costs for K-root hosts. More details on RIPE Labs: https://labs.ripe.net/Members/romeo_zwart/new-architecture-for-k-root-local-... Kind regards, Mirjam Kuehne RIPE NCC
hi mirjam, romeo, et alia, this looke cool, and lighter weight. but a few questions as usual.
In the new model, the K-root hosted node will only peer with one party, the local host, and the local host is responsible for further propagation of the K-root prefix[1]
does this add an as hop, or are you hiding in the host's asn, or using a private (and thus stripped) asn? and can you explain the motivation for radically reducing the bgp out-degree?
1/ For an old-style local node, the anycast prefix was advertised with the "no-export" community string set, in order to limit the scope of the prefix propagation. This is no longer the case in the new design.
are you intending the node actually propagate globally, or hoping the one bgp peer adds NO_EXPORT? randy
Hi Randy, all, Thanks for your questions. On 15/03/12 03:40 , Randy Bush wrote:
hi mirjam, romeo, et alia,
this looke cool, and lighter weight. but a few questions as usual.
In the new model, the K-root hosted node will only peer with one party, the local host, and the local host is responsible for further propagation of the K-root prefix[1]
does this add an as hop, or are you hiding in the host's asn, or using a private (and thus stripped) asn?
There is no hiding or stripping of the prefix, so yes, this will add a hop. The global or core nodes will of course be connected as currently to major IXP's, announcing the prefix directly. It is possible that there is an impact on routing for some of the clients currently behind local/hosted nodes, however this currently adds up to about 10% of the query load. Also, we do expect a substantial uptake of new hosts, so this will result in a more finer grained distribution of K-root.
and can you explain the motivation for radically reducing the bgp out-degree?
There are two drivers for that. We receive many requests from prospective new K-root hosts. We have considered how to respond to this demand from the community. This made us propose this smaller footprint server, enabling a wider variety of hosts at a lower cost. So, one driver for changing the BGP routing model, is in the simplified hardware. We prefer the single server to spend it's time on answering DNS queries, rather than spend its cycles on routing BGP. The second driver is that we spend a relatively large amount of engineering time on peering management for each instance. In the new model that factor is obviously reduced a lot, which allows us to scale the number of instances and achieve a finer grained distribution without a large increase in management and engineering effort.
1/ For an old-style local node, the anycast prefix was advertised with the "no-export" community string set, in order to limit the scope of the prefix propagation. This is no longer the case in the new design.
are you intending the node actually propagate globally, or hoping the one bgp peer adds NO_EXPORT?
We will discuss with hosts how they propagate the K-root prefix on a case by case basis. Some may prefer to distribute only locally, others may propagate globally. We expect of course that they tune this according to local capacity. We will monitor, using for example RIPE Atlas and DNSMON, the actual impact on K-root reachability for end users and work with the local hosts to optimize routing where needed. Thanks, Romeo
randy
Hi Romeo, thanks for the comprehensive replies, both here and on the blog. On 12/03/2015 09:29, Romeo Zwart wrote:
There are two drivers for that. We receive many requests from prospective new K-root hosts. We have considered how to respond to this demand from the community. This made us propose this smaller footprint server, enabling a wider variety of hosts at a lower cost. So, one driver for changing the BGP routing model, is in the simplified hardware. We prefer the single server to spend it's time on answering DNS queries, rather than spend its cycles on routing BGP.
mmm, depends how it's configured. E.g. if you have the device directly connected with a single o/s, bare-metal installation, it will make almost no difference. If you're using the hardware as a hypervisor, then yes there will be a performance impact but it can easily and cheaply be mitigated by adding an extra core or two and assigning those cores to the virtual router.
The second driver is that we spend a relatively large amount of engineering time on peering management for each instance. In the new model that factor is obviously reduced a lot, which allows us to scale the number of instances and achieve a finer grained distribution without a large increase in management and engineering effort.
Peering management is an issue that IXPs are painfully aware of and we've been doing a lot of work on this in the last 6 months. There's a new json formatted ixp participant data export schema which will allow you to import ixp data from multiple IXPs into your own internal systems. This format is being deployed across multiple IXPs. Once you have that data in your systems, it should be easy enough to use it to automatically configure IXP routers (assuming normal intermediate checks and param validation, etc). As you have an open peering policy, you could even do stuff like configuring bgp sessions to all ixp participants by default, but in passive mode. This would allow IXP participants to connect to you at their leisure. More at: https://github.com/euro-ix/json-schemas
We will discuss with hosts how they propagate the K-root prefix on a case by case basis. Some may prefer to distribute only locally, others may propagate globally. We expect of course that they tune this according to local capacity. We will monitor, using for example RIPE Atlas and DNSMON, the actual impact on K-root reachability for end users and work with the local hosts to optimize routing where needed.
I mentioned the side effect of this in a blog posting, namely that this will cause the k root BGP network distance to appear to be the same as transit. This means that in many cases the ixp instance will not be preferred and the ixp participants will not get the full benefit of having a nearby k root instance. Nick
Hi Nick, On 15/03/12 16:44 , Nick Hilliard wrote:
Hi Romeo,
thanks for the comprehensive replies, both here and on the blog.
And likewise, thanks for the discussion. No doubt more people on the list share your questions. :)
On 12/03/2015 09:29, Romeo Zwart wrote:
There are two drivers for that. We receive many requests from prospective new K-root hosts. We have considered how to respond to this demand from the community. This made us propose this smaller footprint server, enabling a wider variety of hosts at a lower cost. So, one driver for changing the BGP routing model, is in the simplified hardware. We prefer the single server to spend it's time on answering DNS queries, rather than spend its cycles on routing BGP.
mmm, depends how it's configured. E.g. if you have the device directly connected with a single o/s, bare-metal installation, it will make almost no difference.
In your example below, where the node peers with all parties on the peering lan, I would expect it to have a noticeable impact. Unless the IXP has no more than two connected members. ;-) But I admit to not having any hard numbers on this at the moment. FYI, the K-root 'hosted' solution does not involve virtualisation.
If you're using the hardware as a hypervisor, then yes there will be a performance impact but it can easily and cheaply be mitigated by adding an extra core or two and assigning those cores to the virtual router.
The second driver is that we spend a relatively large amount of engineering time on peering management for each instance. In the new model that factor is obviously reduced a lot, which allows us to scale the number of instances and achieve a finer grained distribution without a large increase in management and engineering effort.
Peering management is an issue that IXPs are painfully aware of and we've been doing a lot of work on this in the last 6 months. There's a new json formatted ixp participant data export schema which will allow you to import ixp data from multiple IXPs into your own internal systems. This format is being deployed across multiple IXPs.
Interesting. Thanks for that suggestion. We will certainly look into this to see if/how we can use this to improve the fit of our model for IXP's. However, I don't expect that we could support this on the short term.
Once you have that data in your systems, it should be easy enough to use it to automatically configure IXP routers (assuming normal intermediate checks and param validation, etc). As you have an open peering policy, you could even do stuff like configuring bgp sessions to all ixp participants by default, but in passive mode. This would allow IXP participants to connect to you at their leisure.
More at: https://github.com/euro-ix/json-schemas
We will discuss with hosts how they propagate the K-root prefix on a case by case basis. Some may prefer to distribute only locally, others may propagate globally. We expect of course that they tune this according to local capacity. We will monitor, using for example RIPE Atlas and DNSMON, the actual impact on K-root reachability for end users and work with the local hosts to optimize routing where needed.
I mentioned the side effect of this in a blog posting, namely that this will cause the k root BGP network distance to appear to be the same as transit. This means that in many cases the ixp instance will not be preferred and the ixp participants will not get the full benefit of having a nearby k root instance.
That may indeed happen, depending on path length over the transit of course. However, IXP participants may chose to prefer the local route over the transit route to still get that full benefit. Thanks for your comments. Our proposed model is targeting the vast majority of prospective hosts that have approached us, generally service providers. The IXP scenario is somewhat of a corner case. We will use your input to see how we can improve the model to also fit IXP's better. We already discussed this with some other IXP's as well and I propose that we discuss this a bit further, off list, to see how we might better address the IXP requirements. Kind regards, Romeo
Nick
participants (4)
-
Mirjam Kuehne -
Nick Hilliard -
Randy Bush -
Romeo Zwart