NRTM replication inefficiencies
Dear all, There may be some improvement opportunity for NRTMv3. The problem with the RIPE NRTMv2/NRTMv3 format is that it has no 'end-of-stream' indicator, thus making the only way to know that the output has ended (vs. hung/stalled/server dead) is either by observing a time-out & reconnect & compare last serial, or by using single-command connections to the NRTM server - which means even worse performance. When connecting to the RIPE NRTM service and issuing a command like: "-g RIPE:3:11012700-LAST -k" It would be good that when the RIPE NRTM server is done sending its inital blurp of backlogged data, the end of that phase of the stream of objects is marked by the server sending something like: "%CURRENT $TS You are now up to date" ($TS can be a unix timestamp) after this server message the client knows that it can stay connected and that it has received all information so far. I would also welcome an investigation into alternative approaches, (some not-via-WHOIS replication mechanisms), perhaps something over HTTPS can be done? Either way, something more robust would be useful. Kind regards, Job ps. An analogy can be made between BGP route refresh as described in RFC 2918 and Enhanced Route Refresh as described in RFC 7313. One first one didn't have an "End of blurp" marker which negatively impacted its usefulness, the later RFCintroduced the End-of-RIB marker which is very useful.
Hi Job, WG,
On 7 Nov 2017, at 23:11, Job Snijders via db-wg <db-wg@ripe.net> wrote:
I would also welcome an investigation into alternative approaches, (some not-via-WHOIS replication mechanisms), perhaps something over HTTPS can be done? Either way, something more robust would be useful.
We recently developed and implement a standard for something similar for RPKI: https://datatracker.ietf.org/doc/rfc8182/ I believe this approach can be useful here as well. Without going into all the RPKI specifics, it works a little something like this: Starting points: = The state of the rpki repository (or whois) at a given point in time can represented by a ‘snapshot’ - This snapshot is “immutable” - therefore they may be cached indefinitely and we can give it a unique URL and deliver it through a distributed CDN = The delta between two consecutive snapshots is also “immutable” data - so again we can cache it and give it a unique URL and distribute = We can publish a notification file (which should NOT be cached) that points to: - the CURRENT snapshot - a list of deltas (each for 1 increment) - total size of deltas MUST not exceed size of snapshot Clients can then just poll the notification file and work out for themselves whether a list of deltas is available to them, or that they need to get the latest snapshot instead. Yes, we use a session_id and hashes of referenced files for additional checks (details in the RFC). The idea behind this design was that we wanted to minimise the impact on the server. In a chatty protocol (like rsync which is still used in RPKI) the server and client need to work out their differences together to determine what needs to be transferred. This is fine in one on one relations, but when a server needs to serve a multitude of clients this doesn’t scale. We want to be able parallelise as much as we can (Amdahl’s law), so we push the computational burden to the clients. The server just needs a one-off investment to create the snapshot and delta and latest notification which it can then offload. Using HTTPS allows us to leverage one of the many, many CDNs out there. This problem has been solved in the industry. So we do not need to invent our own infrastructure for this. Note that in the case of RPKI the protocol is XML based. This made sense because it leveraged existing definitions in the RPKI space that were also XML based. For whois it may make more sense to look at JSON and/or RDAP. Please let me know if you see merit in this kind of ‘delta’ protocol in the whois space. Kind regards Tim Bruijnzeels Assistant Manager Software Engineering and Senior Technology Officer RIPE NCC
On Thu, Nov 09, 2017 at 04:28:03PM +0100, Tim Bruijnzeels wrote:
On 7 Nov 2017, at 23:11, Job Snijders via db-wg <db-wg@ripe.net> wrote:
I would also welcome an investigation into alternative approaches, (some not-via-WHOIS replication mechanisms), perhaps something over HTTPS can be done? Either way, something more robust would be useful.
We recently developed and implement a standard for something similar for RPKI: https://datatracker.ietf.org/doc/rfc8182/
I believe this approach can be useful here as well. Without going into all the RPKI specifics, it works a little something like this:
Starting points: = The state of the rpki repository (or whois) at a given point in time can represented by a ‘snapshot’ - This snapshot is “immutable” - therefore they may be cached indefinitely and we can give it a unique URL and deliver it through a distributed CDN = The delta between two consecutive snapshots is also “immutable” data - so again we can cache it and give it a unique URL and distribute = We can publish a notification file (which should NOT be cached) that points to: - the CURRENT snapshot - a list of deltas (each for 1 increment) - total size of deltas MUST not exceed size of snapshot
Clients can then just poll the notification file and work out for themselves whether a list of deltas is available to them, or that they need to get the latest snapshot instead.
Yes, we use a session_id and hashes of referenced files for additional checks (details in the RFC).
The idea behind this design was that we wanted to minimise the impact on the server. In a chatty protocol (like rsync which is still used in RPKI) the server and client need to work out their differences together to determine what needs to be transferred. This is fine in one on one relations, but when a server needs to serve a multitude of clients this doesn’t scale. We want to be able parallelise as much as we can (Amdahl’s law), so we push the computational burden to the clients. The server just needs a one-off investment to create the snapshot and delta and latest notification which it can then offload. Using HTTPS allows us to leverage one of the many, many CDNs out there. This problem has been solved in the industry. So we do not need to invent our own infrastructure for this.
Note that in the case of RPKI the protocol is XML based. This made sense because it leveraged existing definitions in the RPKI space that were also XML based. For whois it may make more sense to look at JSON and/or RDAP.
Please let me know if you see merit in this kind of ‘delta’ protocol in the whois space.
yes, I think there may be merit to replacing NRTM, and DELTA would certainly be a good source of inspiration. Would it be fair to ask for a two-pronged approach? DELTA-WHOIS + WHOIS-END-OF-BLURP markings? How much work (or complexity?) is involved for RIPE NCC to develop a marking that is send to the client at the end of a '-g' query that also had '-k' enabled? Kind regards, Job
On 9 Nov 2017, at 17:57, Job Snijders <job@instituut.net> wrote:
On Thu, Nov 09, 2017 at 04:28:03PM +0100, Tim Bruijnzeels wrote:
On 7 Nov 2017, at 23:11, Job Snijders via db-wg <db-wg@ripe.net> wrote:
I would also welcome an investigation into alternative approaches, (some not-via-WHOIS replication mechanisms), perhaps something over HTTPS can be done? Either way, something more robust would be useful.
We recently developed and implement a standard for something similar for RPKI: https://datatracker.ietf.org/doc/rfc8182/
I believe this approach can be useful here as well. Without going into all the RPKI specifics, it works a little something like this:
Starting points: = The state of the rpki repository (or whois) at a given point in time can represented by a ‘snapshot’ - This snapshot is “immutable” - therefore they may be cached indefinitely and we can give it a unique URL and deliver it through a distributed CDN = The delta between two consecutive snapshots is also “immutable” data - so again we can cache it and give it a unique URL and distribute = We can publish a notification file (which should NOT be cached) that points to: - the CURRENT snapshot - a list of deltas (each for 1 increment) - total size of deltas MUST not exceed size of snapshot
Clients can then just poll the notification file and work out for themselves whether a list of deltas is available to them, or that they need to get the latest snapshot instead.
Yes, we use a session_id and hashes of referenced files for additional checks (details in the RFC).
The idea behind this design was that we wanted to minimise the impact on the server. In a chatty protocol (like rsync which is still used in RPKI) the server and client need to work out their differences together to determine what needs to be transferred. This is fine in one on one relations, but when a server needs to serve a multitude of clients this doesn’t scale. We want to be able parallelise as much as we can (Amdahl’s law), so we push the computational burden to the clients. The server just needs a one-off investment to create the snapshot and delta and latest notification which it can then offload. Using HTTPS allows us to leverage one of the many, many CDNs out there. This problem has been solved in the industry. So we do not need to invent our own infrastructure for this.
Note that in the case of RPKI the protocol is XML based. This made sense because it leveraged existing definitions in the RPKI space that were also XML based. For whois it may make more sense to look at JSON and/or RDAP.
Please let me know if you see merit in this kind of ‘delta’ protocol in the whois space.
yes, I think there may be merit to replacing NRTM, and DELTA would certainly be a good source of inspiration.
Good to know, that’s what I was curious about.
Would it be fair to ask for a two-pronged approach? DELTA-WHOIS + WHOIS-END-OF-BLURP markings?
For sure. Delta would require more thought, discussion and proposals
How much work (or complexity?) is involved for RIPE NCC to develop a marking that is send to the client at the end of a '-g' query that also had '-k' enabled?
We can look into this next week. Kind regards Tim
Kind regards,
Job
Hi Job, WG,
On 10 Nov 2017, at 15:31, Tim Bruijnzeels via db-wg <db-wg@ripe.net> wrote:
We can look into this next week.
Kind regards
Tim
there is an existing NRTM 'end-of-stream' indicator, if keep-alive is not used: a comment '%END RIPE' is output after the last object. We could also add an 'end-of-stream' comment in keep-alive mode, something like '%END (starting serial) - (ending serial)', which is output after any backlog of updates. It's still possible not to have any updates for a time, which a client may see as a timeout. We could also add a heartbeat mechanism, to periodically output a comment such as '%HEARTBEAT', if there have not been any updates. Do either of these changes necessitate a new protocol version? We could also provide NRTM over HTTP(S) WebSockets, that protocol includes support for a Ping/Pong heartbeat mechanism. Regards Ed
Or you could use TCP's built-in keepalive feature: http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html On Fri, Dec 8, 2017 at 12:56 PM, Edward Shryane via db-wg <db-wg@ripe.net> wrote:
Hi Job, WG,
On 10 Nov 2017, at 15:31, Tim Bruijnzeels via db-wg <db-wg@ripe.net> wrote:
We can look into this next week.
Kind regards
Tim
there is an existing NRTM 'end-of-stream' indicator, if keep-alive is not used: a comment '%END RIPE' is output after the last object. We could also add an 'end-of-stream' comment in keep-alive mode, something like '%END (starting serial) - (ending serial)', which is output after any backlog of updates.
It's still possible not to have any updates for a time, which a client may see as a timeout. We could also add a heartbeat mechanism, to periodically output a comment such as '%HEARTBEAT', if there have not been any updates.
Do either of these changes necessitate a new protocol version?
We could also provide NRTM over HTTP(S) WebSockets, that protocol includes support for a Ping/Pong heartbeat mechanism.
Regards Ed
Hi Agoston,
On 8 Dec 2017, at 13:43, Horváth Ágoston János via db-wg <db-wg@ripe.net> wrote:
Or you could use TCP's built-in keepalive feature:
Yes, this is already possible, you can set the SO_KEEPALIVE option on the socket. However, at least on CentOS7 (Linux), the default is to wait 2 hours before sending a keepalive probe, then 9 probes have to fail (each 75s apart) before declaring the connection is broken. Changing this default behaviour is applied system-wide. Adding a protocol-specific keepalive mechanism may still be useful. Regards Ed
Hi Agoston,
On 8 Dec 2017, at 15:05, Edward Shryane via db-wg <db-wg@ripe.net> wrote:
Hi Agoston,
On 8 Dec 2017, at 13:43, Horváth Ágoston János via db-wg <db-wg@ripe.net> wrote:
Or you could use TCP's built-in keepalive feature:
Yes, this is already possible, you can set the SO_KEEPALIVE option on the socket.
However, at least on CentOS7 (Linux), the default is to wait 2 hours before sending a keepalive probe, then 9 probes have to fail (each 75s apart) before declaring the connection is broken. Changing this default behaviour is applied system-wide.
Adding a protocol-specific keepalive mechanism may still be useful.
Regards Ed
to clarify my earlier reply - changing these system-wide defaults can also be changed on a per-socket basis. A client could use the TCP keep alive mechanism, or we could add a periodic keepalive comment on the server side. Regards Ed
On 08/12/2017 15:05, Edward Shryane via db-wg wrote:
Yes, this is already possible, you can set the SO_KEEPALIVE option on the socket.
However, at least on CentOS7 (Linux), the default is to wait 2 hours before sending a keepalive probe, then 9 probes have to fail (each 75s apart) before declaring the connection is broken. Changing this default behaviour is applied system-wide.
A RIPE NCC internal python application that uses NRTM, does this: conn = socket.create_connection((self.nrtm_host, self.nrtm_port)) # enable TCP keepalive (but only on Linux - the TCP_KEEPIDLE # option is not available for other OSes) # send keepalives after 60s of inactivity # try 3 probes at 15s intervals before closing the connection if platform.system() == 'Linux': logger.info('enabling TCP keepalive') conn.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) conn.setsockopt(socket.SOL_TCP, socket.TCP_KEEPIDLE, 60) conn.setsockopt(socket.SOL_TCP, socket.TCP_KEEPINTVL, 15) conn.setsockopt(socket.SOL_TCP, socket.TCP_KEEPCNT, 3) Regards, Anand Buddhdev RIPE NCC
On Fri, Dec 08, 2017 at 12:56:11PM +0100, Edward Shryane wrote:
We could also add an 'end-of-stream' comment in keep-alive mode, something like '%END (starting serial) - (ending serial)', which is output after any backlog of updates.
Yeah, that would be useful. Is this easy low hanging fruit?
It's still possible not to have any updates for a time, which a client may see as a timeout. We could also add a heartbeat mechanism, to periodically output a comment such as '%HEARTBEAT', if there have not been any updates.
Do either of these changes necessitate a new protocol version?
I doubt it, since "%END XXX - YYY" is a comment.
We could also provide NRTM over HTTP(S) WebSockets, that protocol includes support for a Ping/Pong heartbeat mechanism.
Shouldn't we do away with NRTM at that point and come up with a DELTA-like protocol? :) Kind regards, Job
Hi Job,
On 8 Dec 2017, at 13:51, Job Snijders <job@instituut.net> wrote:
On Fri, Dec 08, 2017 at 12:56:11PM +0100, Edward Shryane wrote:
We could also add an 'end-of-stream' comment in keep-alive mode, something like '%END (starting serial) - (ending serial)', which is output after any backlog of updates.
Yeah, that would be useful. Is this easy low hanging fruit?
Yes, I've already tested it, this change is straightforward. The only downside is unexpected consequences (i.e. we don't want to break existing clients).
It's still possible not to have any updates for a time, which a client may see as a timeout. We could also add a heartbeat mechanism, to periodically output a comment such as '%HEARTBEAT', if there have not been any updates.
Do either of these changes necessitate a new protocol version?
I doubt it, since "%END XXX - YYY" is a comment.
We could deploy a test NRTM version to our RC environment, for compatibility testing with existing clients.
We could also provide NRTM over HTTP(S) WebSockets, that protocol includes support for a Ping/Pong heartbeat mechanism.
Shouldn't we do away with NRTM at that point and come up with a DELTA-like protocol? :)
Kind regards,
Job
This is also possible (as a replacement, or in addition to, NRTM). Regards Ed
On Fri, Dec 08, 2017 at 03:10:28PM +0100, Edward Shryane wrote:
On 8 Dec 2017, at 13:51, Job Snijders <job@instituut.net> wrote: On Fri, Dec 08, 2017 at 12:56:11PM +0100, Edward Shryane wrote:
We could also add an 'end-of-stream' comment in keep-alive mode, something like '%END (starting serial) - (ending serial)', which is output after any backlog of updates.
Yeah, that would be useful. Is this easy low hanging fruit?
Yes, I've already tested it, this change is straightforward. The only downside is unexpected consequences (i.e. we don't want to break existing clients).
Cool!
It's still possible not to have any updates for a time, which a client may see as a timeout. We could also add a heartbeat mechanism, to periodically output a comment such as '%HEARTBEAT', if there have not been any updates.
Do either of these changes necessitate a new protocol version?
I doubt it, since "%END XXX - YYY" is a comment.
We could deploy a test NRTM version to our RC environment, for compatibility testing with existing clients.
Sounds good. Kind regards, Job
Dear Job, Working Group,
On 7 Nov 2017, at 23:11, Job Snijders via db-wg <db-wg@ripe.net> wrote:
Dear all,
There may be some improvement opportunity for NRTMv3.
The problem with the RIPE NRTMv2/NRTMv3 format is that it has no 'end-of-stream' indicator, thus making the only way to know that the output has ended (vs. hung/stalled/server dead) is either by observing a time-out & reconnect & compare last serial, or by using single-command connections to the NRTM server - which means even worse performance.
When connecting to the RIPE NRTM service and issuing a command like:
"-g RIPE:3:11012700-LAST -k"
It would be good that when the RIPE NRTM server is done sending its inital blurp of backlogged data, the end of that phase of the stream of objects is marked by the server sending something like:
"%CURRENT $TS You are now up to date" ($TS can be a unix timestamp)
after this server message the client knows that it can stay connected and that it has received all information so far.
I would also welcome an investigation into alternative approaches, (some not-via-WHOIS replication mechanisms), perhaps something over HTTPS can be done? Either way, something more robust would be useful.
Kind regards,
Job
ps. An analogy can be made between BGP route refresh as described in RFC 2918 and Enhanced Route Refresh as described in RFC 7313. One first one didn't have an "End of blurp" marker which negatively impacted its usefulness, the later RFCintroduced the End-of-RIB marker which is very useful.
as part of the Whois 1.91 release, we've added an NRTM "end of stream" comment, and deployed to the Release Candidate environment. Try connecting to whois-rc.ripe.net <http://whois-rc.ripe.net/> on TCP port 4444, and enter the command "-g RIPE:3:40956391-LAST -k". Each set of changes will be followed by an "end of stream" comment. Changes to objects in RC will appear in the NRTM stream. Changes can be made in RC using the mntner-id (in uppercase) as the password. Query for existing data: https://rc.db.ripe.net/db-web-ui/#/query <https://rc.db.ripe.net/db-web-ui/#/query> Create a new object: https://rc.db.ripe.net/db-web-ui/#/webupdates/select <https://rc.db.ripe.net/db-web-ui/#/webupdates/select> Multiple changes can be made at once using Syncupdates: https://rc.db.ripe.net/db-web-ui/#/syncupdates <https://rc.db.ripe.net/db-web-ui/#/syncupdates> For more information on the Release Candidate environment, refer to: https://www.ripe.net/manage-ips-and-asns/db/release-notes/rc-release-candida... <https://www.ripe.net/manage-ips-and-asns/db/release-notes/rc-release-candidate-environment> Please let us know if you see any issues with this change. Regards Ed Shryane RIPE NCC
On Wed, Feb 21, 2018 at 04:32:07PM +0100, Edward Shryane wrote:
as part of the Whois 1.91 release, we've added an NRTM "end of stream" comment, and deployed to the Release Candidate environment.
Try connecting to whois-rc.ripe.net <http://whois-rc.ripe.net/> on TCP port 4444, and enter the command "-g RIPE:3:40956391-LAST -k". Each set of changes will be followed by an "end of stream" comment.
Seems I can't verify on that endpoint? [job@kiera ~]$ telnet whois-rc.ripe.net 4444 Trying 193.0.6.145... Connected to whois-rc.ripe.net. Escape character is '^]'. %ERROR:402: not authorised to mirror the database from IP address 165.254.255.26 Connection closed by foreign host. [job@kiera ~]$ Kind regards, Job
Hi Job,
On 21 Feb 2018, at 16:38, Job Snijders <job@instituut.net> wrote:
On Wed, Feb 21, 2018 at 04:32:07PM +0100, Edward Shryane wrote:
as part of the Whois 1.91 release, we've added an NRTM "end of stream" comment, and deployed to the Release Candidate environment.
Try connecting to whois-rc.ripe.net <http://whois-rc.ripe.net/> on TCP port 4444, and enter the command "-g RIPE:3:40956391-LAST -k". Each set of changes will be followed by an "end of stream" comment.
Seems I can't verify on that endpoint?
[job@kiera ~]$ telnet whois-rc.ripe.net 4444 Trying 193.0.6.145... Connected to whois-rc.ripe.net. Escape character is '^]'. %ERROR:402: not authorised to mirror the database from IP address 165.254.255.26 Connection closed by foreign host. [job@kiera ~]$
Kind regards,
Job
Apologies, I've now allowed access from anywhere to the NRTM service in Release Candidate, please try again. Regards Ed
participants (5)
-
Anand Buddhdev -
Edward Shryane -
Horváth Ágoston János -
Job Snijders -
Tim Bruijnzeels