PROPOSED 1) RIPE NCC shall be tasked to begin updating the following file on a near real-time basis, as number resource assignments are transferred either into or out of the authority of RIPE: ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest 2) RIPE NCC shall be tasked to begin making the above file available for incremental remote update via the rsync protrocol. RATIONALE Background At the present time the most reliable and up-to-the-minute information indicating which RIR any given number resource is being administered by is contained within the so-called "daily stats files" that are generated and published by the five RIRs: ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest ftp://ftp.apnic.net/public/apnic/stats/apnic/delegated-apnic-extended-latest ftp://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest ftp://ftp.afrinic.net/stats/afrinic/delegated-afrinic-extended-latest Additionally, a single file representing the union of all of the above five files, along with additional data relating to (some but not all) IANA reserved number resources is also available: https://nro-extended-stats-from-rpki-team-prod.s3.eu-west-1.amazonaws.com/nr... Note however that the data in this (NRO stats) file lags behind the contents of the five RIR daily stats files, and the content of this file is therfore often less fresh and up-to-date than the information contained in the five RIR stats files. Problem Statement (Proposal 1) Because the five RIR stats files are only updated once a day, and at different times for each RIR, and because number resources are, with increasing frequency, being transfered between regions, it is nowadays frequently the case that there will be, in effect, instances of "version skew" between the five RIR stats files. Instance of such inter-RIR stats files version skew can render it impossible to know, based on the stats files, which RIR currently administers a given number resource. Instances of internet-RIR stats file version skew can arise, and in practice they actually do arise as follows: *) RIR 'A' transfers a number resource `R' to RIR `B'. *) RIR `B' records its administration of `R' in its stats file which is then published within 1 hour. *) RIR `A' records the fact that it no longer administers `R' by removing the associated line from its stats file, and RIR `A's stats file is then published 23 hours later. Under the above scenario, the union of all five of the daily RIR stats files will indicate that *both* RIR `A" and also RIR `B' are the responsible RIRs for the resource `R' for a period of 22 hours. An actual example: As of approximately 07:13 (-0700 / PDT) on 2021-09-17 the following blocks were listed in *both* the stats file for ARIN and also the stats file for RIPE: 140.228.32.0/19 140.228.64.0/19 (Evidence indicates that these 2 block have just been transferred from administration by the ARIN region to administration by the RIPE region.) The problem of version skew between the various daily stats files that are currently published by the various RIRs could be largely or entirely eliminated if all five RIRs henceforth began to update these files on a near real-time basis. RIPE can and should begin doing this as an example to the other RIRs, which may then follow suit. Problem Statement (Proposal 2) The RIR daily stats files, including the one that is generated and published by RIPE NCC, consist of plain text files which are themselves composed of a sequence of plain text lines. As a general matter, very few of these lines change, even over time scales as long as a month. Changes to these files, if any, are entirely minimal when viewed over even shorter time scales, e.g. a day or less. The rsync protocol and rsync servers and clients are especially adept at keeping remote copies of exactly such plain text files up-to-date and synchronized with a master copy stored elsewhere on the Internet. They do so while minimizing bandwidth usage by transfering only those portions of a given file that have changed since the last time a local mirrored copy of the file was updated. Publishing RIPE's stats file via rsync would thus be beneficial to all concerned by minimizing bandwidth usage on both the sending and receiving end. Ideally, RIPE NCC could and should take the lead in making its stats file available via rsync. Once it has done so, other RIRs can be encouraged to follow suit and do likewise with their own daily stats files. Regards, rfg P.S. On a personal note I find it nearly unfathomable that these "daily" stats files are being updated and published only once a day. This seems to me almost reminicent of technological solutions considered appropriate in the prior century.
Hi Ronald,
On 18 Sep 2021, at 03:48, Ronald F. Guilmette via db-wg <db-wg@ripe.net> wrote:
...
P.S. On a personal note I find it nearly unfathomable that these "daily" stats files are being updated and published only once a day. This seems to me almost reminicent of technological solutions considered appropriate in the prior century.
Be aware that Whois updates its list of in-region (RIPE) resources from the internal RIPE NCC resource registry every 15 minutes, and not once a day from the stats file. So any changes to the registry are reflected quickly in Whois. Resources can be updated promptly, and queries (with the '--resource' flag or with RDAP) will return the correct source. This only affects *RIPE* resources, we only update the out-of-region resources from the delegated stats once a day. Regards Ed Shryane RIPE NCC
In message <9F5014CA-3A93-432E-BEC2-1712476CA1A7@ripe.net>, Edward Shryane <eshryane@ripe.net> wrote:
Be aware that Whois updates its list of in-region (RIPE) resources from the internal RIPE NCC resource registry every 15 minutes, and not once a day from the stats file.
OK, so can we have the stats file updated on the same schedule as the WHOIS, i.e. once every 15 min? Seems like a no-brainer to me! Regards, rfg
Hi Ronald,
OK, so can we have the stats file updated on the same schedule as the WHOIS, i.e. once every 15 min?
Seems like a no-brainer to me!
Regards, rfg
I will discuss internally and let you know whether we can increase the frequency. Regards Ed
Dear Ronald, Colleagues, I've discussed Ronald's proposal internally, and we can generate a delegated stats file more frequently, keeping in mind: (1) The current delegated stats file is generated once daily according to the "RIR Statistics Exchange Format": "The details of this format are to be considered a published RIR standard, on which other parties are expected to rely. RIRs must comply with this standard in every respect." "This file is to be produced daily. The last time of any record for the file is 23:59:59 in the local time zone of the producing RIR. (i.e. the last possible time on the specified calendar day in that time zone)" Ref. https://ftp.ripe.net/ripe/stats/RIR-Statistics-Exchange-Format.txt We could generate this file more frequently, but would need to update the format in consultation with the other RIRs first. Alternatively, we could generate a separate file instead, whenever a change is made, or on a fixed schedule during the day. (2) If changes in the delegated stats are made visible immediately, any mistakes made during the day will produce incorrect data in the file. Currently we have a chance to correct any mistakes before the daily delegated stats file is generated. (3) Inter-RIR transfers need updates in *both* RIRs delegated stats. The delegated stats data is inconsistent until both RIRs delegated stats are updated, if one side is incrementally updated this inconsistency may be visible for longer. (4) It's technically possible to provide updates via rsync, if the incrementally updated file is insufficient. The RIPE NCC will produce an implementation plan to determine the time and effort needed, if the community finds this useful. I ask the community for feedback whether publishing incremental changes to the delegated stats is useful for them. Regards Ed Shryane RIPE NCC
On 18 Sep 2021, at 03:48, Ronald F. Guilmette via db-wg <db-wg@ripe.net> wrote:
PROPOSED
1) RIPE NCC shall be tasked to begin updating the following file on a near real-time basis, as number resource assignments are transferred either into or out of the authority of RIPE:
ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest
2) RIPE NCC shall be tasked to begin making the above file available for incremental remote update via the rsync protrocol.
RATIONALE
Background
At the present time the most reliable and up-to-the-minute information indicating which RIR any given number resource is being administered by is contained within the so-called "daily stats files" that are generated and published by the five RIRs:
ftp://ftp.arin.net/pub/stats/arin/delegated-arin-extended-latest ftp://ftp.ripe.net/pub/stats/ripencc/delegated-ripencc-extended-latest ftp://ftp.apnic.net/public/apnic/stats/apnic/delegated-apnic-extended-latest ftp://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-extended-latest ftp://ftp.afrinic.net/stats/afrinic/delegated-afrinic-extended-latest
Additionally, a single file representing the union of all of the above five files, along with additional data relating to (some but not all) IANA reserved number resources is also available:
https://nro-extended-stats-from-rpki-team-prod.s3.eu-west-1.amazonaws.com/nr...
Note however that the data in this (NRO stats) file lags behind the contents of the five RIR daily stats files, and the content of this file is therfore often less fresh and up-to-date than the information contained in the five RIR stats files.
Problem Statement (Proposal 1)
Because the five RIR stats files are only updated once a day, and at different times for each RIR, and because number resources are, with increasing frequency, being transfered between regions, it is nowadays frequently the case that there will be, in effect, instances of "version skew" between the five RIR stats files. Instance of such inter-RIR stats files version skew can render it impossible to know, based on the stats files, which RIR currently administers a given number resource.
Instances of internet-RIR stats file version skew can arise, and in practice they actually do arise as follows:
*) RIR 'A' transfers a number resource `R' to RIR `B'.
*) RIR `B' records its administration of `R' in its stats file which is then published within 1 hour.
*) RIR `A' records the fact that it no longer administers `R' by removing the associated line from its stats file, and RIR `A's stats file is then published 23 hours later.
Under the above scenario, the union of all five of the daily RIR stats files will indicate that *both* RIR `A" and also RIR `B' are the responsible RIRs for the resource `R' for a period of 22 hours.
An actual example:
As of approximately 07:13 (-0700 / PDT) on 2021-09-17 the following blocks were listed in *both* the stats file for ARIN and also the stats file for RIPE:
140.228.32.0/19 140.228.64.0/19
(Evidence indicates that these 2 block have just been transferred from administration by the ARIN region to administration by the RIPE region.)
The problem of version skew between the various daily stats files that are currently published by the various RIRs could be largely or entirely eliminated if all five RIRs henceforth began to update these files on a near real-time basis. RIPE can and should begin doing this as an example to the other RIRs, which may then follow suit.
Problem Statement (Proposal 2)
The RIR daily stats files, including the one that is generated and published by RIPE NCC, consist of plain text files which are themselves composed of a sequence of plain text lines. As a general matter, very few of these lines change, even over time scales as long as a month. Changes to these files, if any, are entirely minimal when viewed over even shorter time scales, e.g. a day or less.
The rsync protocol and rsync servers and clients are especially adept at keeping remote copies of exactly such plain text files up-to-date and synchronized with a master copy stored elsewhere on the Internet. They do so while minimizing bandwidth usage by transfering only those portions of a given file that have changed since the last time a local mirrored copy of the file was updated. Publishing RIPE's stats file via rsync would thus be beneficial to all concerned by minimizing bandwidth usage on both the sending and receiving end. Ideally, RIPE NCC could and should take the lead in making its stats file available via rsync. Once it has done so, other RIRs can be encouraged to follow suit and do likewise with their own daily stats files.
Regards, rfg
P.S. On a personal note I find it nearly unfathomable that these "daily" stats files are being updated and published only once a day. This seems to me almost reminicent of technological solutions considered appropriate in the prior century.
I ask the community for feedback whether publishing incremental changes to the delegated stats is useful for them.
not for me. i have no need for up to the minute delegation data, as i do not hook it into fast churn automation. randy --- randy@psg.com `gpg --locate-external-keys --auto-key-locate wkd randy@psg.com` signatures are back, thanks to dmarc header butchery
participants (3)
-
Edward Shryane -
Randy Bush -
Ronald F. Guilmette