I'd like to go out on the record and say that RIPE RIS is invaluable to the internet research and routing analytics world, and a large part of the invaluableness comes from it's longevity in its data archives. That being said, it's understandable that the storage costs to store bgp ribs and update messages would soon become untenable. However I would like to point out (Obviously excluding redundancy) that the currently quoted "raw" data volume for RIS is not that much. (From the blog post linked in the OP)

This dataset currently weighs in at roughly 50 TB of compressed dump and update files, with 80% accounting for the data collected in the last five years

50TB is 12x 4TB drives, 12 3.5" drive slots is a standard size for a 2 Rack Unit sized server. It sounds like the larger concern (at least for the next 5 years) is that the RIPEStat use case is so large (800 TB according to the post). Could you provide more information on what is potentially causing such a large amplified storage usage? Is there a better argument to potentially deprecate some ripe stat tools if they are costing a huge amount of backend storage to provide? Rather than risk degrading RIS's historical archive storage. As a side point I would propose to remove the existence of private right atlas measurements, As almost all things on the internet ( and thus being measured by RIPE Atlas ) are public, the measurements towards such inherently public things probably shouldn't need to be private. On Mon, Dec 11, 2023 at 4:45 AM Stephen Strowes via mat-wg <mat-wg@ripe.net> wrote:

Hi folks,

Bumping this thread in case you missed it:

The NCC is keen to hear the community's thoughts on the principles they should follow for measurement data retention. We had a good discussion on this topic at the WG session during RIPE87, and it'd be good to see it continue here, or on the forum, or privately back to the NCC if you prefer.

The relevant materials are here:

- Recording: https://ripe87.ripe.net/archives/video/1176/ - Slides: https://ripe87.ripe.net/presentations/13-kisteleki-MATWG-RIPE87.pdf, slides 16 & 17 - Labs post: https://labs.ripe.net/author/kistel/ripe-ncc-measurement-data-retention-prin...

Specifically, the principles in question are the bullets on slide 17, or the bullets at the end of the Labs article.

Cheers,

S.

On Wed, Nov 22, 2023 at 11:43 AM Robert Kisteleki <robert@ripe.net> wrote:

...

Dear all,

We've just published a proposal about establishing principles around how the RIPE NCC retains and publishes Internet measurement data, specifically in RIS and RIPE Atlas: https://labs.ripe.net/author/kistel/ripe-ncc-measurement-data-retention-prin...

We would be very happy to see discussions about this here on the mailing list, on the RIPE NCC Forum, or live at RIPE87.

Regards, Robert Kisteleki RIPE NCC

--

To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/mat-wg

--

To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/mat-wg

Thank you Ben, Randy and Joshua for your feedback. The 50 TB (and growing) of storage space needed to hold all of the (compressed) RIS dump and update files for the entire history of the project are (currently) not a big concern. As Ben points out, 50 TB can relatively easily be held in a single machine. These days, three 20 TB disks are sufficient. Obviously, reality is a bit more complex, with redundancy and availability added into the mix. However since this is less problematic, we're not planning any changes in this area at present. Obviously, the 800 TB that was mentioned is a different matter. We store RIS data in a variety of ways for fast access by the RIPEstat front-end servers. We use Apache HBase for this, which uses Apache HDFS as a storage backend, giving us redundant storage. This redundancy comes at a price - by default, HDFS stores its data in triplicate, so the 800 TB of storage used contains just over 250 TB of actual data. Higher replication is possible but will cost even more, and lower is strongly discouraged. Then, for the various widgets / infocards / data endpoints in RIPEstat, the data is transformed and stored in different HBase tables. This does, unfortunately, mean more duplication of data because data is indexed by different aspects in different tables for the specific access pattern. Now these various ways of storing the same data were not a big problem in the past. However, with the growth of RIS and the Internet, the volume of incoming data over time has steadily grown too. Now it has come to a point where we need to start thinking of different ways to make this data available to the community. This is where Robert's RIPE Labs post and his presentation at the meeting in Rome come in. We want to review how we make this data available to you as end users (be it as researchers, as operators or in whatever form that applies) so that we can remain cost effective while giving you the most useful data in a fast and easy-to-access way. So in summary, we're looking at doing exactly what Ben suggests: keep offering the historic dataset as we currently do through the dump and update files, but reviewing how we can reduce cost of the other ways in which we store this data without losing value for our end users. Paul de Weerd RIPE NCC