Issues downloading large result files
Hi everyone, I'm seeing something very strange when trying to download the result files for some large measurements. The download is a different size each time (sometimes 1.4GB, then 3.2GB, then back to 1 GB again) and checking the end of the file I can always see it's not complete as it stops in the middle of a measurement, closing brackets aren't there etc. I guess this is because of the size of the file, not sure if there's some sort of limit on RIPE Atlas? If anyone here has seen anything like this or has any advice on how to fully download the result that would be much appreciated! Thank you, Ana
Hello, We've seen periods where someone was downloading 10-30GB result blobs (years' worth of results), multiple downloads in parallel, and while it stressed the service, it worked for the most part. This *could* work, but it's suboptimal. My advice would be to download in chunks (perhaps daily or weekly ones, depending on how much data that is). You can do this via the start=TIMESTAMP and stop=TIMESTAMP parameters, eg. https://atlas.ripe.net/api/v2/measurements/<ID>/results/?start=1688947200&stop=1689033599&format=txt Hope this helps, Robert On 2023-07-19 11:48, Ana wrote:
Hi everyone,
I'm seeing something very strange when trying to download the result files for some large measurements.
The download is a different size each time (sometimes 1.4GB, then 3.2GB, then back to 1 GB again) and checking the end of the file I can always see it's not complete as it stops in the middle of a measurement, closing brackets aren't there etc.
I guess this is because of the size of the file, not sure if there's some sort of limit on RIPE Atlas?
If anyone here has seen anything like this or has any advice on how to fully download the result that would be much appreciated!
Thank you,
Ana
Hi Robert, This was very helpful and I can confirm the TS approach worked. Thank you! Ana
On 20 Jul 2023, at 14:52, Robert Kisteleki <robert@ripe.net> wrote:
Hello,
We've seen periods where someone was downloading 10-30GB result blobs (years' worth of results), multiple downloads in parallel, and while it stressed the service, it worked for the most part. This *could* work, but it's suboptimal.
My advice would be to download in chunks (perhaps daily or weekly ones, depending on how much data that is). You can do this via the start=TIMESTAMP and stop=TIMESTAMP parameters, eg.
https://atlas.ripe.net/api/v2/measurements/<ID>/results/?start=1688947200&stop=1689033599&format=txt
Hope this helps, Robert
On 2023-07-19 11:48, Ana wrote: Hi everyone, I'm seeing something very strange when trying to download the result files for some large measurements. The download is a different size each time (sometimes 1.4GB, then 3.2GB, then back to 1 GB again) and checking the end of the file I can always see it's not complete as it stops in the middle of a measurement, closing brackets aren't there etc. I guess this is because of the size of the file, not sure if there's some sort of limit on RIPE Atlas? If anyone here has seen anything like this or has any advice on how to fully download the result that would be much appreciated! Thank you, Ana
participants (3)
-
Ana -
Ana Custura
-
Robert Kisteleki