filter ping results by AS
Hi there, I'm currently trying to figure out, how I can get all ping results to(/from) my AS within a certain time frame, to run some statistics. The REST-API documentation [0] states, that measurements can be filtered by dst_asn, but it doesn't seem to influence the results I'm getting:
filter: { 'start_time__gt': 1445231337, 'type': 'ping', 'dst_asn': 553 }
dst_asn dst_name -------------------------------------------- 2607 sk-bts-as2607.anchors.atlas.ripe.net 2602 lu-lux-as2602.anchors.atlas.ripe.net 16154 88.213.204.251 1853 at-vie-as1853.anchors.atlas.ripe.net 8608 195.18.114.5 32475 www.siteground.com 12322 78.194.137.59 None ch-zrh-as559.anchors.atlas.ripe.net
Also, why do some measurements have no dst_asn? Regards, Sebastian [0] https://atlas.ripe.net/docs/rest/
We're working on migrating some of our API code and it looks like you bumped into something I missed when doing that migration. The good news though, is that now that you've found it, I've fixed it and you can now do API calls like this again: https://atlas.ripe.net/api/v1/measurement/?dst_asn=3333 As for the measurements that don't have an ASN value, we're looking into that, but there are a number of legitimate reasons for their not having one, including having no target specified (some DNS measurements for example).
Awesome, thank you for the quick fix! Is there (or will there be) some kind of shortcut for fetching results, without filtering for the corresponding measurement definitions first? Regards, Sebastian On 19.10.2015 12:25, Daniel Quinn wrote:
We're working on migrating some of our API code and it looks like you bumped into something I missed when doing that migration. The good news though, is that now that you've found it, I've fixed it and you can now do API calls like this again:
https://atlas.ripe.net/api/v1/measurement/?dst_asn=3333
As for the measurements that don't have an ASN value, we're looking into that, but there are a number of legitimate reasons for their not having one, including having no target specified (some DNS measurements for example).
It’s not on our List of Things to Do, no. The difficulty is in the massive amounts of data — there’s simply no easy way to fetch arbitrary results from such a large data store based on arbitrary lookup values. Currently we store data keyed by measurement id, probe id, and time, and that covers most of the use-cases. You should know however that we do supply weekly dump files <http://ftp.ripe.net/ripe/atlas/measurements/> of measurement meta data, so you can use these to prepare your results queries without beating the crap out of our API service :-)
So, I figured that I don't have hundreds of TB of storage and dismissed the idea of fetching all the results :-) I got the probe- and measurement dumps and threw them in a database, but it would still be nice for me to fetch some results by source-AS. The participation-request-log API won't give me results filtered by probe_id:
{"detail": "You do not have permission to perform this action."}
Are there any participation-request-log dumps, too? Or is there some other way to do this? Regards, Sebastian On 19.10.2015 15:22, Daniel Quinn wrote:
It’s not on our List of Things to Do, no. The difficulty is in the massive amounts of data — there’s simply no easy way to fetch arbitrary results from such a large data store based on arbitrary lookup values.
Currently we store data keyed by measurement id, probe id, and time, and that covers most of the use-cases.
You should know however that we do supply weekly dump files <http://ftp.ripe.net/ripe/atlas/measurements/> of measurement meta data, so you can use these to prepare your results queries without beating the crap out of our API service :-)
If I’m understanding your situation correctly, you want to see a list of measurements that include probes from a particular AS as their source. At present, there’s no easy way to make a single API call to get that, but you could synthesise that information by comparing two reasonably small bodies of data. There are two bulk data downloads that you can use for this: measurement metadata <http://ftp.ripe.net/ripe/atlas/measurements/> (already mentioned, updated weekly) and the probe data <http://ftp.ripe.net/ripe/atlas/probes/archive/2015/10/> (daily). Basically, you need to do three things: * *Compile a list of probe ids that represent probes within the AS you want*. You can do this by looping over the probes in the daily dump and capturing the id for any probe that has |asn_v4| or |asn_v6| equal to the AS in question. * *Compile a list of measurements using these probes*. You can do this by looping over the list of measurements, capturing the ids of measurements that have the aforementioned probe ids in the |probes| list. * Fetch the probe-specific data from the API using both ids: |https://atlas.ripe.net/api/v1/measurement/**measurement_id**/result/?prb_id=**probe_id**,**probe_id**&format=txt | The above call will return all of the results for measurement |#${measurement_id}|, but only for probes with one of those two ids. That last bit, |format=txt| will ensure that the returning data is in bite-sized chunks so you don’t gobble up 30gb of memory just to parse what comes out. Write that in a loop /with a reasonable wait time between each request or we’ll get mad at you/ and you’ve got a decent body of data to work with. Note that there’s no reason to fetch the entire history of a measurement either. You can always pass |start=<timestamp>| to the above URL and it’ll only give you results returned since that date.
On 26.10.2015 11:47, Daniel Quinn wrote:
Fetch the probe-specific data from the API using both ids:
Oh, I didn't know that it is possible to filter results by probe ID (and/or time). You should add a "result" section to the API docs. That'll make it much easier/faster for me (and the Atlas API) ;-) Thanks! Regards, Sebastian
Actually, all of this is detailed toward the bottom of the REST docs: https://atlas.ripe.net/docs/rest/ But it's not exactly clear. I'll add to our queue-of-things-to-do that we need to clean up the formatting and add a section header there.
participants (2)
-
Daniel Quinn
-
Sebastian Neuner