When to consider a measurement finished
Hi all, I am currently trying to do some measurements according to the following steps: 1. create a one-off measurement 2. wait for it to finish, by periodically polling the msmid and checking the status. 3. fetch the results However, the system takes a pretty long time to set the status to finished, even when all results are already in (maybe 10 minutes?). What heuristics/method do you recommend to determine whether to consider a measurement as finished? One option I consider is to periodically fetch all results for a given msmid and checking if the number of results matches the number of probes assigned to the measurement. However, I suspect that causes a much higher load on the infrastructure. Best regards, Wouter de Vries
Hi Wouter, The status of one-offs is not related to results collected. The stopped status is set by the timer after max 15 minutes. Since the measurement scheduling is based on the best effort approach there cannot be 100% guarantee that all the probes respond with the results so it becomes quite difficult to set the stopped status when all probes finish measuring. Moreover an additional lag can be introduced due to the cluster processing. We saw situations when a measurement had the stopped status but the results were coming. So the workflow would be to start a measurement, allow it to be scheduled and first results collected, then after 5-20 seconds you can start polling it with the same or larger interval. The polling interval should be the longer the more probes you use in your measurement. Additionally you can estimate the data processing delay using the https://atlas.ripe.net/api/v2/system/data-delay/ API call that shows the lag in seconds and the latest received measurement timestamp. If you want it to be truly synchronous you can use the streaming API. In this case the results will be pushed to the client as soon as they are coming from the probes. WBR /vty On 6/18/19 8:08 PM, Wouter de Vries wrote:
Hi all,
I am currently trying to do some measurements according to the following steps:
1. create a one-off measurement 2. wait for it to finish, by periodically polling the msmid and checking the status. 3. fetch the results
However, the system takes a pretty long time to set the status to finished, even when all results are already in (maybe 10 minutes?).
What heuristics/method do you recommend to determine whether to consider a measurement as finished?
One option I consider is to periodically fetch all results for a given msmid and checking if the number of results matches the number of probes assigned to the measurement. However, I suspect that causes a much higher load on the infrastructure.
Best regards,
Wouter de Vries
On Tue, Jun 18, 2019 at 08:08:23PM +0200, Wouter de Vries <w.b.devries@utwente.nl> wrote a message of 26 lines which said:
What heuristics/method do you recommend to determine whether to consider a measurement as finished?
Probing until N % of the requested probes replied. This is implemented in blaeu <https://framagit.org/bortzmeyer/blaeu>, you may check the code <https://framagit.org/bortzmeyer/blaeu/blob/master/Blaeu.py#L466>.
participants (3)
-
Stephane Bortzmeyer
-
Viktor Naumov
-
Wouter de Vries