Rate limiting on the APIs
Dear all, We have seen in the past that even with the best intentions, every now and then some users are using excessive amounts of requests to the RIPE Atlas APIs, in some cases resulting in degraded performance to all users. An example for the curious: some AWS account asking the probe API for live queries once a day at 8am like clockwork; over a million queries for probe statuses (note: we don't have a million probes - every query is repeated hundreds or thousands of times) [1] We will apply rate limiting to protect from (some of) these. In particular: * The API in general will impose a limit of no more than 300 queries/second (if done over multiple connections) * The measurement API in particular will impose a limit of 150 queries/second (if done over multiple connections) * When reaching the limit, an HTTP response code of 429 (Too Many Requests; https://www.rfc-editor.org/rfc/rfc6585#section-4) will be given. According to our logs these limits will in general not be visible to users. Users going over these limits will see the 429 response code; client libraries should be able to handle this. We will observe the effect of these measures to determine if we need to tighten further - or perhaps relax them. We plan to apply these changes sometime next week; I'll let you know once they are in place. Regards, Robert [1] a much better way of doing this is to use the probe archive (https://ftp.ripe.net/ripe/atlas/probes/archive/); this is one-time fetch of 2MB or so and probe statuses are pretty stable in general
Update: we applied this change a few minutes ago. We'll keep observing the system for ill effects. Regards, Robert On 2023-01-30 14:57, Robert Kisteleki wrote:
Dear all,
We have seen in the past that even with the best intentions, every now and then some users are using excessive amounts of requests to the RIPE Atlas APIs, in some cases resulting in degraded performance to all users. An example for the curious: some AWS account asking the probe API for live queries once a day at 8am like clockwork; over a million queries for probe statuses (note: we don't have a million probes - every query is repeated hundreds or thousands of times) [1]
We will apply rate limiting to protect from (some of) these. In particular:
* The API in general will impose a limit of no more than 300 queries/second (if done over multiple connections)
* The measurement API in particular will impose a limit of 150 queries/second (if done over multiple connections)
* When reaching the limit, an HTTP response code of 429 (Too Many Requests; https://www.rfc-editor.org/rfc/rfc6585#section-4) will be given.
According to our logs these limits will in general not be visible to users. Users going over these limits will see the 429 response code; client libraries should be able to handle this. We will observe the effect of these measures to determine if we need to tighten further - or perhaps relax them.
We plan to apply these changes sometime next week; I'll let you know once they are in place.
Regards, Robert
[1] a much better way of doing this is to use the probe archive (https://ftp.ripe.net/ripe/atlas/probes/archive/); this is one-time fetch of 2MB or so and probe statuses are pretty stable in general
participants (1)
-
Robert Kisteleki