Hi, I would like to pull data from DNSMON probe measurements to build a real-time view. Considering the combination of IPv4/IPv6, UDP/TCP, multiple nameservers per TLD, it quickly adds up to 150+ measurements. I started with the streaming API, sending register messages for the measurements of interest, but quite quickly I am getting errors with "slow_down" response. I could not find the rate limit values in the documentation of the streaming API. Using the HTTP API would also mean hundreds of API calls, one per measurement. What is the recommended way to regularly pull data from many measurements, to avoid rate limiting or generating too much load on the API ? Regards
Hi, I have increased the maximum number of subscriptions per minute to 512, because it was a bit low at 120. You're right that the rate limit values aren't documented, as they are tweaked now and then based on usage patterns. If you don't mind including measurement results from other running measurements (not just DNSMON), you could alternatively experiment with filtering by "destinationAddress". For example, this subscription would capture *every* result to k-root over IPv4, all in one sub: ["atlas_subscribe",{"streamType":"result","destinationAddress":"193.0.14.129"}] Hopefully one or both of these help you! Regards, Chris Amin RIPE NCC On Sat, 11 May 2024 at 09:11, <techtebatoye@gmail.com> wrote:
Hi,
I would like to pull data from DNSMON probe measurements to build a real-time view.
Considering the combination of IPv4/IPv6, UDP/TCP, multiple nameservers per TLD, it quickly adds up to 150+ measurements.
I started with the streaming API, sending register messages for the measurements of interest, but quite quickly I am getting errors with "slow_down" response. I could not find the rate limit values in the documentation of the streaming API.
Using the HTTP API would also mean hundreds of API calls, one per measurement.
What is the recommended way to regularly pull data from many measurements, to avoid rate limiting or generating too much load on the API ?
Regards
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
participants (2)
-
Christopher Amin
-
techtebatoye@gmail.com