Hello, Some clarifications below. On 2016-08-25 16:51, Max Mühlbronner wrote:
Hi,
i noticed another problem besides the hardware, the "controllers" are not really highly-available? (E.g. ctr-ams07, NL)
They are not individually highly-available, but we have many of them for redundancy. So if some of them go down, only a portion of the network is affected.
Sometimes the probes are disconnected from a RIPE controller for some time even though there is no absolutely no network issue at the probes site. I suppose this could happen from time to time but it's too often. (Maybe it's just me, maybe everyone should check the connection history at RIPE atlas).
Every now and then, the NCC network has a hickup, or scheduled network maintenance. This can break the connections of the probes currently connected to us. We have controllers in other networks too (in Germany, US east/west and Singapore) which of course can have similar issues. As you probably know, probes keep an open TCP (SSH) connection to the controller. If any part of the network between the probe and the controller fails hard enough, there is a disconnection. In other words, probe disconnection can be caused by issues in the probe's network, in ours, or in anything in between :)
Also if a controller is not working sometimes it seems to take forever until the probe will use a failover controller and in the meanwhile its "disconnected"... Any chance to improve the availability?
Generally speaking, probes try hard to get back to the same controller they were connected to before disconnection. If this does not succeed for two hours, then they connect to the reg.servers ("trust anchors" if you will) to ask for a new controller. In the meantime they continue to measure and buffer results -- though they indeed show up as disconnected and therefore can't participate in the newest measurements. We have this mechanism in place to avoid probes jumping around from controller to controller in case there's a glitch somewhere. Hope this clarifies, Robert
Best Regards
Max M.
On 25.08.2016 16:37, Colin Johnston wrote:
maybe virtual vm probes might be useful to investigate and also a split file system for v3 probs with /var/log on a different file system ?
Colin
On 25 Aug 2016, at 15:30, Mirjam Kuehne <mir@ripe.net> wrote:
Dear colleagues,
We took another, more detailed look at probe lifetimes and the dynamics of probes connecting and disconnecting from the RIPE Atlas infrastructure to try to understand how to keep the network growing in the long term.
Kind regards, Mirjam Kuehne RIPE NCC