Emergency RIPE Atlas firmware upgrade
Dear RIPE Atlas users, We want to let you know about a recent issue in the RIPE Atlas system involving an open port on the latest generation (v3) probes. Due to an issue with the latest firmware release, a subset of the v3 probes were listening to incoming connections on an open port that should not have been left open. As a secondary measure, however, access to this port required credentials only available to the RIPE Atlas probe developers. It therefore never presented open access to the probes. This port (SSH) is used for development purposes in our internal development environment. We upgraded the v3 probes to a new, corrected firmware version, and improved the checks in our firmware release process. We would like to thank ZeelandNet for reporting the issue to us. We want to assure you that the security of the RIPE Atlas system is important to us. We are continually improving the controlling infrastructure as well as the probes themselves in order to prevent vulnerabilities. We apologise for this issue and want to stress that no probes were actually affected in any way. You can learn more about the security of the RIPE Atlas system on our dedicated security page at: https://atlas.ripe.net/docs/security/ Thank you for your continuing support of RIPE Atlas. Regards, Robert Kisteleki RIPE NCC
Hi, On Mon, Dec 16, 2013 at 03:11:37PM +0100, Robert Kisteleki wrote:
Due to an issue with the latest firmware release, a subset of the v3 probes were listening to incoming connections on an open port that should not have been left open. As a secondary measure, however, access to this port required credentials only available to the RIPE Atlas probe developers. It therefore never presented open access to the probes. This port (SSH) is used for development purposes in our internal development environment.
... actually, it would be great to have some sort of local "tell me if you are fine?" port to connect to... One of my probes is being difficult (it has network, it connects to the NCC, but it claims to not have an USB stick - which requires an on-site visit) and the lack of basic local diagnostic besides "run tcpdump and see what is happening" has already caused me two on-site visits, and now a third one for "replace the USB stick" (and/or find a USB port with more power). Two of those trips could have been saved by being able to just connect to some sort of status port... Better diagnostics at the atlas dashboard would have helped me, too, but for cases like "I think I have network by my DNS isn't working" or "I think I have network but can't connect to my controllers" errors, local data might still be beneficial. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi Gert,
Better diagnostics at the atlas dashboard would have helped me, too, but for cases like "I think I have network by my DNS isn't working" or "I think I have network but can't connect to my controllers" errors, local data might still be beneficial.
This is reasonable, but it leads to a place where probes *do* run services that are open to "all". I guess it's possible to redefine "all" to be "my local network" or "the same /24 or /64 as the probe" or such, but that adds complexity and I'd be reluctant to go that way. What we're trying to do is what you describe above -- based on signals sent by the probe we're trying to give useful clues to the hosts about what's going on. We can look at the current diagnostics and see if we already squeeze out as much diagnostics as possible. We also plan to include IPv6 PMTU problems, broken local resolvers and such in an upcoming iteration. Regards, Robert
Speaking with my Board hat on at the moment, I do not want to wake up one morning and read the headline "RIPE NCC deploys world's largest botnet" Keep them simple, keep them safe Nigel -----Original Message----- From: ripe-atlas-bounces@ripe.net [mailto:ripe-atlas-bounces@ripe.net] On Behalf Of Robert Kisteleki Sent: 16 December 2013 15:14 To: Gert Doering Cc: ripe-atlas@ripe.net Subject: Re: [atlas] management access...? Hi Gert,
Better diagnostics at the atlas dashboard would have helped me, too, but for cases like "I think I have network by my DNS isn't working" or "I think I have network but can't connect to my controllers" errors, local data might still be beneficial.
This is reasonable, but it leads to a place where probes *do* run services that are open to "all". I guess it's possible to redefine "all" to be "my local network" or "the same /24 or /64 as the probe" or such, but that adds complexity and I'd be reluctant to go that way. What we're trying to do is what you describe above -- based on signals sent by the probe we're trying to give useful clues to the hosts about what's going on. We can look at the current diagnostics and see if we already squeeze out as much diagnostics as possible. We also plan to include IPv6 PMTU problems, broken local resolvers and such in an upcoming iteration. Regards, Robert
I think that's a bit of an exaggeration :) I agree with Gert that probes need better diagnostic capabilities - and tcpdump just doesn't cut it. And probe or not probe, small form factor or not - it's a Linux box. I would like to be able to troubleshoot it like any other *nix box. There IS a middle ground here, I think. The current firmware image could stay as it is - no SSH, no incoming connections. A "full" image could be provided, that would allow: A) starting the SSH daemon B) configuring key-based authentication for SSH, allow importing of public keys C) configuring IP-based restrictions (iptables, or AllowUsers) As none of those would be the default configuration, and would NOT be enable thru the web-based admin interface, they shouldn't have a security impact on anyone that does NOT choose to enable them - but would be available for "power users", so to speak. On 12/16/13 10:23 AM, "Nigel Titley" <nigel.titley@easynet.com> wrote:
Speaking with my Board hat on at the moment, I do not want to wake up one morning and read the headline "RIPE NCC deploys world's largest botnet"
Keep them simple, keep them safe
Nigel -----Original Message----- From: ripe-atlas-bounces@ripe.net [mailto:ripe-atlas-bounces@ripe.net] On Behalf Of Robert Kisteleki Sent: 16 December 2013 15:14 To: Gert Doering Cc: ripe-atlas@ripe.net Subject: Re: [atlas] management access...?
Hi Gert,
Better diagnostics at the atlas dashboard would have helped me, too, but for cases like "I think I have network by my DNS isn't working" or "I think I have network but can't connect to my controllers" errors, local data might still be beneficial.
This is reasonable, but it leads to a place where probes *do* run services that are open to "all". I guess it's possible to redefine "all" to be "my local network" or "the same /24 or /64 as the probe" or such, but that adds complexity and I'd be reluctant to go that way.
What we're trying to do is what you describe above -- based on signals sent by the probe we're trying to give useful clues to the hosts about what's going on. We can look at the current diagnostics and see if we already squeeze out as much diagnostics as possible. We also plan to include IPv6 PMTU problems, broken local resolvers and such in an upcoming iteration.
Regards, Robert
Of course it was an exaggeration.... but think about it.... 5000 identical machines with the same software. An exploit would go through them like a curry and a couple of pints of lager. Your half way house might be acceptable though Nigel -----Original Message----- From: Dario Ciccarone [mailto:dario.ciccarone@gmail.com] Sent: 16 December 2013 15:53 To: Nigel Titley; Robert Kisteleki; Gert Doering Cc: ripe-atlas@ripe.net Subject: Re: [atlas] management access...? I think that's a bit of an exaggeration :) I agree with Gert that probes need better diagnostic capabilities - and tcpdump just doesn't cut it. And probe or not probe, small form factor or not - it's a Linux box. I would like to be able to troubleshoot it like any other *nix box. There IS a middle ground here, I think. The current firmware image could stay as it is - no SSH, no incoming connections. A "full" image could be provided, that would allow: A) starting the SSH daemon B) configuring key-based authentication for SSH, allow importing of public keys C) configuring IP-based restrictions (iptables, or AllowUsers) As none of those would be the default configuration, and would NOT be enable thru the web-based admin interface, they shouldn't have a security impact on anyone that does NOT choose to enable them - but would be available for "power users", so to speak. On 12/16/13 10:23 AM, "Nigel Titley" <nigel.titley@easynet.com> wrote:
Speaking with my Board hat on at the moment, I do not want to wake up one morning and read the headline "RIPE NCC deploys world's largest botnet"
Keep them simple, keep them safe
Nigel -----Original Message----- From: ripe-atlas-bounces@ripe.net [mailto:ripe-atlas-bounces@ripe.net] On Behalf Of Robert Kisteleki Sent: 16 December 2013 15:14 To: Gert Doering Cc: ripe-atlas@ripe.net Subject: Re: [atlas] management access...?
Hi Gert,
Better diagnostics at the atlas dashboard would have helped me, too, but for cases like "I think I have network by my DNS isn't working" or "I think I have network but can't connect to my controllers" errors, local data might still be beneficial.
This is reasonable, but it leads to a place where probes *do* run services that are open to "all". I guess it's possible to redefine "all" to be "my local network" or "the same /24 or /64 as the probe" or such, but that adds complexity and I'd be reluctant to go that way.
What we're trying to do is what you describe above -- based on signals sent by the probe we're trying to give useful clues to the hosts about what's going on. We can look at the current diagnostics and see if we already squeeze out as much diagnostics as possible. We also plan to include IPv6 PMTU problems, broken local resolvers and such in an upcoming iteration.
Regards, Robert
I indeed agree that they would be a tempting target - IPv4 and IPv6 both. However, they're low powered devices - DDoS sources, maybe - amplification attacks. Don't see them doing any BitCoin mining any time soon :) But yes, depending on where on the host network they sit, they could be used as stepping stones . . . A good reason for people deploying a probe to understand the inherent risks of deploying it. And an opportunity for the Atlas community to share best practices/deployment scenarios. My probe sits on a DMZ, w/o access to the internal network. And I haven't implemented rate-limiting for it - but come to think of it, it may be a good idea to do so - limit the probe to 512Kbps or less. On 12/16/13 10:58 AM, "Nigel Titley" <nigel.titley@easynet.com> wrote:
Of course it was an exaggeration.... but think about it.... 5000 identical machines with the same software. An exploit would go through them like a curry and a couple of pints of lager.
Your half way house might be acceptable though
Nigel
-----Original Message----- From: Dario Ciccarone [mailto:dario.ciccarone@gmail.com] Sent: 16 December 2013 15:53 To: Nigel Titley; Robert Kisteleki; Gert Doering Cc: ripe-atlas@ripe.net Subject: Re: [atlas] management access...?
I think that's a bit of an exaggeration :)
I agree with Gert that probes need better diagnostic capabilities - and tcpdump just doesn't cut it. And probe or not probe, small form factor or not - it's a Linux box. I would like to be able to troubleshoot it like any other *nix box.
There IS a middle ground here, I think. The current firmware image could stay as it is - no SSH, no incoming connections. A "full" image could be provided, that would allow:
A) starting the SSH daemon B) configuring key-based authentication for SSH, allow importing of public keys C) configuring IP-based restrictions (iptables, or AllowUsers)
As none of those would be the default configuration, and would NOT be enable thru the web-based admin interface, they shouldn't have a security impact on anyone that does NOT choose to enable them - but would be available for "power users", so to speak.
On 12/16/13 10:23 AM, "Nigel Titley" <nigel.titley@easynet.com> wrote:
Speaking with my Board hat on at the moment, I do not want to wake up one morning and read the headline "RIPE NCC deploys world's largest botnet"
Keep them simple, keep them safe
Nigel -----Original Message----- From: ripe-atlas-bounces@ripe.net [mailto:ripe-atlas-bounces@ripe.net] On Behalf Of Robert Kisteleki Sent: 16 December 2013 15:14 To: Gert Doering Cc: ripe-atlas@ripe.net Subject: Re: [atlas] management access...?
Hi Gert,
Better diagnostics at the atlas dashboard would have helped me, too, but for cases like "I think I have network by my DNS isn't working" or "I think I have network but can't connect to my controllers" errors, local data might still be beneficial.
This is reasonable, but it leads to a place where probes *do* run services that are open to "all". I guess it's possible to redefine "all" to be "my local network" or "the same /24 or /64 as the probe" or such, but that adds complexity and I'd be reluctant to go that way.
What we're trying to do is what you describe above -- based on signals sent by the probe we're trying to give useful clues to the hosts about what's going on. We can look at the current diagnostics and see if we already squeeze out as much diagnostics as possible. We also plan to include IPv6 PMTU problems, broken local resolvers and such in an upcoming iteration.
Regards, Robert
C'mon, Nigel - such an event every now and then would just keep you and your esteemed board collegues alert and prevent you from feeling bored. ;-b On 16.12.2013 16:23, Nigel Titley wrote:
Speaking with my Board hat on at the moment, I do not want to wake up one morning and read the headline "RIPE NCC deploys world's largest botnet"
Boredom is good..... we like boredom -----Original Message----- From: Carsten Schiefner [mailto:carsten@schiefner.de] Sent: 16 December 2013 15:58 To: Nigel Titley Cc: ripe-atlas@ripe.net Subject: Re: [atlas] management access...? C'mon, Nigel - such an event every now and then would just keep you and your esteemed board collegues alert and prevent you from feeling bored. ;-b On 16.12.2013 16:23, Nigel Titley wrote:
Speaking with my Board hat on at the moment, I do not want to wake up one morning and read the headline "RIPE NCC deploys world's largest botnet"
On 16/12/2013 15:59, Nigel Titley wrote:
Boredom is good..... we like boredom
"chairman of the bored". Nick
I wonder why no one else suggested this before. As v3 probes have physical switch, it could be used to turn on some debugging mode temporarily... or what?
On 2013/12/17 0:20 , Sulev-Madis Silber wrote:
I wonder why no one else suggested this before. As v3 probes have physical switch, it could be used to turn on some debugging mode temporarily... or what?
Maybe it is good to take a step backward and focus on what should be reported instead of how it should be reported. On v3 probes, the LEDs blink in various patterns to indicate what is going on. As soon as probes have any kind of network connectivity they start reporting what they are doing. Now, it is quite possible that some of that only makes it to our logs and not to the user interface. But in that case it would probably more effective to update the user interface rather than to create alternative mechanisms of reporting status information. If anyone has stories on what the probe didn't report, then I'd like to hear them. Philip
Hi list, Dne 17.12.2013 10:09, Philip Homburg napsal(a):
If anyone has stories on what the probe didn't report, then I'd like to hear them.
My probe had perfect network connectivity, however it somehow turned its status into „disconnected“ and then „abandoned“, although it still responded to ICMP(v6) ping. Visiting it personally, I found nothing wrong, except the LEDs was blinking in the pattern „running from internal memory“. Long story short, after few round trips with Atlas support, I found out that the probe may be repaired by unplugging the USB flash drive, overwritting it with zeros and plugging it back in. It would be really nice if this state of probe was displayed somewhere, because the probe was certainly able to talk to the controller. Personally, I would prefer simple status webpage on the probe itself. Read-only access without authentication would suit this purpose perfectly without much extra complexity or security issues. Regards, Ondřej Caletka
On 2013/12/17 13:05 , Ondřej Caletka wrote:
My probe had perfect network connectivity, however it somehow turned its status into „disconnected“ and then „abandoned“, although it still responded to ICMP(v6) ping. Visiting it personally, I found nothing wrong, except the LEDs was blinking in the pattern „running from internal memory“. Long story short, after few round trips with Atlas support, I found out that the probe may be repaired by unplugging the USB flash drive, overwritting it with zeros and plugging it back in.
It would be really nice if this state of probe was displayed somewhere, because the probe was certainly able to talk to the controller. Personally, I would prefer simple status webpage on the probe itself. Read-only access without authentication would suit this purpose perfectly without much extra complexity or security issues.
In this case, you should have had a couple of emails from the Atlas system that your probe was not connected. I think the first one is sent when the probe is not connected for two weeks. This is a situation where the user interface has to be updated. The probe did report that it could not find its USB stick. But that information is not shown to the probe host. Unfortunately, in situations with marginal USB power supplies, probe behavior can become weird. Like overwriting the USB stick is not supposed to have any effect. But in this case it did. Philip
On 12/16/2013 at 3:59 PM Nigel Titley wrote: |Boredom is good..... we like boredom ============= Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction. --- Albert Einstein
Hi, On Mon, Dec 16, 2013 at 04:13:37PM +0100, Robert Kisteleki wrote:
What we're trying to do is what you describe above -- based on signals sent by the probe we're trying to give useful clues to the hosts about what's going on. We can look at the current diagnostics and see if we already squeeze out as much diagnostics as possible. We also plan to include IPv6 PMTU problems, broken local resolvers and such in an upcoming iteration.
Well, right now the probe in question is listed in the dashboard as "abandoned", while it does send a panic DNS query every few minutes... Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi, On Mon, Dec 16, 2013 at 04:13:37PM +0100, Robert Kisteleki wrote:
Better diagnostics at the atlas dashboard would have helped me, too, but for cases like "I think I have network by my DNS isn't working" or "I think I have network but can't connect to my controllers" errors, local data might still be beneficial.
This is reasonable, but it leads to a place where probes *do* run services that are open to "all". I guess it's possible to redefine "all" to be "my local network" or "the same /24 or /64 as the probe" or such, but that adds complexity and I'd be reluctant to go that way.
I can understand that, though. I wouldn't want an SSH access to it, more something what the old TTM boxes had - a port where you can telnet to, which would completely ignore whatever you send to it, but which would spit out some sort of one-line health status. Limited to "the local network" (which is not *that* hard to figure out). Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
participants (10)
-
Carsten Schiefner -
Dario Ciccarone -
Gert Doering -
Mike. -
Nick Hilliard -
Nigel Titley -
Ondřej Caletka -
Philip Homburg -
Robert Kisteleki -
Sulev-Madis Silber