Encouraging people to upgrade software probe versions
Hi, I recently posted two threads ultimately related to the same question, where it turned out that a feature I wanted to use was present in the current software probe release, but not in the old version that I had installed. It was indeed relatively easy to upgrade (as a reply on this list predicted, the upgrade didn't break anything or require any reconfiguration). But I'd still say "relatively easy", because, on the other hand, the upgrade wasn't managed through my OS package manager like the rest of the software on my VPS server. It appears to me that a lot of software probes (like mine was until recently!) are still running whichever software probe version was current when they were first deployed. After all, I don't believe there's automatic software upgrade functionality in the methods that most people are using to deploy these. Is that true from the point of view of the rest of the Atlas community? Is there anything that might usefully be done to encourage people to upgrade periodically? (For example, either sending some kind of e-mail reminder, or distributing probe software via OS package managers, or even including a mechanism in the probe client itself by which it could upgrade itself -- even if as an opt-in feature?)
Hi Seth, I have an older probe, and I wasn't aware I needed to update its software. How do we do that? Best, -Michael On Sat, Jan 7, 2023 at 6:03 AM Seth David Schoen <schoen@loyalty.org> wrote:
Hi,
I recently posted two threads ultimately related to the same question, where it turned out that a feature I wanted to use was present in the current software probe release, but not in the old version that I had installed. It was indeed relatively easy to upgrade (as a reply on this list predicted, the upgrade didn't break anything or require any reconfiguration).
But I'd still say "relatively easy", because, on the other hand, the upgrade wasn't managed through my OS package manager like the rest of the software on my VPS server.
It appears to me that a lot of software probes (like mine was until recently!) are still running whichever software probe version was current when they were first deployed. After all, I don't believe there's automatic software upgrade functionality in the methods that most people are using to deploy these.
Is that true from the point of view of the rest of the Atlas community? Is there anything that might usefully be done to encourage people to upgrade periodically? (For example, either sending some kind of e-mail reminder, or distributing probe software via OS package managers, or even including a mechanism in the probe client itself by which it could upgrade itself -- even if as an opt-in feature?)
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
Hi, ...on 2023-01-07 09:15:33, Michael J. Oghia wrote:
I have an older probe, and I wasn't aware I needed to update its software. How do we do that?
The howto on software probes(*) sais:
As a (future) host of a RIPE Atlas software probe you are expected to: [..] Keep the version of your software up-to-date by upgrading to newer versions as they become available
When using the CentOS binary repo as recommended, the software should auto-update during normal system update procedures. For source releases (Debian), you're more or less on your own. Updating is still easy - just go through the initial installation again, minus registration. What's not easy (for me, at least) is to find out when a new software probe version has been made available. Releases aren't being tagged on the repo, and the version tags seem inconsistent? The current history shows 5081 at the top, and then 5082, 5083, 5080? The version I last checked out apparently was 5080, but I have no idea if I should now try to update or not. Having some kind of announcement would be great. Alex. (*) https://atlas.ripe.net/docs/howtos/software-probes.html#good-to-know
Alexander Bochmann writes:
When using the CentOS binary repo as recommended, the software should auto-update during normal system update procedures.
For source releases (Debian), you're more or less on your own. Updating is still easy - just go through the initial installation again, minus registration.
I see, so that's a pretty significant difference in the experiences of people using different systems. Is it clear how many probe users use each of these methods? (which might indicate whether it could be worth maintaining a more automated installation method for Debian-based systems)
Hi guys, The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies. As to updating the package, we are looking to see if it is possible to make an RPM that is part of the base RedHat repository, together with a packager. We could do something similar for Debian (possibly others, such as OpenWRT). I’m not sure what you mean with methods: How many probe users are using auto-update is a bit difficult to answer, since right now it is forced, but as of the next software release it will not be. Which operating systems are used is also a bit difficult since we do not provide an official Debian package yet, and the software does not differentiate between different flavours of Linux. Hope this helps. Regards, Michel
On 8 Jan 2023, at 01:36, Seth David Schoen <schoen@loyalty.org> wrote:
Alexander Bochmann writes:
When using the CentOS binary repo as recommended, the software should auto-update during normal system update procedures.
For source releases (Debian), you're more or less on your own. Updating is still easy - just go through the initial installation again, minus registration.
I see, so that's a pretty significant difference in the experiences of people using different systems.
Is it clear how many probe users use each of these methods? (which might indicate whether it could be worth maintaining a more automated installation method for Debian-based systems)
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
Hi, On Mon, Jan 09, 2023 at 10:19:34AM +0100, Michel Stam wrote:
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
I find this surprising, to say the least. The whole point about ATLAS Probes is "RIPE NCC manages them" - as host of a handful of hardware probes, I have no say in whether I want them upgraded or not, or what they should do or not do. So I would expect all software probes to always upgrade themselves, and this be clearly communicated to SW probe hosts. If this violates your sysadmin policies, you shouldn't run 3rd-party maintained *and operated* software. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi Gert, Thank you for pitching in. The hardware probes are fully managed by RIPE NCC, and their only purpose is to execute measurements. I agree here, whether to update is up to RIPE NCC. On software probes, RIPE NCC is not in control of the unit. The software probe may run other services and updating is up to its local sysadmin. Consider operating system updates, which are updated by RIPE NCC on the hardware probes, but cannot be updated on software probes. To automatically update, the solution here is a simple CRON job which executes yum -q update atlasswprobe periodically (say once per day or so), which was the solution that was part of the RPM up to 5080. Would it help if we add this to the instructions? Regards, Michel
On 9 Jan 2023, at 10:52, Gert Doering <gert@space.net> wrote:
Hi,
On Mon, Jan 09, 2023 at 10:19:34AM +0100, Michel Stam wrote:
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
I find this surprising, to say the least.
The whole point about ATLAS Probes is "RIPE NCC manages them" - as host of a handful of hardware probes, I have no say in whether I want them upgraded or not, or what they should do or not do.
So I would expect all software probes to always upgrade themselves, and this be clearly communicated to SW probe hosts.
If this violates your sysadmin policies, you shouldn't run 3rd-party maintained *and operated* software.
Gert Doering -- NetMaster -- have you enabled IPv6 on something today...?
SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi, On Mon, Jan 09, 2023 at 11:25:50AM +0100, Michel Stam wrote:
Thank you for pitching in.
The hardware probes are fully managed by RIPE NCC, and their only purpose is to execute measurements. I agree here, whether to update is up to RIPE NCC. On software probes, RIPE NCC is not in control of the unit. The software probe may run other services and updating is up to its local sysadmin. Consider operating system updates, which are updated by RIPE NCC on the hardware probes, but cannot be updated on software probes.
So don't do OS updates. But *do* update the probe software, automatically, and in normal intervals. Having probes out there with wildly varying firmware is not helping measurement accuracy.
To automatically update, the solution here is a simple CRON job which executes yum -q update atlasswprobe periodically (say once per day or so), which was the solution that was part of the RPM up to 5080. Would it help if we add this to the instructions?
I'm very much of the opinion that upgrades should not be an opt-in service, *and not even an opt-out*. If you want a measurement box that does something non-ATLAS based on ATLAS software - all good, but do not call it an ATLAS probe. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
...on 2023-01-09 11:37:25, Gert Doering wrote:
Having probes out there with wildly varying firmware is not helping measurement accuracy.
I don't think that's directly comparable to hardware probes. For example, there's a software probe running in a VM behind my residential DSL line. The availability of the tags "VDSL2" and "Homelab" (for example) made that seem like not a completely absurd setup. I was about to spawn a couple of software probes as VMs in various places at work. (I could probably request hardware probes, but VMs are much less of a hassle in terms of internal management processes.) Still, even the latter will probably be less accurate than hardware probes, being a (possibly low-priority) virtualized workload. As for updates, I would really like to have that automated away, preferably out of the box. When I installed the first software probe, I for some reason didn't notice there were RPM packages available, and went with the manual install on my preferred distribution. Now I know there's the RPMs and I can have them updated, I'll go that way with the next setup. I don't remember how I noticed there was an update last time - maybe some warning in the portal can act as a reminder when a probe is out of date? Would it be useful to package the probe as something like a Docker container, even though that's yet more indirection in terms of network access? I have the impression that nowadays more people are prepared to manage that kind of thing instead of a distribution package... Alex.
I fully agree with Gert. There's nothing to add to his statement, except that we could consider to opt-out automatic updates? A small switch like RXTXRPT. But automatic updates set to enabled should be the default setting. BR, Simon On 09.01.23 10:52, Gert Doering wrote:
Hi,
On Mon, Jan 09, 2023 at 10:19:34AM +0100, Michel Stam wrote:
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies. I find this surprising, to say the least.
The whole point about ATLAS Probes is "RIPE NCC manages them" - as host of a handful of hardware probes, I have no say in whether I want them upgraded or not, or what they should do or not do.
So I would expect all software probes to always upgrade themselves, and this be clearly communicated to SW probe hosts.
If this violates your sysadmin policies, you shouldn't run 3rd-party maintained *and operated* software.
Gert Doering -- NetMaster
Hello, On 2023-01-09 10:52, Gert Doering wrote:
Hi,
On Mon, Jan 09, 2023 at 10:19:34AM +0100, Michel Stam wrote:
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
I find this surprising, to say the least.
The whole point about ATLAS Probes is "RIPE NCC manages them" - as host of a handful of hardware probes, I have no say in whether I want them upgraded or not, or what they should do or not do.
That is indeed true for hardware probes - the host has little influence on the firmware upgrade. It is intended to be plug and play. It is also true for anchors.
So I would expect all software probes to always upgrade themselves, and this be clearly communicated to SW probe hosts.
That is reasonable; the difference is that we are not in control; the host OS is. Redhat/Fedora/derivatives as well as Debian/derivatives have an official solution to this via their package management services and I believe this is the standard way (surely with exceptions :-) ) of handling these matters. We are in the process of adopting these.
If this violates your sysadmin policies, you shouldn't run 3rd-party maintained *and operated* software.
The intention here is to do less special handling, and conform more to the operational practices. The case is entirely solvable using those, and this direction is expected to make deployment and maintenance easier on hosts (as well as the Atlas team) in the long run. I hope this explains the motivation, Robert
Gert Doering -- NetMaster
Hi Robert, The existence of software probes is great, but instead (or besides) of providing packages or source code, why not distribute a prebuild VM as OVF file? Advantages: - The RIPE Atlas team manages the whole OS, like it's doing for the hardware probes. Thus, updates can be deployed whenever needed. - You can even use OpenWRT as VM operatingsystem. This means all the same premises/conditions as for hardware probes. - an OVF file is easier to deploy, for the community - RXTXRPT switch is obsolete - No more false RXTXRPT data, since the report counts all traffic of the host, not only the traffic that is generated by the SW probe application. Is there an actual reason, why it was decided to let users manage the software probe installation? Please consider to distribute a prebuild VM *additionally* to the existing ways and see what happens. I'm sure, most new users will choose a prebuild VM. BR, Simon On 19.01.23 12:48, Robert Kisteleki wrote:
That is reasonable; the difference is that we are not in control; the host OS is. Redhat/Fedora/derivatives as well as Debian/derivatives have an official solution to this via their package management services and I believe this is the standard way (surely with exceptions :-) ) of handling these matters. We are in the process of adopting these.
Hi, To me it seems that there are oh-so-many ways of packaging and distributing this software to the match the multitude of needs (RPMs, DEBs, openwrt, docker, VMs, ...) and us giving support to multitude of these stretches our resources thin. As I wrote before, such packaging would be preferably achieved via professional maintainers.
Is there an actual reason, why it was decided to let users manage the software probe installation?
The intention here is/was that many users already have their own machine (VM or server or home router or such) that can be used as the platform. One can also easily spin up and dedicate a new HW, like a lingering Raspberry Pi, to this. Cheers, Robert On 2023-01-27 10:43, Simon Brandt via ripe-atlas wrote:
Hi Robert,
The existence of software probes is great, but instead (or besides) of providing packages or source code, why not distribute a prebuild VM as OVF file?
Advantages: - The RIPE Atlas team manages the whole OS, like it's doing for the hardware probes. Thus, updates can be deployed whenever needed. - You can even use OpenWRT as VM operatingsystem. This means all the same premises/conditions as for hardware probes. - an OVF file is easier to deploy, for the community - RXTXRPT switch is obsolete - No more false RXTXRPT data, since the report counts all traffic of the host, not only the traffic that is generated by the SW probe application.
Is there an actual reason, why it was decided to let users manage the software probe installation? Please consider to distribute a prebuild VM *additionally* to the existing ways and see what happens. I'm sure, most new users will choose a prebuild VM.
BR, Simon
On 19.01.23 12:48, Robert Kisteleki wrote:
That is reasonable; the difference is that we are not in control; the host OS is. Redhat/Fedora/derivatives as well as Debian/derivatives have an official solution to this via their package management services and I believe this is the standard way (surely with exceptions :-) ) of handling these matters. We are in the process of adopting these.
On Mon, 30 Jan 2023 at 13:34, Robert Kisteleki <robert@ripe.net> wrote:
Hi,
To me it seems that there are oh-so-many ways of packaging and distributing this software to the match the multitude of needs (RPMs, DEBs, openwrt, docker, VMs, ...) and us giving support to multitude of these stretches our resources thin.
Which is why a unified approach is needed. Drop all SW support and do dedicated VMs only for SW probes? Drop all SW support and do Docker only for SW probes? I don't think doing everything with questionable quality is a desired outcome. What's worse? A lot of outdated, unhandled probes without upgrade instructions or a slight decrease in the number of probes?
As I wrote before, such packaging would be preferably achieved via professional maintainers.
This is really not that realistic. Debian and Red Hat repositories are not all designed for this: they don't update their packages in stable releases, they backport changes eligible based on their backporting policy, which doesn't address this problem at all. Vendoring (shipping your own libssl for example) is also not allowed at least in Debian, I doubt it is in RedHat. The "number of probes" can't be the only benchmark here, we need to benchmark "the number of probes properly handled running supported software". Lukas
Is there an actual reason, why it was decided to let users manage the software probe installation?
The intention here is/was that many users already have their own machine (VM or server or home router or such) that can be used as the platform. One can also easily spin up and dedicate a new HW, like a lingering Raspberry Pi, to this.
Cheers, Robert
On 2023-01-27 10:43, Simon Brandt via ripe-atlas wrote:
Hi Robert,
The existence of software probes is great, but instead (or besides) of providing packages or source code, why not distribute a prebuild VM as OVF file?
Advantages: - The RIPE Atlas team manages the whole OS, like it's doing for the hardware probes. Thus, updates can be deployed whenever needed. - You can even use OpenWRT as VM operatingsystem. This means all the same premises/conditions as for hardware probes. - an OVF file is easier to deploy, for the community - RXTXRPT switch is obsolete - No more false RXTXRPT data, since the report counts all traffic of the host, not only the traffic that is generated by the SW probe application.
Is there an actual reason, why it was decided to let users manage the software probe installation? Please consider to distribute a prebuild VM *additionally* to the existing ways and see what happens. I'm sure, most new users will choose a prebuild VM.
BR, Simon
On 19.01.23 12:48, Robert Kisteleki wrote:
That is reasonable; the difference is that we are not in control; the host OS is. Redhat/Fedora/derivatives as well as Debian/derivatives have an official solution to this via their package management services and I believe this is the standard way (surely with exceptions :-) ) of handling these matters. We are in the process of adopting these.
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
Lukas, Keep in mind that - as I experienced - any VM or Docker container introduces some problems, such as higher latency. My CentOS VM has 2 ms. higher latency in the first hop compared to a v5 probe on the same fiber. And a bug in Docker causes traceroute to fail. So a hardware or a native install of a software probe is the best way to go. Regards, Ernst J. Oud
On 30 Jan 2023, at 14:07, Lukas Tribus <lukas@ltri.eu> wrote:
On Mon, 30 Jan 2023 at 13:34, Robert Kisteleki <robert@ripe.net> wrote:
Hi,
To me it seems that there are oh-so-many ways of packaging and distributing this software to the match the multitude of needs (RPMs, DEBs, openwrt, docker, VMs, ...) and us giving support to multitude of these stretches our resources thin.
Which is why a unified approach is needed.
Drop all SW support and do dedicated VMs only for SW probes? Drop all SW support and do Docker only for SW probes?
I don't think doing everything with questionable quality is a desired outcome.
What's worse? A lot of outdated, unhandled probes without upgrade instructions or a slight decrease in the number of probes?
As I wrote before, such packaging would be preferably achieved via professional maintainers.
This is really not that realistic.
Debian and Red Hat repositories are not all designed for this: they don't update their packages in stable releases, they backport changes eligible based on their backporting policy, which doesn't address this problem at all. Vendoring (shipping your own libssl for example) is also not allowed at least in Debian, I doubt it is in RedHat.
The "number of probes" can't be the only benchmark here, we need to benchmark "the number of probes properly handled running supported software".
Lukas
Is there an actual reason, why it was decided to let users manage the software probe installation?
The intention here is/was that many users already have their own machine (VM or server or home router or such) that can be used as the platform. One can also easily spin up and dedicate a new HW, like a lingering Raspberry Pi, to this.
Cheers, Robert
On 2023-01-27 10:43, Simon Brandt via ripe-atlas wrote: Hi Robert,
The existence of software probes is great, but instead (or besides) of providing packages or source code, why not distribute a prebuild VM as OVF file?
Advantages: - The RIPE Atlas team manages the whole OS, like it's doing for the hardware probes. Thus, updates can be deployed whenever needed. - You can even use OpenWRT as VM operatingsystem. This means all the same premises/conditions as for hardware probes. - an OVF file is easier to deploy, for the community - RXTXRPT switch is obsolete - No more false RXTXRPT data, since the report counts all traffic of the host, not only the traffic that is generated by the SW probe application.
Is there an actual reason, why it was decided to let users manage the software probe installation? Please consider to distribute a prebuild VM *additionally* to the existing ways and see what happens. I'm sure, most new users will choose a prebuild VM.
BR, Simon
On 19.01.23 12:48, Robert Kisteleki wrote:
That is reasonable; the difference is that we are not in control; the host OS is. Redhat/Fedora/derivatives as well as Debian/derivatives have an official solution to this via their package management services and I believe this is the standard way (surely with exceptions :-) ) of handling these matters. We are in the process of adopting these.
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
On Mon, 30 Jan 2023 at 17:34, Ernst J. Oud <ernstoud@gmail.com> wrote:
Lukas,
Keep in mind that - as I experienced - any VM or Docker container introduces some problems, such as higher latency. My CentOS VM has 2 ms. higher latency in the first hop compared to a v5 probe on the same fiber.
Any virtualization may introduce variables, any WAN circuit, any Firewall and any NAT device. Whether the ATLAS SW probe runs as a "native install" on a virtualized VM managed by you, or whether the ATLAS SW probe runs in a VM managed by RIPE doesn't make a difference in your latency deviation example. We need to stomach 2 ms of deviation if we want to allow SW probes, the alternative is to go HW only.
And a bug in Docker causes traceroute to fail.
Docker is probably not the way to go, containers make it more easy to install, but I don't think a container can trigger its own upgrade.
So a hardware or a native install of a software probe is the best way to go.
I disagree, unmaintained and unsupported probe installation can never be the way to go. Lukas
Hello Lukas, On Mon, 30 Jan 2023, Lukas Tribus wrote:
Which is why a unified approach is needed.
Drop all SW support and do dedicated VMs only for SW probes? Drop all SW support and do Docker only for SW probes?
I don't think doing everything with questionable quality is a desired outcome.
similar like with RPM packages for software probes, Docker/OCI container based software probes have issues as well, which might lead to questionable quality: The typical standalone Docker (or Podman) setup does not update any container images. And while performing some runtime-based updates inside the container might work (IMHO only partially, less on long-term), it has to be repeated after host reboots and/or Docker/Podman updates, if nobody updates the container image on the host itself. This also requires, for runtime-based updates inside the container, an endless upgrade path from oldest to latest versions be kept on an update server, otherwise you could end up with a poorly outdated (or defunct) container-based software probe, too. From my point of view, any non-hardware probe might lead to questionable quality, if the administrator doesn't maintain its environment properly. But maybe that's just an obligation for software probes that needs to be accepted? Because even administrators of hardware probes have obligations: Connect the probe (and, at least historically, handle the USB flash drive issues after some time).
Debian and Red Hat repositories are not all designed for this: they don't update their packages in stable releases, they backport changes eligible based on their backporting policy, which doesn't address this problem at all. Vendoring (shipping your own libssl for example) is also not allowed at least in Debian, I doubt it is in RedHat.
As the Fedora and EPEL packager who came up with the idea to finally ship a policy-compliant RPM of the current software probe RPM in Fedora and EPEL repositories, I would like to clarify that: The software probe would not be in the official RHEL repositories itself (would require coordination of updates with Red Hat regarding release cycles, freezes etc., there can be rebases instead of only backports, usually in the first 5 of 10 years of the RHEL lifecycle), but in Fedora and EPEL repositories. EPEL is a Fedora- driven add-on repository, commonly used, which IMHO adds the real value to RHEL/clones. Updates to EPEL packages can also happen at any time, updates are landing after 7 days in testing repository in stable repository, if no negative karma has been added (practically it's 1-2 days more, depending on when the update was exactly submitted and the signing/pushing cronjobs are run). EPEL permits package updates/rebases during the full RHEL lifecycle with the expectation of compatibility, but that's anyway targeted for probes, especially for the hardware probes. A realworld example for package updates/rebases is OpenBSD rpki-client, which I walked over time from 0.2.0 to (currently) 8.2 in EPEL 7. Similar is BIRD, from 2.0.2 to 2.0.12 in the last 5 years in EPEL 7. While these packages are mostly trivial, there are also soname version bumps (changed ABIs) with coordinated rebuilds of dependent packages. Vendoring in EPEL is not excluded, but also not liked (sometimes it's just needed; Red Hat's security team also nicely files bug reports for security flaws on a best effort base for these bundled libraries, requires however packagers to mark such bundled libraries according to the packaging policy properly). And to catch up your specific libssl example: I am maintaining OpenSSL 1.1 (taken from RHEL 8) in EPEL 7 to provide e.g. TLSv1.3 support to other RPM packages in EPEL. There is also OpenSSL 3 (taken from RHEL 9) in EPEL 8 to provide OpenSSL 3 APIs to other RPM packages in EPEL (this works because RHEL 7 reaches EOL before RHEL 8, RHEL 8 reaches EOL before RHEL 9). I am maintaining it in EPEL 7 for about 3 years now, because OpenBSD rpki-client requires OpenSSL >= 1.1 rather than OpenSSL 1.0.x as shipped in RHEL 7. I can not speak for Debian, but as a system administrator also using Debian here and there...their releases and updates are IMHO not that flexible and agile. Last but not least: An RPM package of a software probe could IMHO allow a virtual machine image preconfigured for DHCP/SLAAC etc. with auto-updates from a RHEL clone. Should reduce the administrator's maintenance work quite a lot, no? Finally: I see usecases for hardware probes and software probes (and there for at least RPM packages, containers and virtual machines). And it leaves some responsibilities to their administrators depending on their decision of the type - even for hardware probes. Regards, Robert
...on 2023-01-30 13:34:18, Robert Kisteleki wrote:
Is there an actual reason, why it was decided to let users manage the software probe installation? The intention here is/was that many users already have their own machine (VM or server or home router or such) that can be used as the platform. One can also easily spin up and dedicate a new HW, like a lingering Raspberry Pi, to this.
I think the Software Probe is a good idea, as it lowers the entry bar to running a probe considerably. Nevertheless I would second the suggestion of providing a "virtual hardware" probe as virtual machine template, using the same OS and software stack as the current hardware probes. Yes, any virtualization will introduce some additional noise into the measurements... But looking at our datacenters - other than network infrastructure components, a few firewalls, and a pair of NTP appliances, everything is running on a hypervisor. Whatever might be a target for measurements (other than the Anchors), will probably be virtualized in some form too. Alex.
Hi Gert, On Mon, 09 Jan 2023, Gert Doering wrote:
So don't do OS updates. But *do* update the probe software, automatically, and in normal intervals.
writing while wearing my hat as a RPM packager: I'm not really sure how this shall work practically: If the probe software is run on regular (non- hardware probe) systems, there are runtime dependencies to other software, e.g. due to dynamic linking. Latter is encouraged by Linux distributions, but might lead to the risk that a software probe update either installs a new dependency (installs a new package) or updates a dependency due to e.g. a ABI soversion bump. And an updated package can cause new issues to other packages or software, especially on a poorly to not maintained system (I assume here that a system that doesn't get software probe updates doesn't get much updates at all). In the end,
I'm very much of the opinion that upgrades should not be an opt-in service, *and not even an opt-out*.
As much as I would like to see an always up-to-date software probe, I can not think of a way conform to e.g. Fedora/EPEL packaging policies. Updates of a package can not be enforced or triggered by the package itself. Other ways would be user-written cronjobs/systemd.timers or administrators using yum-updatesd/dnf-automatic - but that's nothing a RPM package in Fedora/ EPEL is allowed to influence/configure/change during its own installation. My suggestion is to provide documentation how system administrators who set up software probes can easily configure automatic system updates. But these are practically often disliked by administrators (from my point of view). Regards, Robert
Hi, On Fri, Jan 27, 2023 at 02:15:44AM +0100, Robert Scheck wrote:
On Mon, 09 Jan 2023, Gert Doering wrote:
So don't do OS updates. But *do* update the probe software, automatically, and in normal intervals.
writing while wearing my hat as a RPM packager: I'm not really sure how this shall work practically: If the probe software is run on regular (non- hardware probe) systems, there are runtime dependencies to other software, e.g. due to dynamic linking. Latter is encouraged by Linux distributions, [..]
This is all good and true - so, do not distribute as RPM. Distribute as Docker image, or Flatpack, or whatever all these new "encapsulate a program and all its dependencies" approaches are called, and I'm sure some of them have mechanisms to enforce timely updates. First and foremost, a RIPE Atlas probe is a centrally managed piece of software, and this is what it needs to be: centrally managed, centrally upgraded, always at the upgrade level the central controller wants it to be. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi, On 2023-01-29 20:11, Gert Doering wrote:
Hi,
On Fri, Jan 27, 2023 at 02:15:44AM +0100, Robert Scheck wrote:
On Mon, 09 Jan 2023, Gert Doering wrote:
So don't do OS updates. But *do* update the probe software, automatically, and in normal intervals.
writing while wearing my hat as a RPM packager: I'm not really sure how this shall work practically: If the probe software is run on regular (non- hardware probe) systems, there are runtime dependencies to other software, e.g. due to dynamic linking. Latter is encouraged by Linux distributions, [..]
This is all good and true - so, do not distribute as RPM.
We produce the ("self signed") RPMs because we need them internally for the anchors anyway, so figured we may as well publish them, plus some people actually need/prefer binary RPMs. Otherwise we're heading to a direction where we act as the upstream for the software, and professional package maintainers do the proper signing, distribution and updates (like Robert S. could become one for RPMs).
Distribute as Docker image, or Flatpack, or whatever all these new "encapsulate a program and all its dependencies" approaches are called, and I'm sure some of them have mechanisms to enforce timely updates.
Docker files exist, though we are not maintaining official images. That may change in the future. As said before, "have mechanisms to enforce timely updates" indeed has proper solutions for basically all platforms.
First and foremost, a RIPE Atlas probe is a centrally managed piece of software, and this is what it needs to be: centrally managed, centrally upgraded, always at the upgrade level the central controller wants it to be.
As I tried to explain before, this is true for the HW probes, not so much for SW. Cheers, Robert
Gert Doering -- NetMaster
Hi, On Mon, Jan 30, 2023 at 01:24:38PM +0100, Robert Kisteleki wrote:
On 2023-01-29 20:11, Gert Doering wrote:
First and foremost, a RIPE Atlas probe is a centrally managed piece of software, and this is what it needs to be: centrally managed, centrally upgraded, always at the upgrade level the central controller wants it to be.
As I tried to explain before, this is true for the HW probes, not so much for SW.
And I keep totally ignoring all arguments why it should stay that way. So, yes, *today* it is the way it is, and I think "automated and mandatory updates" should have been designed into SW probes from day one - that boat has sailed, but I still think that this is a desirable goal. To repeat my argument: if I schedule a measurement across 100 probes, I do not want to deal with different results purely caused by different probe software versions (like, "30 of them do not support TLS1.3 yet, the other 970 do", which might look like "something is interfering with TLS1.3 measurements for these 30 probes"). Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi Michel, all: I'm a big fan of the Atlas programme, and have hosted probes for years now (and I'm an active member of this community). However, I don't have a technical or engineering background and while one of my probes is close to me, another is not. Both of the probes I host have firmware version 5080 (1100). Is there anything I need to do right now to update the software? And if not (e.g., it's the latest for now), can the NCC provide detailed instructions for how to do so (i.e., in the style of "WikiHow") so we can? Best, -Michael On Mon, Jan 9, 2023 at 10:19 AM Michel Stam <mstam@ripe.net> wrote:
Hi guys,
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
As to updating the package, we are looking to see if it is possible to make an RPM that is part of the base RedHat repository, together with a packager. We could do something similar for Debian (possibly others, such as OpenWRT).
I’m not sure what you mean with methods: How many probe users are using auto-update is a bit difficult to answer, since right now it is forced, but as of the next software release it will not be. Which operating systems are used is also a bit difficult since we do not provide an official Debian package yet, and the software does not differentiate between different flavours of Linux.
Hope this helps.
Regards,
Michel
On 8 Jan 2023, at 01:36, Seth David Schoen <schoen@loyalty.org> wrote:
Alexander Bochmann writes:
When using the CentOS binary repo as recommended, the software should auto-update during normal system update procedures.
For source releases (Debian), you're more or less on your own. Updating is still easy - just go through the initial installation again, minus registration.
I see, so that's a pretty significant difference in the experiences of people using different systems.
Is it clear how many probe users use each of these methods? (which might indicate whether it could be worth maintaining a more automated installation method for Debian-based systems)
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
Hi Michael, Are you running hardware probes (the small white boxes)? 1100 only applies to those. Let me know. Regards, Michel
On 9 Jan 2023, at 11:12, Michael J. Oghia <mike.oghia@gmail.com> wrote:
Hi Michel, all:
I'm a big fan of the Atlas programme, and have hosted probes for years now (and I'm an active member of this community). However, I don't have a technical or engineering background and while one of my probes is close to me, another is not. Both of the probes I host have firmware version 5080 (1100).
Is there anything I need to do right now to update the software? And if not (e.g., it's the latest for now), can the NCC provide detailed instructions for how to do so (i.e., in the style of "WikiHow") so we can?
Best, -Michael
On Mon, Jan 9, 2023 at 10:19 AM Michel Stam <mstam@ripe.net <mailto:mstam@ripe.net>> wrote:
Hi guys,
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
As to updating the package, we are looking to see if it is possible to make an RPM that is part of the base RedHat repository, together with a packager. We could do something similar for Debian (possibly others, such as OpenWRT).
I’m not sure what you mean with methods: How many probe users are using auto-update is a bit difficult to answer, since right now it is forced, but as of the next software release it will not be. Which operating systems are used is also a bit difficult since we do not provide an official Debian package yet, and the software does not differentiate between different flavours of Linux.
Hope this helps.
Regards,
Michel
On 8 Jan 2023, at 01:36, Seth David Schoen <schoen@loyalty.org <mailto:schoen@loyalty.org>> wrote:
Alexander Bochmann writes:
When using the CentOS binary repo as recommended, the software should auto-update during normal system update procedures.
For source releases (Debian), you're more or less on your own. Updating is still easy - just go through the initial installation again, minus registration.
I see, so that's a pretty significant difference in the experiences of people using different systems.
Is it clear how many probe users use each of these methods? (which might indicate whether it could be worth maintaining a more automated installation method for Debian-based systems)
-- ripe-atlas mailing list ripe-atlas@ripe.net <mailto:ripe-atlas@ripe.net> https://lists.ripe.net/mailman/listinfo/ripe-atlas
-- ripe-atlas mailing list ripe-atlas@ripe.net <mailto:ripe-atlas@ripe.net> https://lists.ripe.net/mailman/listinfo/ripe-atlas
Hi Michel, Happy New Year! Yes, exactly, the small white probes. Best, -Michael On Mon, Jan 9, 2023 at 11:27 AM Michel Stam <mstam@ripe.net> wrote:
Hi Michael,
Are you running hardware probes (the small white boxes)? 1100 only applies to those.
Let me know.
Regards,
Michel
On 9 Jan 2023, at 11:12, Michael J. Oghia <mike.oghia@gmail.com> wrote:
Hi Michel, all:
I'm a big fan of the Atlas programme, and have hosted probes for years now (and I'm an active member of this community). However, I don't have a technical or engineering background and while one of my probes is close to me, another is not. Both of the probes I host have firmware version 5080 (1100).
Is there anything I need to do right now to update the software? And if not (e.g., it's the latest for now), can the NCC provide detailed instructions for how to do so (i.e., in the style of "WikiHow") so we can?
Best, -Michael
On Mon, Jan 9, 2023 at 10:19 AM Michel Stam <mstam@ripe.net> wrote:
Hi guys,
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
As to updating the package, we are looking to see if it is possible to make an RPM that is part of the base RedHat repository, together with a packager. We could do something similar for Debian (possibly others, such as OpenWRT).
I’m not sure what you mean with methods: How many probe users are using auto-update is a bit difficult to answer, since right now it is forced, but as of the next software release it will not be. Which operating systems are used is also a bit difficult since we do not provide an official Debian package yet, and the software does not differentiate between different flavours of Linux.
Hope this helps.
Regards,
Michel
On 8 Jan 2023, at 01:36, Seth David Schoen <schoen@loyalty.org> wrote:
Alexander Bochmann writes:
When using the CentOS binary repo as recommended, the software should auto-update during normal system update procedures.
For source releases (Debian), you're more or less on your own. Updating is still easy - just go through the initial installation again, minus registration.
I see, so that's a pretty significant difference in the experiences of people using different systems.
Is it clear how many probe users use each of these methods? (which might indicate whether it could be worth maintaining a more automated installation method for Debian-based systems)
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
Hi Michael, Oh of course, happy new year to you too (keep forgetting). In that case, you don’t have to do anything to keep your probe up to date, this should all happen automatically. Cheers, Michel
On 9 Jan 2023, at 11:31, Michael J. Oghia <mike.oghia@gmail.com> wrote:
Hi Michel,
Happy New Year! Yes, exactly, the small white probes.
Best, -Michael
On Mon, Jan 9, 2023 at 11:27 AM Michel Stam <mstam@ripe.net <mailto:mstam@ripe.net>> wrote:
Hi Michael,
Are you running hardware probes (the small white boxes)? 1100 only applies to those.
Let me know.
Regards,
Michel
On 9 Jan 2023, at 11:12, Michael J. Oghia <mike.oghia@gmail.com <mailto:mike.oghia@gmail.com>> wrote:
Hi Michel, all:
I'm a big fan of the Atlas programme, and have hosted probes for years now (and I'm an active member of this community). However, I don't have a technical or engineering background and while one of my probes is close to me, another is not. Both of the probes I host have firmware version 5080 (1100).
Is there anything I need to do right now to update the software? And if not (e.g., it's the latest for now), can the NCC provide detailed instructions for how to do so (i.e., in the style of "WikiHow") so we can?
Best, -Michael
On Mon, Jan 9, 2023 at 10:19 AM Michel Stam <mstam@ripe.net <mailto:mstam@ripe.net>> wrote:
Hi guys,
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
As to updating the package, we are looking to see if it is possible to make an RPM that is part of the base RedHat repository, together with a packager. We could do something similar for Debian (possibly others, such as OpenWRT).
I’m not sure what you mean with methods: How many probe users are using auto-update is a bit difficult to answer, since right now it is forced, but as of the next software release it will not be. Which operating systems are used is also a bit difficult since we do not provide an official Debian package yet, and the software does not differentiate between different flavours of Linux.
Hope this helps.
Regards,
Michel
On 8 Jan 2023, at 01:36, Seth David Schoen <schoen@loyalty.org <mailto:schoen@loyalty.org>> wrote:
Alexander Bochmann writes:
When using the CentOS binary repo as recommended, the software should auto-update during normal system update procedures.
For source releases (Debian), you're more or less on your own. Updating is still easy - just go through the initial installation again, minus registration.
I see, so that's a pretty significant difference in the experiences of people using different systems.
Is it clear how many probe users use each of these methods? (which might indicate whether it could be worth maintaining a more automated installation method for Debian-based systems)
-- ripe-atlas mailing list ripe-atlas@ripe.net <mailto:ripe-atlas@ripe.net> https://lists.ripe.net/mailman/listinfo/ripe-atlas
-- ripe-atlas mailing list ripe-atlas@ripe.net <mailto:ripe-atlas@ripe.net> https://lists.ripe.net/mailman/listinfo/ripe-atlas
Ah OK, great! Thank you, and sorry for being such a nooby luddite :-) Best, -Michael On Mon, Jan 9, 2023 at 11:42 AM Michel Stam <mstam@ripe.net> wrote:
Hi Michael,
Oh of course, happy new year to you too (keep forgetting).
In that case, you don’t have to do anything to keep your probe up to date, this should all happen automatically.
Cheers,
Michel
On 9 Jan 2023, at 11:31, Michael J. Oghia <mike.oghia@gmail.com> wrote:
Hi Michel,
Happy New Year! Yes, exactly, the small white probes.
Best, -Michael
On Mon, Jan 9, 2023 at 11:27 AM Michel Stam <mstam@ripe.net> wrote:
Hi Michael,
Are you running hardware probes (the small white boxes)? 1100 only applies to those.
Let me know.
Regards,
Michel
On 9 Jan 2023, at 11:12, Michael J. Oghia <mike.oghia@gmail.com> wrote:
Hi Michel, all:
I'm a big fan of the Atlas programme, and have hosted probes for years now (and I'm an active member of this community). However, I don't have a technical or engineering background and while one of my probes is close to me, another is not. Both of the probes I host have firmware version 5080 (1100).
Is there anything I need to do right now to update the software? And if not (e.g., it's the latest for now), can the NCC provide detailed instructions for how to do so (i.e., in the style of "WikiHow") so we can?
Best, -Michael
On Mon, Jan 9, 2023 at 10:19 AM Michel Stam <mstam@ripe.net> wrote:
Hi guys,
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
As to updating the package, we are looking to see if it is possible to make an RPM that is part of the base RedHat repository, together with a packager. We could do something similar for Debian (possibly others, such as OpenWRT).
I’m not sure what you mean with methods: How many probe users are using auto-update is a bit difficult to answer, since right now it is forced, but as of the next software release it will not be. Which operating systems are used is also a bit difficult since we do not provide an official Debian package yet, and the software does not differentiate between different flavours of Linux.
Hope this helps.
Regards,
Michel
On 8 Jan 2023, at 01:36, Seth David Schoen <schoen@loyalty.org> wrote:
Alexander Bochmann writes:
When using the CentOS binary repo as recommended, the software should auto-update during normal system update procedures.
For source releases (Debian), you're more or less on your own. Updating is still easy - just go through the initial installation again, minus registration.
I see, so that's a pretty significant difference in the experiences of people using different systems.
Is it clear how many probe users use each of these methods? (which might indicate whether it could be worth maintaining a more automated installation method for Debian-based systems)
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
No worries Michael, all good :) Cheers, Michel
On 9 Jan 2023, at 11:47, Michael J. Oghia <mike.oghia@gmail.com> wrote:
Ah OK, great! Thank you, and sorry for being such a nooby luddite :-)
Best, -Michael
On Mon, Jan 9, 2023 at 11:42 AM Michel Stam <mstam@ripe.net <mailto:mstam@ripe.net>> wrote:
Hi Michael,
Oh of course, happy new year to you too (keep forgetting).
In that case, you don’t have to do anything to keep your probe up to date, this should all happen automatically.
Cheers,
Michel
On 9 Jan 2023, at 11:31, Michael J. Oghia <mike.oghia@gmail.com <mailto:mike.oghia@gmail.com>> wrote:
Hi Michel,
Happy New Year! Yes, exactly, the small white probes.
Best, -Michael
On Mon, Jan 9, 2023 at 11:27 AM Michel Stam <mstam@ripe.net <mailto:mstam@ripe.net>> wrote:
Hi Michael,
Are you running hardware probes (the small white boxes)? 1100 only applies to those.
Let me know.
Regards,
Michel
On 9 Jan 2023, at 11:12, Michael J. Oghia <mike.oghia@gmail.com <mailto:mike.oghia@gmail.com>> wrote:
Hi Michel, all:
I'm a big fan of the Atlas programme, and have hosted probes for years now (and I'm an active member of this community). However, I don't have a technical or engineering background and while one of my probes is close to me, another is not. Both of the probes I host have firmware version 5080 (1100).
Is there anything I need to do right now to update the software? And if not (e.g., it's the latest for now), can the NCC provide detailed instructions for how to do so (i.e., in the style of "WikiHow") so we can?
Best, -Michael
On Mon, Jan 9, 2023 at 10:19 AM Michel Stam <mstam@ripe.net <mailto:mstam@ripe.net>> wrote:
Hi guys,
The automatic update for the CentOS package was removed as of 5080. This means that the update to 5080 will still be automatic, but not any more after this (say 5090 onwards). This was a request by several users because Atlas forced the update, which violated their sysadmin policies.
As to updating the package, we are looking to see if it is possible to make an RPM that is part of the base RedHat repository, together with a packager. We could do something similar for Debian (possibly others, such as OpenWRT).
I’m not sure what you mean with methods: How many probe users are using auto-update is a bit difficult to answer, since right now it is forced, but as of the next software release it will not be. Which operating systems are used is also a bit difficult since we do not provide an official Debian package yet, and the software does not differentiate between different flavours of Linux.
Hope this helps.
Regards,
Michel
> On 8 Jan 2023, at 01:36, Seth David Schoen <schoen@loyalty.org <mailto:schoen@loyalty.org>> wrote: > > Alexander Bochmann writes: > >> When using the CentOS binary repo as recommended, the software should >> auto-update during normal system update procedures. >> >> For source releases (Debian), you're more or less on your own. Updating >> is still easy - just go through the initial installation again, minus >> registration. > > I see, so that's a pretty significant difference in the experiences of > people using different systems. > > Is it clear how many probe users use each of these methods? (which > might indicate whether it could be worth maintaining a more automated > installation method for Debian-based systems) > > -- > ripe-atlas mailing list > ripe-atlas@ripe.net <mailto:ripe-atlas@ripe.net> > https://lists.ripe.net/mailman/listinfo/ripe-atlas
-- ripe-atlas mailing list ripe-atlas@ripe.net <mailto:ripe-atlas@ripe.net> https://lists.ripe.net/mailman/listinfo/ripe-atlas
I have to check but I believe the CentOS version does automatically upgrade. Pretty sure it did with my CentOS probe running on VMware on Windows. It ran 5070 when I installed it, it now runs 5080. I did not re-install it or did anything else. Regards, Ernst J. Oud
On 7 Jan 2023, at 06:03, Seth David Schoen <schoen@loyalty.org> wrote:
Hi,
I recently posted two threads ultimately related to the same question, where it turned out that a feature I wanted to use was present in the current software probe release, but not in the old version that I had installed. It was indeed relatively easy to upgrade (as a reply on this list predicted, the upgrade didn't break anything or require any reconfiguration).
But I'd still say "relatively easy", because, on the other hand, the upgrade wasn't managed through my OS package manager like the rest of the software on my VPS server.
It appears to me that a lot of software probes (like mine was until recently!) are still running whichever software probe version was current when they were first deployed. After all, I don't believe there's automatic software upgrade functionality in the methods that most people are using to deploy these.
Is that true from the point of view of the rest of the Atlas community? Is there anything that might usefully be done to encourage people to upgrade periodically? (For example, either sending some kind of e-mail reminder, or distributing probe software via OS package managers, or even including a mechanism in the probe client itself by which it could upgrade itself -- even if as an opt-in feature?)
-- ripe-atlas mailing list ripe-atlas@ripe.net https://lists.ripe.net/mailman/listinfo/ripe-atlas
participants (10)
-
Alexander Bochmann -
Ernst J. Oud -
Gert Doering -
Lukas Tribus -
Michael J. Oghia -
Michel Stam -
ripe.net＠toppas.net -
Robert Kisteleki -
Robert Scheck -
Seth David Schoen