Hi All, What about HTTP/HTTPS requests from Atlas probes? I hear it is technically possible. This test can provide statistics about quality and availability of certain websites, as well as the information about content is delivered unmodified, not blocked, correctly/incorrectly cached, etc. Is it possible to enable that kind of probes?
Httping! Op 1 okt. 2013 17:27 schreef "Max Tulyev" <maxtul@netassist.ua> het volgende:
Hi All,
What about HTTP/HTTPS requests from Atlas probes? I hear it is technically possible.
This test can provide statistics about quality and availability of certain websites, as well as the information about content is delivered unmodified, not blocked, correctly/incorrectly cached, etc.
Is it possible to enable that kind of probes?
On 2013.10.01. 17:05, Max Tulyev wrote:
Hi All,
What about HTTP/HTTPS requests from Atlas probes? I hear it is technically possible.
This test can provide statistics about quality and availability of certain websites, as well as the information about content is delivered unmodified, not blocked, correctly/incorrectly cached, etc.
Is it possible to enable that kind of probes?
Hi, This topic has surfaced a couple of times already (in this list: Aug 2012, July 2013), probably on other related lists too. Bottom line: * indeed, technically it is possible to do HTTP requests, and every now and then we're doing tests ourselves or with external researchers * public release needs community discussion, as this feature can have dire consequences to probe hosts (especially in terms of bandwidth usage and privacy consequences) * HTTPS is a different question: the "SSL get certificate" measurement is available to all users See these threads: https://www.ripe.net/ripe/mail/archives/ripe-atlas/2012-August/000552.html https://www.ripe.net/ripe/mail/archives/ripe-atlas/2013-July/000929.html Regards, Robert
On Tue, Oct 01, 2013 at 08:01:28PM +0200, Robert Kisteleki <robert@ripe.net> wrote a message of 33 lines which said:
This topic has surfaced a couple of times already (in this list: Aug 2012, July 2013), probably on other related lists too.
It would surface less often if the doc were fixed <https://atlas.ripe.net/doc/udm#creating-a-new-measurement>.
On 2013.10.02. 10:13, Stephane Bortzmeyer wrote:
On Tue, Oct 01, 2013 at 08:01:28PM +0200, Robert Kisteleki <robert@ripe.net> wrote a message of 33 lines which said:
This topic has surfaced a couple of times already (in this list: Aug 2012, July 2013), probably on other related lists too.
It would surface less often if the doc were fixed <https://atlas.ripe.net/doc/udm#creating-a-new-measurement>.
We updated the doc right after the last time you asked :-) "Important note: HTTP(6) measurements are not available to all UDM users; they are restricted while the potential impact is evaluated." Regards, Robert
Robert Kisteleki <robert@ripe.net> escribió:
On 2013.10.02. 10:13, Stephane Bortzmeyer wrote:
On Tue, Oct 01, 2013 at 08:01:28PM +0200, Robert Kisteleki <robert@ripe.net> wrote a message of 33 lines which said:
This topic has surfaced a couple of times already (in this list: Aug 2012, July 2013), probably on other related lists too.
It would surface less often if the doc were fixed <https://atlas.ripe.net/doc/udm#creating-a-new-measurement>.
We updated the doc right after the last time you asked :-)
"Important note: HTTP(6) measurements are not available to all UDM users; they are restricted while the potential impact is evaluated."
Hi, my proposal: - Two different checks in the probe configuration: - http:// small file download - http:// big file download Then, the user can select if their probe will download big or small files (bw impact). - Fixed targets. The probe can connect only selected hosts and files (privacy impact). IMO, the targets should be non commercial sites, with good bw. Regards, kix
Regards, Robert
Rodolfo García Peñas (kix) http://www.kix.es/
I was one of the people that urged caution with regard to HTTP probes. There's some ambiguity that needs to be resolved first, and then some security risks to consider. The basic question is "what is being measured"? There are several possible answers: 1. Transport-layer connectivity 2. HTTP header information 3. HTTP body information Each of these has some risks that would beed to be controlled. (1) is obviously the safest; that's part of what the SSL cert measurement does. But (1) is not really an HTTP measurement, it's a TCP measurement, and it would be better to cast it that way. (2) could probably be implemented pretty safely by sending a HEAD request. However, there's still a risk that private user information would leak in such requests. For example, if a web site is doing IP-address based access control, and the probe is behind the same NAT as a user's laptop, then even a HEAD request might return user information (e.g., session cookies). (3) is a huge security risk, because of the wide variety of things that are done with HTTP requests. For simplicity, let's assume the probe would send a GET request, and not anything more sophisticated (POST, PUT, DELETE, etc.). You could use a GET request to download a file, but you can also a GET request to do things to supply responses to HTTP forms. Want to make sure your favorite band wins the EuroVision Song Contest? Just task the Atlas network have 1000 probes vote for them every 5 minutes. There's also the question of what you do with the downloaded content. Returning it to the measurement owner would raise huge security issues, not to mention bandwidth issues. But if you don't return it, then the system will need to constrain the questions the experimenter can ask, e.g., "How many bytes were received?" "What was the SHA-1 digest of the file?". --Richard On Wed, Oct 2, 2013 at 5:21 AM, Rodolfo García Peñas (kix) <kix@kix.es>wrote:
Robert Kisteleki <robert@ripe.net> escribió:
On 2013.10.02. 10:13, Stephane Bortzmeyer wrote:
On Tue, Oct 01, 2013 at 08:01:28PM +0200, Robert Kisteleki <robert@ripe.net> wrote a message of 33 lines which said:
This topic has surfaced a couple of times already (in this list: Aug
2012, July 2013), probably on other related lists too.
It would surface less often if the doc were fixed <https://atlas.ripe.net/doc/**udm#creating-a-new-measurement<https://atlas.ripe.net/doc/udm#creating-a-new-measurement> **>.
We updated the doc right after the last time you asked :-)
"Important note: HTTP(6) measurements are not available to all UDM users; they are restricted while the potential impact is evaluated."
Hi,
my proposal:
- Two different checks in the probe configuration: - http:// small file download - http:// big file download Then, the user can select if their probe will download big or small files (bw impact). - Fixed targets. The probe can connect only selected hosts and files (privacy impact). IMO, the targets should be non commercial sites, with good bw.
Regards, kix
Regards,
Robert
Rodolfo García Peñas (kix) http://www.kix.es/
On Wed, 2 Oct 2013 14:13:11 -0400 Richard Barnes <rlb@ipv.sx> wrote:
(3) is a huge security risk, because of the wide variety of things that are done with HTTP requests. For simplicity, let's assume the probe would send a GET request, and not anything more sophisticated (POST, PUT, DELETE, etc.). You could use a GET request to download a file, but you can also a GET request to do things to supply responses to HTTP forms. Want to make sure your favorite band wins the EuroVision Song Contest? Just task the Atlas network have 1000 probes vote for them every 5 minutes.
GET requests should not alter state; if they do, arguably the problem there lies with the design of the faulty website.
On Thursday, November 21, 2013, David Precious wrote:
On Wed, 2 Oct 2013 14:13:11 -0400 Richard Barnes <rlb@ipv.sx> wrote:
(3) is a huge security risk, because of the wide variety of things that are done with HTTP requests. For simplicity, let's assume the probe would send a GET request, and not anything more sophisticated (POST, PUT, DELETE, etc.). You could use a GET request to download a file, but you can also a GET request to do things to supply responses to HTTP forms. Want to make sure your favorite band wins the EuroVision Song Contest? Just task the Atlas network have 1000 probes vote for them every 5 minutes.
GET requests should not alter state; if they do, arguably the problem there lies with the design of the faulty website.
Indeed, that is what the HTTP spec says. But there are a good number of fault websites out there, and it seems bad to have Atlas be a tool to exploit them. In theory, there's no difference between theory and practice, but in practice there is :)
On 21Nov13, Richard Barnes allegedly wrote:
GET requests should not alter state; if they do, arguably the problem there lies with the design of the faulty website.
Indeed, that is what the HTTP spec says. But there are a good number of fault websites out there, and it seems bad to have Atlas be a tool to exploit them.
Agreed. Given the infinite monkeys that have written piblic facing web services, there is bound to be web sites that use HTTP verbs in weird and wonderful ways. But what about using HEAD? That would serve a lot of monitoring purposes as it can give you connect time and time to first byte, it doesn't return any content so the problem of fetching dodgy content is mitigated and the size of the payload is much more constrained. Another alternative is to only allow something like the "OPTION" or "TRACE" verbs. For those probing their own systems they could implement these VERBs but even if those VERBS aren't implemented you still get time to first byte as a consequence of the error returned. Mark.
Hi, HEAD would be better imho because TRACE mode is usually disabled. (vulnerability scanners tend to complain about it so it will be disabled most of the time...) ax On Thu, Nov 21, 2013 at 7:23 PM, Mark Delany <f4w@echo.emu.st> wrote:
On 21Nov13, Richard Barnes allegedly wrote:
GET requests should not alter state; if they do, arguably the problem there lies with the design of the faulty website.
Indeed, that is what the HTTP spec says. But there are a good number of fault websites out there, and it seems bad to have Atlas be a tool to exploit them.
Agreed. Given the infinite monkeys that have written piblic facing web services, there is bound to be web sites that use HTTP verbs in weird and wonderful ways.
But what about using HEAD?
That would serve a lot of monitoring purposes as it can give you connect time and time to first byte, it doesn't return any content so the problem of fetching dodgy content is mitigated and the size of the payload is much more constrained.
Another alternative is to only allow something like the "OPTION" or "TRACE" verbs.
For those probing their own systems they could implement these VERBs but even if those VERBS aren't implemented you still get time to first byte as a consequence of the error returned.
Mark.
I think HEAD would probably be OK. At least, I'm not aware of any exploits that would enable. --Richard On Thu, Nov 21, 2013 at 1:30 PM, Imre Szvorenyi <ax@initrd.net> wrote:
Hi,
HEAD would be better imho because TRACE mode is usually disabled. (vulnerability scanners tend to complain about it so it will be disabled most of the time...)
ax
On Thu, Nov 21, 2013 at 7:23 PM, Mark Delany <f4w@echo.emu.st> wrote:
On 21Nov13, Richard Barnes allegedly wrote:
GET requests should not alter state; if they do, arguably the problem there lies with the design of the faulty website.
Indeed, that is what the HTTP spec says. But there are a good number of fault websites out there, and it seems bad to have Atlas be a tool to exploit them.
Agreed. Given the infinite monkeys that have written piblic facing web services, there is bound to be web sites that use HTTP verbs in weird and wonderful ways.
But what about using HEAD?
That would serve a lot of monitoring purposes as it can give you connect time and time to first byte, it doesn't return any content so the problem of fetching dodgy content is mitigated and the size of the payload is much more constrained.
Another alternative is to only allow something like the "OPTION" or "TRACE" verbs.
For those probing their own systems they could implement these VERBs but even if those VERBS aren't implemented you still get time to first byte as a consequence of the error returned.
Mark.
On 2013.11.21. 20:41, Richard Barnes wrote:
I think HEAD would probably be OK. At least, I'm not aware of any exploits that would enable.
--Richard
A completely different dimension of this is something that's (almost) unique to Atlas: you can use vantage points that are in others' networks or homes. Seemingly the request (whether it's GET or HEAD) will come from there. If one uses this to access illegal content (for whatever definition of illegal), then the host will be in trouble, and I don't think explanations about "it was not me, it was someone else" or "it was just a HEAD request, I didn't really see that picture" will make a difference in all cases. (We will likely be able to trace back the actual request to someone, but it may not help the host in question.) Maybe I'm a pessimist here, but if I'm not, then it means we can only use some kind of opt-in method, and even that does not solve the root problem. The alternative is to limit the potential destinations system-wide, which severely limits the usefulness of such measurements (or it boosts the Atlas Anchor deployment rate ;-) Robert
I suggest to start making HEAD requests to a limited set of destinations available to all users. Initially limit the list destinations under control of RIPE NCC, including anchors. I feel that this would not require an opt-in question to probe hosts, just information. As a next step we could then define a process for adding other destinations. This would prevent Atlas from becoming a general purpose tool for HTTP performance measurement and abuse. How's that to start with? Daniel On 22.11.2013, at 12:32 , Robert Kisteleki <robert@ripe.net> wrote:
On 2013.11.21. 20:41, Richard Barnes wrote:
I think HEAD would probably be OK. At least, I'm not aware of any exploits that would enable.
--Richard
A completely different dimension of this is something that's (almost) unique to Atlas: you can use vantage points that are in others' networks or homes. Seemingly the request (whether it's GET or HEAD) will come from there.
If one uses this to access illegal content (for whatever definition of illegal), then the host will be in trouble, and I don't think explanations about "it was not me, it was someone else" or "it was just a HEAD request, I didn't really see that picture" will make a difference in all cases. (We will likely be able to trace back the actual request to someone, but it may not help the host in question.)
Maybe I'm a pessimist here, but if I'm not, then it means we can only use some kind of opt-in method, and even that does not solve the root problem.
The alternative is to limit the potential destinations system-wide, which severely limits the usefulness of such measurements (or it boosts the Atlas Anchor deployment rate ;-)
Robert
about "it was not me, it was someone else" or "it was just a HEAD request, I didn't really see that picture" will make a difference in all cases. (We will likely be able to trace back the actual request to someone, but it may not help the host in question.)
That's why I suggested "OPTIONS" or "TRACE". There is no real content exchanged in either of those two verbs. Well, I guess a truly subverted HTTP server could send content with any verb, but equally you can run a content server over DNS if you try hard enough... I don't think it matters that many web servers don't implement those VERBs as the exchange from a TCP perspective is identical to a GET or HEAD in either case. The only difference is that the content may be an options list or a 400 response. Not something a TCP stack cares about. That is not to ignore the concern you raised. I agree that it's very real. But it's a question of degree as the risk already exists. If someone was trawling thru my ISPs dnscache logs and saw queries to unsavoury host names referred to by UDMs, that could similarly be embarrassing. Or worse, raise an alert with the local authorities if the host names in question contains words on a government proscribed list - as exist in some countries already. So if TCP measurements were to be allowed - then HTTP OPTIONS in cleartext is almost as innocuous as a DNS query over TCP which is almost as innocuous as a DNS query over UDP. It's also pretty innocuous from the receivers perspective too. Seeing an occasional OPTIONS request in an HTTP log is pretty mild compared to all the other cruft sent by spiders and bots. On my personal web server I see about 1 real request for every 30-40 spider/bot requests. The anomaly for me is a real person looking at my site :-) Mark.
So if I have the http measurement could be globally interested, what is the algorythm to realize it? On 02.10.13 11:42, Robert Kisteleki wrote:
On 2013.10.02. 10:13, Stephane Bortzmeyer wrote:
On Tue, Oct 01, 2013 at 08:01:28PM +0200, Robert Kisteleki <robert@ripe.net> wrote a message of 33 lines which said:
This topic has surfaced a couple of times already (in this list: Aug 2012, July 2013), probably on other related lists too.
It would surface less often if the doc were fixed <https://atlas.ripe.net/doc/udm#creating-a-new-measurement>.
We updated the doc right after the last time you asked :-)
"Important note: HTTP(6) measurements are not available to all UDM users; they are restricted while the potential impact is evaluated."
Regards, Robert
participants (10)
-
Daniel Karrenberg -
David Precious -
Folkert van Heusden -
Imre Szvorenyi -
Mark Delany -
Max Tulyev -
Richard Barnes -
Robert Kisteleki -
Rodolfo García Peñas (kix) -
Stephane Bortzmeyer