Re: [anti-abuse-wg] [Misc] Research project on blacklists
* ac <ac@main.me> [2019-07-18 08:29]:
It is about: "evaluating and improving the accuracy of blacklists."
The entire post is arrogant, obnoxious, offensive and inaccurate and is an oxymoron.
Tbh the only mails I get this vibe from in this thread are yours. Could you tune it down a bit? There is no reason to be this aggressive. It seems this is legitimate research and if you do not agree with it don't take the survey. Regards Sebastian -- GPG Key: 0x58A2D94A93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant
On Thu, 18 Jul 2019 09:02:01 +0200 Sebastian Wiesinger <sebastian@karotte.org> wrote:
* ac <ac@main.me> [2019-07-18 08:29]:
It is about: "evaluating and improving the accuracy of blacklists." The entire post is arrogant, obnoxious, offensive and inaccurate and is an oxymoron.
Tbh the only mails I get this vibe from in this thread are yours. Could you tune it down a bit? There is no reason to be this aggressive. It seems this is legitimate research and if you do not agree with it don't take the survey.
You are correct of course. I am angry now that it is clear that it is not a joke. And; still assuming that we are talking about email "blacklists" and not routing "blacklists" - routing blacklists are always DROP, but there has been a shift in "free" tech... here is the crux: Most of ALL of us that are not Microsoft or Google (the minority) Use "blacklists" for reputational scoring now (and not for 'drop') Since we started doing that, much of the non private anti abuse tech (in use by Microsoft and Google) has seen their emails to us, ending up in SPAM boxes... - We have seen our spam drop off as this reputational scoring use for blacklists, work! So, now this... And, yes there is a lot of legit research in the world, mostly paid for by whomever has whatever agenda. for example pork farmers paying for research to show that pork meat has less cholesterol I am just becoming tired of the angles big tech works to dominate. Already Microsoft and Google relays most of the email on the planet. they also need to get off ipv4 as they know on ipv6 we have to "whitelist" them.... and then all of us also has to expand our use of less private communication inspection and content analysis... RIPE already accepts whitelist ipv6 from Google, for example... anyway.... Ongoing battle and most people do not even see the angles Mostly, what makes me very angry is the audacity and then the "anonymous" and I can already see the "findings" of this research... based on random anonymous, hidden and secret inputs.... so, sorry for my aggression. I guess big tech, up and coming (well funded new tech), cyber crime and all the other abuse relatives just get to me sometimes. Andre
In message , ac <ac@main.me> writes
Mostly, what makes me very angry is the audacity
this does seem a reasonable list to ask for assistance on ... but being around to answer questions promptly would be appropriately polite surprisingly, I haven't seen the request on any other lists that are (a) relevant and (b) open -- perhaps they and their project team are not especially well connected in this space :( though there is a recent "anonymous" survey request about router configurations on the NANOG list
and then the "anonymous"
the Qualtrics platform is available over Tor (unlike some online survey platforms) so if you declined to answer the questions about which AS and company you were associated with then there is a substantial amount of anonymity available to you should you wish to use it...
and I can already see the "findings" of this research... based on random anonymous, hidden and secret inputs....
that is a concern -- this type of questionnaire pretty much never leads to high quality research directly (since there are significant biases in who might choose to give replies and there is scope for multiple responses from a single person, bots filling it in etc) nevertheless as a starting point for qualitative research (rather than quantitative) it can be very useful in allowing a researcher to identify general trends in the answers and -- importantly -- to help the researcher frame good research questions that are capable of being investigated in more detail as John Levine already noted, the questionnaire seems somewhat confused as to whether it cares about routing issues (bogon lists, the Spamhaus DROP list etc) or spam filtering (bad domains, phishing feeds, botnet IPs etc etc) it also asked if internally generated lists were used, but seemed curiously uninterested in anything other than if the answer to that was yes or no -- a missed opportunity I thought. -- richard Richard Clayton Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755
Apologies for my slow response - I have been traveling and also consulting with my team members on how best to respond (as you might have gleaned from my profile linked upthread, my own background is not in networking or security :)). I hope to share more thorough responses with you once the sun rises in their timezones.
surprisingly, I haven't seen the request on any other lists that are (a) relevant and (b) open -- perhaps they and their project team are not especially well connected in this space :(
This is true. We were advised to share to RIPE and regional NOG mailing lists. Are there others you would have recommended?
as John Levine already noted, the questionnaire seems somewhat confused as to whether it cares about routing issues (bogon lists, the Spamhaus DROP list etc) or spam filtering (bad domains, phishing feeds, botnet IPs etc etc)
Hm, I think we are interested in quite the range of blacklists. Here is a table of what my colleagues are monitoring: [image: image.png]
it also asked if internally generated lists were used, but seemed curiously uninterested in anything other than if the answer to that was yes or no -- a missed opportunity I thought.
What would you have recommended probing here? I do genuinely appreciate your discussion and patience. It is very interesting and useful for me to see what topics matter to you most and where we might have misdirected our attention. Just as background, we did pilot the survey with a smaller set of network operators and felt it had been straightforward to respond to, given their reactions. But as many of you have noted, the survey is rather general. I have been conducting interviews with those working in abuse prevention (even at some of the companies that have been mentioned upthread) to collect more specific anecdotes about how dynamic addressing has lowered the accuracy of certain feeds, for example, or how errors in geo-IP feeds affected them. The interviews allow for a bit more elucidation, but it has been difficult to recruit participants. Hence the survey. All the best, Anushah On Thu, Jul 18, 2019 at 2:36 PM Richard Clayton <richard@highwayman.com> wrote:
In message , ac <ac@main.me> writes
Mostly, what makes me very angry is the audacity
this does seem a reasonable list to ask for assistance on ... but being around to answer questions promptly would be appropriately polite
surprisingly, I haven't seen the request on any other lists that are (a) relevant and (b) open -- perhaps they and their project team are not especially well connected in this space :( though there is a recent "anonymous" survey request about router configurations on the NANOG list
and then the "anonymous"
the Qualtrics platform is available over Tor (unlike some online survey platforms) so if you declined to answer the questions about which AS and company you were associated with then there is a substantial amount of anonymity available to you should you wish to use it...
and I can already see the "findings" of this research... based on random anonymous, hidden and secret inputs....
that is a concern -- this type of questionnaire pretty much never leads to high quality research directly (since there are significant biases in who might choose to give replies and there is scope for multiple responses from a single person, bots filling it in etc)
nevertheless as a starting point for qualitative research (rather than quantitative) it can be very useful in allowing a researcher to identify general trends in the answers and -- importantly -- to help the researcher frame good research questions that are capable of being investigated in more detail
as John Levine already noted, the questionnaire seems somewhat confused as to whether it cares about routing issues (bogon lists, the Spamhaus DROP list etc) or spam filtering (bad domains, phishing feeds, botnet IPs etc etc)
it also asked if internally generated lists were used, but seemed curiously uninterested in anything other than if the answer to that was yes or no -- a missed opportunity I thought.
-- richard Richard Clayton
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755
-- Anushah Hossain, PhD Student Energy and Resources Group, UC Berkeley
Hi, I note that I am involved in one or more of the blacklists in your image.png As you are aware, I have decided to take offense at the stated goal of your research, (to "improve") as I know enough, to know, that you do not understand all the protocols to generally make any such oxymoronic statement as it regards certain aspects of blacklist management and/or protocols. More so as it pertains specifically to what I do. Even if your research would result in any measurable improvement in any other (and operationally unknown to myself) blacklist data it would still be far less meaningful as actual useful abuse tech research, more specifically, I mean less of a shotgun and more of a rifle. You also clearly do not understand how the differences in protocols of your blacklist classification manages data and how this affects accuracy, as is demonstrated by the blocklists as they are reflected in your own image.png, yet you want to improve data accuracy and you want to be taken seriously. Anyway, this will need a review if it is to be useful, accurate or not whitewashed paid for research. And no, I am not keen on putting it on a spoon for you or adding any meat. Andre On Thu, 18 Jul 2019 15:33:39 +0200 Anushah Hossain <anushah@icsi.berkeley.edu> wrote:
Apologies for my slow response - I have been traveling and also consulting with my team members on how best to respond (as you might have gleaned from my profile linked upthread, my own background is not in networking or security :)). I hope to share more thorough responses with you once the sun rises in their timezones.
surprisingly, I haven't seen the request on any other lists that are (a) relevant and (b) open -- perhaps they and their project team are not especially well connected in this space :(
This is true. We were advised to share to RIPE and regional NOG mailing lists. Are there others you would have recommended?
as John Levine already noted, the questionnaire seems somewhat confused as to whether it cares about routing issues (bogon lists, the Spamhaus DROP list etc) or spam filtering (bad domains, phishing feeds, botnet IPs etc etc)
Hm, I think we are interested in quite the range of blacklists. Here is a table of what my colleagues are monitoring:
[image: image.png]
it also asked if internally generated lists were used, but seemed curiously uninterested in anything other than if the answer to that was yes or no -- a missed opportunity I thought.
What would you have recommended probing here?
I do genuinely appreciate your discussion and patience. It is very interesting and useful for me to see what topics matter to you most and where we might have misdirected our attention. Just as background, we did pilot the survey with a smaller set of network operators and felt it had been straightforward to respond to, given their reactions. But as many of you have noted, the survey is rather general. I have been conducting interviews with those working in abuse prevention (even at some of the companies that have been mentioned upthread) to collect more specific anecdotes about how dynamic addressing has lowered the accuracy of certain feeds, for example, or how errors in geo-IP feeds affected them. The interviews allow for a bit more elucidation, but it has been difficult to recruit participants. Hence the survey.
All the best, Anushah
On Thu, Jul 18, 2019 at 2:36 PM Richard Clayton <richard@highwayman.com> wrote:
In message , ac <ac@main.me> writes
Mostly, what makes me very angry is the audacity
this does seem a reasonable list to ask for assistance on ... but being around to answer questions promptly would be appropriately polite
surprisingly, I haven't seen the request on any other lists that are (a) relevant and (b) open -- perhaps they and their project team are not especially well connected in this space :( though there is a recent "anonymous" survey request about router configurations on the NANOG list
and then the "anonymous"
the Qualtrics platform is available over Tor (unlike some online survey platforms) so if you declined to answer the questions about which AS and company you were associated with then there is a substantial amount of anonymity available to you should you wish to use it...
and I can already see the "findings" of this research... based on random anonymous, hidden and secret inputs....
that is a concern -- this type of questionnaire pretty much never leads to high quality research directly (since there are significant biases in who might choose to give replies and there is scope for multiple responses from a single person, bots filling it in etc)
nevertheless as a starting point for qualitative research (rather than quantitative) it can be very useful in allowing a researcher to identify general trends in the answers and -- importantly -- to help the researcher frame good research questions that are capable of being investigated in more detail
as John Levine already noted, the questionnaire seems somewhat confused as to whether it cares about routing issues (bogon lists, the Spamhaus DROP list etc) or spam filtering (bad domains, phishing feeds, botnet IPs etc etc)
it also asked if internally generated lists were used, but seemed curiously uninterested in anything other than if the answer to that was yes or no -- a missed opportunity I thought.
-- richard Richard Clayton
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755
In message <CAKcP59JPJT2LsUTrtgAsLeUTDsCrVBWq0_Cuas8LAvNAApQ7UQ@mail.gma il.com>, Anushah Hossain <anushah@icsi.berkeley.edu> writes
surprisingly, I haven't seen the request on any other lists that are (a) relevant and (b) open -- perhaps they and their project team are not especially well connected in this space :(
This is true. We were advised to share to RIPE and regional NOG mailing lists. Are there others you would have recommended?
ask the APWG to circulate the request to their members, and you might do the same with M3AAWG
as John Levine already noted, the questionnaire seems somewhat confused as to whether it cares about routing issues (bogon lists, the Spamhaus DROP list etc) or spam filtering (bad domains, phishing feeds, botnet IPs etc etc)
Hm, I think we are interested in quite the range of blacklists.
The issues will vary considerably between different types of list
Here is a table of what my colleagues are monitoring:
image.png
it also asked if internally generated lists were used, but seemed curiously uninterested in anything other than if the answer to that was yes or no -- a missed opportunity I thought.
What would you have recommended probing here?
you could have asked an open ended question which asked what they did, how they were built, why they were built in house and how significant they were.
I have been conducting interviews with those working in abuse prevention (even at some of the companies that have been mentioned upthread) to collect more specific anecdotes about how dynamic addressing has lowered the accuracy of certain feeds,
we've had DHCP for decades (and everyone knows the issues) ... are you sure they weren't discussing Carrier Grade NAT ?
for example, or how errors in geo-IP feeds affected them.
my own impression of these is that you get what you pay for ... but unless you are buying proxies I'm sceptical that large scale abuse filtering systems use this type of info as more than a one indicator amongst many. if you buying a proxy you may care a lot more ! Zachary Weinberg, Shinyoung Cho, Nicolas Christin, Vyas Sekar, and Phillipa Gill. How to Catch when Proxies Lie: Verifying the Physical Locations of Network Proxies with Active Geolocation. In Proceedings of the 2018 ACM Internet Measurement Conference (IMC'18). Boston, MA. October 2018. -- richard Richard Clayton Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755
anushah, next study: distribution of operator reactions to a simple request of an academic study questionnaire :) uncloak: the work icsi does is a couple decade trail of operationally relevant awesomeness. so, though i loathe survey quetionnaires, i answered directly as appended. i suspect your question was clear enough that even i understood it. randy --- From: Randy Bush <randy@psg.com> Subject: Re: [anti-abuse-wg] [Misc] Research project on blacklists To: Anushah Hossain <anushah@icsi.berkeley.edu> Date: Wed, 17 Jul 2019 08:02:31 -0700 i loathe surveys i use dialups.mail-abuse.org dnsbl.sorbs.net zen.spamhaus.org the one i really miss is the whitelist which seems to have died a few months back, dnswl.org i specifically whitelist sender addresses of *@bigglobe.ne.jp *@earthlink.net *@google.com *@hotmail.com *@teleport.com *@yahoo.co.jp *@yahoo.com randy
* Randy Bush <randy@psg.com> [2019-07-18 19:06]:
the one i really miss is the whitelist which seems to have died a few months back, dnswl.org
Hi Randy, dnswl.org is still operating, I'm also involved with them. Is there anything you need/miss? If so I'm able to relay it to the rest of the dnswl.org team. Reply off list if you want. Regards Sebastian -- GPG Key: 0x58A2D94A93A0B9CE (F4F6 B1A3 866B 26E9 450A 9D82 58A2 D94A 93A0 B9CE) 'Are you Death?' ... IT'S THE SCYTHE, ISN'T IT? PEOPLE ALWAYS NOTICE THE SCYTHE. -- Terry Pratchett, The Fifth Elephant
participants (5)
-
ac -
Anushah Hossain -
Randy Bush -
Richard Clayton -
Sebastian Wiesinger