Re: [db-wg] Changes to the RIPE Database Acceptable Use Policy and Daily Limit Accounting
Hello, this discussion is annoying and in the circle. You're still trying to give the term "IP address" a different meaning than it has. IP address is IP addres. Period. It's not a subnet of any size, as you're trying to say. AUP clearly talks about IP address, not about end-user site etc. You put that in there arbitrarily. You are trying to give a strange interpretation of a standard technical term, which, however, has a clearly defined meaning. To defend yourself, you're thinking of hypothetical scenarios. But these can similarly occur with both IPv4 and IPv6. But as as a result, using IPv6 is a disadvantage now, you artificially penalize IPv6 more than IPv4. I can have IPv4 LAN with /24 and I'll no face any issue, but I'll have on LAN even with /64 IPv6. It's just stupid what you're doing. In fact, what the NCC tells us here is to turn off IPv6 and we'll be fine. And as it was already mentioned by Denis at db-wg, there're much more ways to scrap personal data if someone really wants to do that. But your pseudo-protection code is only throwing the baby out with the bath water. And create unnecessary problems for users on IPv6, thanks to developers incompetence (or laziness?). I understand, some employees and RIPE NCC doesn't understand IPv6 basics and make no difference between IP address and IP subnet (site). But there's a difference. Again. IPv6 address has 128 bits, not 64 bits, which NCC staff uses for AUP accounting. I raise question to services WG: How it's possible that an employees of RIPE NCC interprets standard terminology in such strange way and bends the written rules in the direction they are not written? Who approved the blocking of the entire subnet, when even AUP exactly says that IP addresses should be blocked in case of violation? Who is responsible for this creativity? I would like to hear the answers, because it seems that there is anarchy in NCC and the developers implements what they want, not what they should (with respect to published rules/documents). - Daniel On 8/6/24 3:45 PM, Edward Shryane wrote:
Hello Daniel,
On 6 Aug 2024, at 10:01, Daniel Suchy <danny@danysek.cz> wrote:
Hello,
On 8/5/24 3:32 PM, Edward Shryane wrote:
The current system is a compromise between allowing queries containing personal data, and complying with the Acceptable Use Policy: https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-datab...
The current system is a design bug, not a compromise.
It was a deliberate choice to account by /64 prefix size for IPv6 (the smallest subnet size) and /32 for IPv4 (the smallest prefix size). While it’s true that an IPv6 end user site could contain multiple users, this is the same for IPv4 where you could have a single public NAT address with many users behind it. However, within an IPv6 /64 end user site, the user is free to configure their global address. If we account for greater than a /64 then it becomes trivial for a user to change their host address and avoid the limit accounting. Changing the accounting to limit both by /128 and /64 prefixes complicates the implementation and operation of the system and the user must still comply with the AUP daily limit of 1,000 objects (it’s just harder to identify them).
To avoid getting blocked, a user (one or many) must reduce the queries for objects containing personal information, no matter how the accounting is done.
Acceptable Use Policy clearly defines limits per IP address (without distinguishing whether IPv4 or IPv6).
Using a /64 as the smallest subnet size is a reasonable compromise for the implementation, given the huge address space and how trivial it is to change host address, to avoid the accounting. Furthermore, if we account for every single /128 address, this would allow malicious actors to DDoS the accounting by simply sending each query from a new, different IPv6 address.
Your implementation at this time blocks whole /64 subnet in case of IPv6, not only single address violating AUP.
See above. A single user can be assigned a whole /64 subnet.
The limit is 1,000 objects that could contain personal data, which is not normally reached by most users (< 0.02%), and it is clear what can be done if this is exceeded.
The limit value isn't problem as well as limiting the number of queries per IP address. But the developers probably wanted to make their life easier and implemented the AUP incorrectly.
Blocking entire subnets as an initial reaction simply isn't the spirit of AUP. IPv6 address has 128 bits, not only 64 you're using - whatever the "reason" is.
We do not want to make it easier to avoid the AUP limit.
Rather than re-write the accounting code, can the community review why objects containing personal data is returned by default? Can we make "-r" the default?
If code contains bug (and that's now quite clear), it needs to be fixed regardless of the data returned with the default settings.
Again, there was a deliberate decision to account by /64 and it’s not an implementation bug.
I personally don't consider as a bad idea not to display personal data at all (have "-r" as default), it's important that the abuse contact is always displayed (and it is, even with "-r"). But that's different topic.
- Daniel
Changing the accounting implementation does not fix the root cause in terms of why users are getting blocked. By default, RIPE Database queries are returning referenced personal data that the user did not ask for. Rather than change the accounting, we can change the default query behaviour. Users should be accountable for the personal data that they requested.
In conclusion, there are good reasons for the current implementation, but if there is consensus for reviewing how AUP accounting is done, then we will prioritise it.
Regards Ed Shryane RIPE NCC
Hi,
I raise question to services WG:
How it's possible that an employees of RIPE NCC interprets standard terminology in such strange way and bends the written rules in the direction they are not written?
Who approved the blocking of the entire subnet, when even AUP exactly says that IP addresses should be blocked in case of violation? Who is responsible for this creativity?
I would like to hear the answers, because it seems that there is anarchy in NCC and the developers implements what they want, not what they should (with respect to published rules/documents).
I think this could be phrased a little more constructively. I believe it is pretty common to rate-limit based on the /32 for IPv4 and the /64 for IPv6, this isn’t something the NCC has invented. Personally I think pragmatism might win out against literal interpretation — especially as this doesn’t appear to be something that many users are noticing. We could have an endless discussion on why IP addresses are locators rather than identifiers, so poor metrics for preventing abuse, but without enforcing a login to query the database, they’re all we’ve got. The AUP states that an individual IP address cannot request > 1,000 personal data sets in 24 hours. It does not state that every IP address can query up to 1,000 personal data sets. In my opinion, that doesn’t prohibit the database from proactively defending itself by blocking a larger related prefix, especially referring to the footnote in the AUP on the basis of ‘reasonable use.' The subsequent discussion appears to have two ways forward: 1. Making ‘-r’ the default so personal data is not returned by default. 2. Tweaking the rate limiting so that the /128 is blocked initially, but blocking the /64 if there are more rapid queries from the same /64. Both of those seem like items that could become NWIs if the DB-WG agrees to them, noting that Ed has commented that the latter could involve a greater amount of work, so other things might have higher priority. Cheers, Rob [Speaking for myself, but paying attention because the NCC Services WG was tagged.]
Rob Evans wrote on 06/08/2024 17:48:
I believe it is pretty common to rate-limit based on the /32 for IPv4 and the /64 for IPv6, this isn’t something the NCC has invented.
the rationale for /64 is that ipv6 privacy addresses will cause the source IP address to change for each successive query, i.e. on a standard SLAAC network, the limit of 1000 person objects per-IP-address-per-day won't apply by default. Applying the rate limit on the entire /64 for ipv6 closes off this rather embarrassing loophole. Does the EB even need to approve text to clarify this? It's a standard and completely reasonable approach to rate-limiting. Nick
Hi, On Tue, Aug 06, 2024 at 06:15:28PM +0100, Nick Hilliard wrote:
Rob Evans wrote on 06/08/2024 17:48:
I believe it is pretty common to rate-limit based on the /32 for IPv4 and the /64 for IPv6, this isn???t something the NCC has invented.
the rationale for /64 is that ipv6 privacy addresses will cause the source IP address to change for each successive query, i.e. on a standard SLAAC
Uh, when did *this* happen? I see privacy addresses change "every few hours" and "on reconnect", but not for every single connect (tested from MacOS and Linux, curling http://v6only.v6.de/ip.html 10 times in sequence, no change in IPv6 address displayed). If this happens on mobile phones (which are known to do funny things), the point is a bit moot for the discussion here... Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Ingo Lalla, Karin Schuler, Sebastian Cler Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi, On 8/6/24 6:48 PM, Rob Evans wrote:
I think this could be phrased a little more constructively. > I believe it is pretty common to rate-limit based on the /32 for IPv4 and the /64 for IPv6, this isn’t something the NCC has invented. Personally I think pragmatism might win out against literal interpretation — especially as this doesn’t appear to be something that many users are noticing.
It's from other perspective also an attack vector. Just come to conference with IPv6-enabled network (like RIPEMTG), send large ammount of queries... and everone present will notice this. It's simply a DoS condition. And it can be abused on such places, where many people meet. In most cases, every connected station will not have its own dedicated /64 subnet. They will share single subnet. If you block on IPv4 single host and on IPv6 whole network (LAN) represented by /64, it's only kind of security through obscurity. And no one sane will actually do this if he takes security seriously. And an approach that only disadvantages IPv6. How many times have we witnessed that the initial recommendation in any problem was to turn off IPv6? We really want it this way? If we embrace blocking /64 like this as a great defense, then we will achieve exactly that. Just because each small incident breaks the entire local network from user perspective. Even if the affected user himself is not the cause.
We could have an endless discussion on why IP addresses are locators rather than identifiers, so poor metrics for preventing abuse, but without enforcing a login to query the database, they’re all we’ve got.
It was mentioned - if someone wants to circumvent the rules, they will find a way. Always. But making life easier at server side with the rule "lets block the entire LAN on IPv6" will do more harm than good. And it's just a false feeling of some security. For example - it's easy to run VPS somewhere for a few bucks... using an API calls, perform few queries... detroy it and so on and on. This is a technique that a real attacker will use in practice. Because of course even real attacker knows that some AUP limits exist and will be really motivated to hide his activity. While case rapid address changes within single /64 on IPv6 are hypothetical and speculative. Because it will be quickly visible. Does anyone really think that the attacker wants to be caught quickly?
The AUP states that an individual IP address cannot request > 1,000 personal data sets in 24 hours. It does not state that every IP address can query up to 1,000 personal data sets. In my opinion, that doesn’t prohibit the database from proactively defending itself by blocking a larger related prefix, especially referring to the footnote in the AUP on the basis of ‘reasonable use.'
In particular *real* case, it happened that accesses from one station actually disabled the service for all users of that LAN over IPv6, just due to unfiltered queries passed to grep. It wouldn't happen with IPv4 (as each user in particular LAN has own public IPv4). Affected users around didn't know what was going on, they weren't doing anything wrong. But the discussion about bad design, not about particular issue (it only poited out the problem). And the goal isn't to help scrappers, but the defense must be solved meaningfully, not "simply". The implementation of IPv6 sometimes brings new issues (in this case confusion of terms address / site). Use of IPv6 is still quite small and so similar problems generally don't manifest much. As mentioned above, that "protection" can be in other hand used as an DoS attack. What is better - a non-functioning service or a false feeling that data were protected? (of course they weren't)
The subsequent discussion appears to have two ways forward: 1. Making ‘-r’ the default so personal data is not returned by default. 2. Tweaking the rate limiting so that the /128 is blocked initially, but blocking the /64 if there are more rapid queries from the same /64.
Both of those seem like items that could become NWIs if the DB-WG agrees to them, noting that Ed has commented that the latter could involve a greater amount of work, so other things might have higher priority.
Well, so first we will implement technical nonsense and then we will deal with the complex bureaucracy to solve the problem as it arose? :-) There may be also third option - just stop to offering only personal data after limit this type of query is hit. It will certainly break not so many things. Queries not containing personal data aren't subject of limits (sure. there're some slight limits always). But I myself prefer the option where as little personal data as possible is implicitly returned. In general, in most cases, it is enough to get abuse contact, which also isn't subject of limits. With the current implementation of the limits, we also block access to abuse contacts. Is that really the goal? Sure, it can also be done in another way, but again - it's unnecessary complication. It's not good again. Again we return to the fact that it is better to use IPv4 in order not to risk similar "problems". We're throwing out the baby with the bathwater. IPv6 is the baby here. And the features of IPv6 are used as an argument for making things obscure and sometimes worse than with IPv4. We're inventing theoretical problems to support such obscurities. Actually, we're only adding additional arguments to IPv6 haters. But... why? But it won't solve the problems we want to solve. Here we primarily deal with the protection of personal data. But the real attack vector eludes us. The real problem will not arise from single /64. On the contrary, I think that database scrapping is simply happening and is unnoticed... despite AUP. Sometimes it's good to think about how the attacker would probably do it. Daniel
On 6. Aug 2024, at 20:19, Daniel Suchy via ncc-services-wg <ncc-services-wg@ripe.net> wrote:
[...] It's simply a DoS condition. And it can be abused on such places, where many people meet. In most cases, every connected station will not have its own dedicated /64 subnet. They will share single subnet.
If you block on IPv4 single host and on IPv6 whole network (LAN) represented by /64, it's only kind of security through obscurity. And no one sane will actually do this if he takes security seriously.
In probably the majority of cases these days, end user devices or even whole companies that use IPv4 are behind NAT. So by blocking a single IPv4 address you will in reality block a lot more than just a single user.
For example - it's easy to run VPS somewhere for a few bucks... using an API calls, perform few queries... detroy it and so on and on. This is a technique that a real attacker will use in practice. Because of course even real attacker knows that some AUP limits exist and will be really motivated to hide his activity.
While case rapid address changes within single /64 on IPv6 are hypothetical and speculative. Because it will be quickly visible. Does anyone really think that the attacker wants to be caught quickly?
Even a cheap VPS typically gets a whole /64 per host (at least in my experience). So, the possibility to rotate through IPv6 addresses is actually cheap, easy, and far from hypothetical. From that POV it makes perfect sense to me to block whole /64s and _not_ bother with individual /128s. Cheers -Andi
Hi, On 8/7/24 9:06 AM, Andreas Härpfer wrote:
For example - it's easy to run VPS somewhere for a few bucks... using an API calls, perform few queries... detroy it and so on and on. This is a technique that a real attacker will use in practice. Because of course even real attacker knows that some AUP limits exist and will be really motivated to hide his activity.
While case rapid address changes within single /64 on IPv6 are hypothetical and speculative. Because it will be quickly visible. Does anyone really think that the attacker wants to be caught quickly?
Even a cheap VPS typically gets a whole /64 per host (at least in my experience). So, the possibility to rotate through IPv6 addresses is actually cheap, easy, and far from hypothetical.
From that POV it makes perfect sense to me to block whole /64s and _not_ bother with individual /128s.
But this is still not a solution for situations where the machines used for scraping personal change rapidly. The attacker with knowledge the AUP limits (which are public) will simply change source /64 with sufficient cadence just as it will change the IPv4 source. This is also a real experience with DDoS attacks that targeted the application. Addresses change so quickly and there are so many source networks that such kind of blocking is essentially ineffective. A real attacker who aims to obtain personal data from RIPE will unsurprisingly proceed similarly. - Daniel
Hi, On Tue, Aug 06, 2024 at 04:23:32PM +0200, Daniel Suchy via db-wg wrote:
I would like to hear the answers, because it seems that there is anarchy in NCC and the developers implements what they want, not what they should (with respect to published rules/documents).
We do not micromanage the NCC. We trust them to come up with reasonable decisions for day-to-day business. Sometimes we disagree on what "reasonable" might have been, but then we talk, in (ahem) reasonable terms, and try to bring across our point in friendly and polite form. Your style of discussion leaves room for improvement. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Ingo Lalla, Karin Schuler, Sebastian Cler Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi Gert,
We do not micromanage the NCC. We trust them to come up with reasonable decisions for day-to-day business.
As a new RIPE NCC EB member I appreciate your trust :)
Sometimes we disagree on what "reasonable" might have been, but then we talk, in (ahem) reasonable terms, and try to bring across our point in friendly and polite form.
And I appreciate this even more! Yes, talking with each other in a friendly and polite way is definitely the recommended approach. Daniel: please respect that not everything can be hard-coded in rules and policies, and respect the people who make judgement calls in good faith. You may not agree with everything, but there is no need to imply malice or stupidity in your feedback. That is not productive, and not ok. When disagreeing with what you observe, please first ask what the reason was for people to make the choices they did, take that feedback seriously, and work together to improve the situation for all involved. That is the way this bottom-up community is supposed to work. Cheers, Sander Disclaimer: I’m speaking as an individual, not on behalf of the board. The standpoints expressed are my own. They only reflect how I as a person see things.
Hello, On 8/7/24 3:30 PM, Sander Steffann wrote:
Daniel: please respect that not everything can be hard-coded in rules and policies, and respect the people who make judgement calls in good faith. You may not agree with everything, but there is no need to imply malice or stupidity in your feedback. That is not productive, and not ok.
When disagreeing with what you observe, please first ask what the reason was for people to make the choices they did, take that feedback seriously, and work together to improve the situation for all involved. That is the way this bottom-up community is supposed to work.
In this case, there was *no* discussion from the bottom before limits were implemented. Aplication of limits to single /64 subnet instead of IP address was not even notified [1], even if we are talking about a relatively fundamental functional difference here. In the announcement [1], theres also clearly stated, that the queries will be counted based on IP addresses (again). AUP talks about IP addresses. There's no mention about subnets, end sites etc. That mental exercise followed my complaint. And we're failing in basic (technical!) terminology here. I was happy with form which was announced on maling list in May. And I was really surprised few days back, when I noticed something else is really implemented. And I tried to solve it first with NCC directly, but without success. The form of my messages may not be perfect - but the primary failure, in my opinion, happened somewhere else. This is not how things are done in the bottom-up driven community. - Daniel [1] https://mailman.ripe.net/archives/list/db-wg@ripe.net/thread/VNZW422LST24X6I...
participants (6)
-
Andreas Härpfer -
Daniel Suchy -
Gert Doering -
Nick Hilliard -
Rob Evans -
Sander Steffann