email's disappeared from RDAP output
Hi everybody, I was just re-running some tests internally within bgp.tools and I've noticed that the RDAP responses from RIPE no longer include the email address of the person object. For example I used to (and expected) to get the email address in the object for https://rdap.db.ripe.net/entity/BC6775-RIPE, but it seems to have been removed from the output at some point. That does not appear to be a way to log in to authenticate trustworthiness of RDAP responses, is there any other way to get this information? I can see this information on the RIPE website ( https://apps.db.ripe.net/db-web-ui/lookup?source=ripe&key=BC6775-RIPE&type=person ) but I do not want to scrape that, and this information is incredibly useful for helping users onboard ( with a trustworthy email address ) to services like bgp.tools. Is this an intended change? Have I done something silly? All of the above ;) ? By looking at the release info I can see it was added in November (#1333) Thanks Ben
Good Afternoon Ben, This change was made as part of 1.109 release. We added that attribute into the redacted field and removed it from the RDAP entity. Email is considered to be personal data, so we must be consistent between our APIs and keep it filtered in all of them. If an email address is returned, this is accounted agains a daily limit according to the Acceptable Use Policy <https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-database-acceptable-use-policy> , so your IP will get blocked if you request too many objects which have personal attributes like email. We are open to discussion, so if there is an agreement in db-wg we will investigate how to include personal data in the RDAP response. Thanks Miguel
On 4 Jan 2024, at 17:39, Ben Cartwright-Cox via db-wg <db-wg@ripe.net> wrote:
Hi everybody,
I was just re-running some tests internally within bgp.tools and I've noticed that the RDAP responses from RIPE no longer include the email address of the person object. For example I used to (and expected) to get the email address in the object for https://rdap.db.ripe.net/entity/BC6775-RIPE, but it seems to have been removed from the output at some point.
That does not appear to be a way to log in to authenticate trustworthiness of RDAP responses, is there any other way to get this information? I can see this information on the RIPE website ( https://apps.db.ripe.net/db-web-ui/lookup?source=ripe&key=BC6775-RIPE&type=person ) but I do not want to scrape that, and this information is incredibly useful for helping users onboard ( with a trustworthy email address ) to services like bgp.tools.
Is this an intended change? Have I done something silly? All of the above ;) ?
By looking at the release info I can see it was added in November (#1333)
Thanks Ben
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
I'm fine with being held accountable to a daily limit ( I don't on board that many new accounts anyway per day ), however the main issue is now I no longer know of a way to get the email addresses ( and I guess be held accountable to a daily limit ) from something that looks ”machine readable” ( aka whois or RDAP ) Is there some kind of magic flag I can pass into the RDAP system to get personal data and be held against a limit? I'm not sure I know of a way also to get the email addresses out of the whois interface either, meaning I am simply left with scraping the website (something I really don't want to do) On Fri, Jan 5, 2024 at 10:38 AM Miguel Herran <mherran@ripe.net> wrote:
Good Afternoon Ben,
This change was made as part of 1.109 release. We added that attribute into the redacted field
and removed it from the RDAP entity.
Email is considered to be personal data, so we must be consistent between our APIs and keep
it filtered in all of them. If an email address is returned, this is accounted agains a daily limit
according to the Acceptable Use Policy , so your IP will get blocked if you request too many objects
which have personal attributes like email.
We are open to discussion, so if there is an agreement in db-wg we will investigate how to include
personal data in the RDAP response.
Thanks
Miguel
On 4 Jan 2024, at 17:39, Ben Cartwright-Cox via db-wg <db-wg@ripe.net> wrote:
Hi everybody,
I was just re-running some tests internally within bgp.tools and I've noticed that the RDAP responses from RIPE no longer include the email address of the person object. For example I used to (and expected) to get the email address in the object for https://rdap.db.ripe.net/entity/BC6775-RIPE, but it seems to have been removed from the output at some point.
That does not appear to be a way to log in to authenticate trustworthiness of RDAP responses, is there any other way to get this information? I can see this information on the RIPE website ( https://apps.db.ripe.net/db-web-ui/lookup?source=ripe&key=BC6775-RIPE&type=person ) but I do not want to scrape that, and this information is incredibly useful for helping users onboard ( with a trustworthy email address ) to services like bgp.tools.
Is this an intended change? Have I done something silly? All of the above ;) ?
By looking at the release info I can see it was added in November (#1333)
Thanks Ben
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
Hi, On Fri, Jan 05, 2024 at 11:37:29AM +0000, Ben Cartwright-Cox via db-wg wrote:
Is there some kind of magic flag I can pass into the RDAP system to get personal data and be held against a limit? I'm not sure I know of a way also to get the email addresses out of the whois interface either, meaning I am simply left with scraping the website (something I really don't want to do)
whois will give you mail addresses if you pass in "-B", or, more modern "--show-personal". The whois server conveniently understands "help" as well ;-) gert@moebius4:/home/gert$ telnet whois.ripe.net 43 ... help % NAME % whois query server % % DESCRIPTION % The following options are available: ... % --show-personal % Include PERSON and ROLE objects in results passing those options to the server might need some fiddling with -- to stop the local whois client from trying to interpret them (I use the RIPE "whois3" client which doesn't try to, but YMMV). Having never used RDAP, I am not sure if an option exists for it. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Sadly --show-personal over whois port 43 also does not return email for person objects: $ whois -h whois.ripe.net --show-personal BC6775-RIPE |& grep email: $ The only way to do this is I guess to scrape the website... On Fri, Jan 5, 2024 at 2:34 PM Gert Doering <gert@space.net> wrote:
Hi,
On Fri, Jan 05, 2024 at 11:37:29AM +0000, Ben Cartwright-Cox via db-wg wrote:
Is there some kind of magic flag I can pass into the RDAP system to get personal data and be held against a limit? I'm not sure I know of a way also to get the email addresses out of the whois interface either, meaning I am simply left with scraping the website (something I really don't want to do)
whois will give you mail addresses if you pass in "-B", or, more modern "--show-personal".
The whois server conveniently understands "help" as well ;-)
gert@moebius4:/home/gert$ telnet whois.ripe.net 43 ... help % NAME % whois query server % % DESCRIPTION % The following options are available: ... % --show-personal % Include PERSON and ROLE objects in results
passing those options to the server might need some fiddling with -- to stop the local whois client from trying to interpret them (I use the RIPE "whois3" client which doesn't try to, but YMMV).
Having never used RDAP, I am not sure if an option exists for it.
Gert Doering -- NetMaster -- have you enabled IPv6 on something today...?
SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
On Fri, Jan 05, 2024 at 02:49:43PM +0000, Ben Cartwright-Cox via db-wg wrote:
Sadly --show-personal over whois port 43 also does not return email for person objects:
$ whois -h whois.ripe.net --show-personal BC6775-RIPE |& grep email: $
Grep for 'e-mail' instead $ whois -h whois.ripe.net -- 'BC6775-RIPE -B' | grep e-mail > /dev/zero $ echo $? 0
Ben Cartwright-Cox via db-wg wrote on 05/01/2024 14:49:
Sadly --show-personal over whois port 43 also does not return email for person objects:
$ whois-h whois.ripe.net --show-personal BC6775-RIPE |& grep email: $
The only way to do this is I guess to scrape the website...
oops, pebkak. Try this instead:
% whois -h whois.ripe.net " --show-personal -B BC6775-RIPE" | grep e-mail
Nick
Hi, On Fri, Jan 05, 2024 at 02:49:43PM +0000, Ben Cartwright-Cox wrote:
Sadly --show-personal over whois port 43 also does not return email for person objects:
$ whois -h whois.ripe.net --show-personal BC6775-RIPE |& grep email: $
I mixed up options, sorry for that. "-B" works, --show/-no-personal seem to be for filtering whole object classes (-r/-R). -B is documented there, as % -B, --no-filtering % Disables the filtering of "notify:" and "e-mail:" attributes. so... whois3 -h whois.ripe.net -B BC6775-RIPE |grep e-mail e-mail: ripe-whois@benjojo... Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
pebkak indeed, thanks everyone! On Fri, Jan 5, 2024 at 2:57 PM Gert Doering <gert@space.net> wrote:
Hi,
On Fri, Jan 05, 2024 at 02:49:43PM +0000, Ben Cartwright-Cox wrote:
Sadly --show-personal over whois port 43 also does not return email for person objects:
$ whois -h whois.ripe.net --show-personal BC6775-RIPE |& grep email: $
I mixed up options, sorry for that. "-B" works, --show/-no-personal seem to be for filtering whole object classes (-r/-R).
-B is documented there, as
% -B, --no-filtering % Disables the filtering of "notify:" and "e-mail:" attributes.
so...
whois3 -h whois.ripe.net -B BC6775-RIPE |grep e-mail e-mail: ripe-whois@benjojo...
Gert Doering -- NetMaster -- have you enabled IPv6 on something today...?
SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
I am sorry to say that currently “no” there is no any option to return the personal data in RDAP. We want to be consistent across the interfaces so since the beginning returning email there was inconsistent and we don’t want to block users due to inadvertently reaching the daily limit. Currently there is no mechanism in the RDAP spec to allow returning personal data, but we can investigate how to do this, in collaboration with the community.
On 5 Jan 2024, at 11:38, Miguel Herran via db-wg <db-wg@ripe.net> wrote:
Good Afternoon Ben,
This change was made as part of 1.109 release. We added that attribute into the redacted field and removed it from the RDAP entity.
Email is considered to be personal data, so we must be consistent between our APIs and keep it filtered in all of them. If an email address is returned, this is accounted agains a daily limit according to the Acceptable Use Policy <https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-database-acceptable-use-policy> , so your IP will get blocked if you request too many objects which have personal attributes like email.
We are open to discussion, so if there is an agreement in db-wg we will investigate how to include personal data in the RDAP response.
Thanks Miguel
On 4 Jan 2024, at 17:39, Ben Cartwright-Cox via db-wg <db-wg@ripe.net> wrote:
Hi everybody,
I was just re-running some tests internally within bgp.tools and I've noticed that the RDAP responses from RIPE no longer include the email address of the person object. For example I used to (and expected) to get the email address in the object for https://rdap.db.ripe.net/entity/BC6775-RIPE, but it seems to have been removed from the output at some point.
That does not appear to be a way to log in to authenticate trustworthiness of RDAP responses, is there any other way to get this information? I can see this information on the RIPE website ( https://apps.db.ripe.net/db-web-ui/lookup?source=ripe&key=BC6775-RIPE&type=person ) but I do not want to scrape that, and this information is incredibly useful for helping users onboard ( with a trustworthy email address ) to services like bgp.tools.
Is this an intended change? Have I done something silly? All of the above ;) ?
By looking at the release info I can see it was added in November (#1333)
Thanks Ben
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
Hi, On Fri, Jan 05, 2024 at 03:30:23PM +0100, Miguel Herran via db-wg wrote:
I am sorry to say that currently ???no??? there is no any option to return the personal data in RDAP. We want to be consistent across the interfaces so since the beginning returning email there was inconsistent and we don???t want to block users due to inadvertently reaching the daily limit.
whois always had the corresponding options, so unilateraly breaking this for RDAP without prior discussion with the WG (at least I did not see anything) might not have been such a good plan after all... Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hi Gert,
On 5 Jan 2024, at 15:36, Gert Doering via db-wg <db-wg@ripe.net> wrote:
Hi,
On Fri, Jan 05, 2024 at 03:30:23PM +0100, Miguel Herran via db-wg wrote:
I am sorry to say that currently ???no??? there is no any option to return the personal data in RDAP. We want to be consistent across the interfaces so since the beginning returning email there was inconsistent and we don???t want to block users due to inadvertently reaching the daily limit.
whois always had the corresponding options, so unilateraly breaking this for RDAP without prior discussion with the WG (at least I did not see anything) might not have been such a good plan after all...
This change was made to fix a bug in RDAP where e-mail was returned without any daily limit accounting. We removed e-mail to avoid blocking clients querying for entities (as RDAP does not give a client options for filtering responses, the server must decide what to send). This change was documented in the Whois 1.109 release notes: "Removed organisation "e-mail" from RDAP to match Whois and added to redaction list (#1333)" I also included it in my operational update in the DB-WG session at RIPE87. We can always make improvements to the release process to be clearer and more transparent. For example, we can include more detail on each change. Regards Ed Shryane RIPE NCC
Hello Ben, Firstly apologies that you were affected by this change. We did this to fix a bug in RDAP that we discovered while implementing the Redacted feature: https://datatracker.ietf.org/doc/draft-ietf-regext-rdap-redacted/ We found that RDAP always returned e-mail attributes without any daily limit accounting, which could have allowed a client to collect every e-mail in the database without limit (including organisation, person, role, mntner entities). This bug had to be fixed to protect personal data. We had a choice either to keep e-mail in the RDAP response and add daily limit accounting, or remove e-mail and include a redaction in the response. We decided to remove e-mail because it's consistent with the other Whois interfaces that filter responses by default. Secondly because RDAP does not use query flags, there is no way for a client to opt-out of receiving personal data. The server decides what to send, and the client can be blocked whether it wants personal data or not. If the DB-WG community prefers to include e-mail by default in RDAP entity responses, and turn on daily limit accounting, we will do this. Regards Ed Shryane RIPE NCC
On 4 Jan 2024, at 17:39, Ben Cartwright-Cox via db-wg <db-wg@ripe.net> wrote:
Hi everybody,
I was just re-running some tests internally within bgp.tools and I've noticed that the RDAP responses from RIPE no longer include the email address of the person object. For example I used to (and expected) to get the email address in the object for https://rdap.db.ripe.net/entity/BC6775-RIPE, but it seems to have been removed from the output at some point.
That does not appear to be a way to log in to authenticate trustworthiness of RDAP responses, is there any other way to get this information? I can see this information on the RIPE website ( https://apps.db.ripe.net/db-web-ui/lookup?source=ripe&key=BC6775-RIPE&type=person ) but I do not want to scrape that, and this information is incredibly useful for helping users onboard ( with a trustworthy email address ) to services like bgp.tools.
Is this an intended change? Have I done something silly? All of the above ;) ?
By looking at the release info I can see it was added in November (#1333)
Thanks Ben
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
Dear colleagues,
On 8 Jan 2024, at 11:44, Edward Shryane via db-wg <db-wg@ripe.net> wrote: ... We had a choice either to keep e-mail in the RDAP response and add daily limit accounting, or remove e-mail and include a redaction in the response. We decided to remove e-mail because it's consistent with the other Whois interfaces that filter responses by default. Secondly because RDAP does not use query flags, there is no way for a client to opt-out of receiving personal data. The server decides what to send, and the client can be blocked whether it wants personal data or not. ...
I checked the RDAP query logs and found that only about 10-20 client IPs would be blocked daily (out of 100K's total client IPs) if we enabled daily accounting on RDAP entity responses. Therefore I propose that we restore e-mail to RDAP entity responses and enable daily limit accounting to protect personal data. Any RDAP client that is making /entity/ requests must comply with the daily limit according to the AUP: https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-datab... We will continue to filter e-mail in entities in RDAP /ip/ and /autnum/ responses, so that clients do not get blocked just by querying for resources (i.e. if you want an unfiltered entity, make an /entity/ request separately). If there are no objections, I propose to include this change in the next Whois release. Please let me know your feedback. Regards Ed Shryane RIPE NCC
Sounds good to me
Hi Ed, I do feel like there should be some way to opt out from receiving personal data and avoid rate limits. Would it maybe make sense to have an alternative RDAP service with rate limits at another "base URL"? (such as https://rdap-rl.db.ripe.net/[...] or whatever) This is assuming that it is not possible to have something equivalent to flags. I do still feel like there should probably be some more discussion in general for how to get RDAP closer to being able to do all the normal things WHOIS can. -Cynthia On Tue, Jan 9, 2024 at 4:39 PM Edward Shryane via db-wg <db-wg@ripe.net> wrote:
Dear colleagues,
On 8 Jan 2024, at 11:44, Edward Shryane via db-wg <db-wg@ripe.net> wrote: ... We had a choice either to keep e-mail in the RDAP response and add daily limit accounting, or remove e-mail and include a redaction in the response. We decided to remove e-mail because it's consistent with the other Whois interfaces that filter responses by default. Secondly because RDAP does not use query flags, there is no way for a client to opt-out of receiving personal data. The server decides what to send, and the client can be blocked whether it wants personal data or not. ...
I checked the RDAP query logs and found that only about 10-20 client IPs would be blocked daily (out of 100K's total client IPs) if we enabled daily accounting on RDAP entity responses.
Therefore I propose that we restore e-mail to RDAP entity responses and enable daily limit accounting to protect personal data.
Any RDAP client that is making /entity/ requests must comply with the daily limit according to the AUP: https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-datab...
We will continue to filter e-mail in entities in RDAP /ip/ and /autnum/ responses, so that clients do not get blocked just by querying for resources (i.e. if you want an unfiltered entity, make an /entity/ request separately).
If there are no objections, I propose to include this change in the next Whois release. Please let me know your feedback.
Regards Ed Shryane RIPE NCC
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
Hello Cynthia,
On 9 Jan 2024, at 16:56, Cynthia Revström <me@cynthia.re> wrote:
Hi Ed,
I do feel like there should be some way to opt out from receiving personal data and avoid rate limits.
Would it maybe make sense to have an alternative RDAP service with rate limits at another "base URL"? (such as https://rdap-rl.db.ripe.net/[...] or whatever) This is assuming that it is not possible to have something equivalent to flags.
I prefer to build this facility into the protocol and not use alternative service names if we can. I will discuss with the other RIRs how we can best do this.
I do still feel like there should probably be some more discussion in general for how to get RDAP closer to being able to do all the normal things WHOIS can.
I agree, hopefully we have time at the interim session next week to continue this.
-Cynthia
Regards Ed Shryane RIPE NCC
Hi, On Tue, Jan 09, 2024 at 04:39:01PM +0100, Edward Shryane via db-wg wrote:
Any RDAP client that is making /entity/ requests must comply with the daily limit according to the AUP: https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-datab...
We will continue to filter e-mail in entities in RDAP /ip/ and /autnum/ responses, so that clients do not get blocked just by querying for resources (i.e. if you want an unfiltered entity, make an /entity/ request separately).
Sounds good to me. Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Michael Emmer Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Dear Ben, A reminder that since the Whois 1.111 release last week, Whois now includes e-mail in RDAP entity responses, subject to the AUP daily limit. I will investigate separately how we can allow RDAP clients to indicate whether they want personal data in responses or not. Regards Ed Shryane RIPE NCC
On 9 Jan 2024, at 16:39, Edward Shryane via db-wg <db-wg@ripe.net> wrote:
Dear colleagues,
On 8 Jan 2024, at 11:44, Edward Shryane via db-wg <db-wg@ripe.net> wrote: ... We had a choice either to keep e-mail in the RDAP response and add daily limit accounting, or remove e-mail and include a redaction in the response. We decided to remove e-mail because it's consistent with the other Whois interfaces that filter responses by default. Secondly because RDAP does not use query flags, there is no way for a client to opt-out of receiving personal data. The server decides what to send, and the client can be blocked whether it wants personal data or not. ...
I checked the RDAP query logs and found that only about 10-20 client IPs would be blocked daily (out of 100K's total client IPs) if we enabled daily accounting on RDAP entity responses.
Therefore I propose that we restore e-mail to RDAP entity responses and enable daily limit accounting to protect personal data.
Any RDAP client that is making /entity/ requests must comply with the daily limit according to the AUP: https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-datab...
We will continue to filter e-mail in entities in RDAP /ip/ and /autnum/ responses, so that clients do not get blocked just by querying for resources (i.e. if you want an unfiltered entity, make an /entity/ request separately).
If there are no objections, I propose to include this change in the next Whois release. Please let me know your feedback.
Regards Ed Shryane RIPE NCC
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
participants (7)
-
Ben Cartwright-Cox -
Cynthia Revström -
Edward Shryane -
Gert Doering -
Job Snijders -
Miguel Herran -
Nick Hilliard