Hi Sander and others On Wed, 20 Mar 2024 at 23:06, Sander Steffann <sander@steffann.nl> wrote:

Hi Denis,

...

So we just stay where we are, yet again, with the NCC sending out tens of thousands of emails every day to some of the almost a million email addresses in the database and you all continue to get spam sent to those harvested email addresses.

You seem to perceive a problem that not everybody else is perceiving. Maybe we should first see what the users of the database think of this instead of assuming there is a problem.

Not everybody has OCD. I know this is a long email. I know few people will read it. I don't expect anyone to respond to it as I talk about issues that no one else in this community will talk about, no matter how important they are. But as I say in my disclaimer, my mind is driven by detail. Things I see need to be said and archived, even if nothing is done. I would LOVE to hear what users of the database think. A 'perceived' problem, HHmmm. -The database design and data model are about 30 years old -the current java based software was first written about 12 years ago -the data model was built around storing unedited, partially un-formatted, human readable text blocks, with all indexed and machine parsable data a fudge on top of the text blocks -few centralised, distributed tools for processing data from the database, it's mostly diy -everytime someone asks for a new feature, like assigning an allocation, it needs open heart surgery -2023-04 has highlighted that the community has collectively forgotten what some of the data means or why rules were put in place -2023-04 has also shown that we have lost track of what a public registry is, who it serves and in what way -we have no documented business requirements for either the RIPE Database or a public registry -there is still a mindset problem with many users still seeing it as an operators tool and denying the need to serve other stakeholders -it has security issues -it has privacy issues -it has commercial confidentiality issues -geofeed references to external data (itself not covered by policy) allows any policy constraints on the database to be undermined -there is no simple way for data users to even query their own data, never mind manage it -2023-04 has also stressed that it is a very manual process to update the database which is not convenient for operators -it is not possible to have a full audit trail of all changes to your own data, despite the notif type attributes -the data model does not handle different business models adopted by some resource holders now, especially where resource holders are financial investors and some brokers operate as commercial IRs below the RIPE NCC -it operates in an evolving environment increasingly subject to regulatory control that is outside the control of this community, while the database itself remains static -no possibility of introducing new concepts like customised authentication or verified query users -the lack of simple concepts like inheritance results in massive duplication of data across millions of objects .... Just a quick list from memory of some problems surrounding the database. 'Hear from users'...Whenever I bring up any issues like this, I either hit a total wall of silence or I am hit with familiar negativity. NO one ever talks about the future of the RIPE Database in a positive way. Daniel said at the BOFF in Iceland, "It's time to stop tinkering around the edges of the RIPE Database". I thought that would be the trigger to start this positive discussion. At the BOFF there were lots of heads nodding in agreement. Any thought of a positive discussion was lost as quickly as people left the BOFF heading for the bar. A Task Force was set up, I thought, to document the business requirements for the RIPE Database and public registry. All they did was look backwards and write a document on the history of the database...many of us already knew that. The RIPE NCC has a mandate to operate a public registry. We don't even know what that means. We don't know what a public registry is in 2024. We don't know what civil society or regulators (that John Curran talks about in his speech at NANOG) expect from a public registry today or tomorrow. All we know is what we gave them yesterday. 2023-04 shows that some people don't care what they need tomorrow. The virtually static RIPE Database operates in an evolving environment that we have no control over. Assigning an allocation and new anti-spam rules have shown how difficult it is to manage change with this database. We are one feature, one policy change, one regulatory change away from this database breaking. Gaffa tape will not always fix it. We had two options on the table for assigning an allocation. No matter which option we chose, many of those diy tools people use will break. That is how fragile this system is. If we were to decide today to document the business requirements for the RIPE Database and public registry, consult with major stakeholders, have a conversation about how to iteratively redevelop the current system, then do it, we are talking several years before we have a new system resilient enough to operate in today's and tomorrow's environment and satisfy the requirements. You all know that this process will take years!!! But you will not even start the conversation. I was a database engineer and analyst, not an operator. You are right that I don't know what an LIR does on a daily basis. I have no idea how many staff you have, who does what, how much time is spent working with the database, how important (or not) it is to you. I know how the database works. I wrote the specifications and test suites for the whole system when we changed the software from C to Java. (Something else we had never documented. It was just in the collective minds of those who had worked on the software.) When I see problems and throw out ideas like having a centralised, complete audit trail, it is from my perspective of understanding how the database works. I hope that it starts a positive and professional conversation about options and possibilities. Of course the usual option of doing nothing is generally on the table (until the database really does break). But all I ever get back is total silence or sarcasm and negativity or sometimes even personal attacks and insults. Being the type of co-chair that I am, where I try to promote constructive conversation, sets me up to be a fall guy. There are some well known members of this community who take every opportunity to attack me personally. Sometimes publicly, sometimes privately. They love to highlight anything I say about internet operations that is wrong. They know that is not my area of expertise, but it doesn't stop them shooting at me. These are people who should know better. (And of course when I genuinely question someone's motivation I get banned from a mailing list for a month. There are double standards in this community.) So finally the bottom line: It really is time to start this conversation about the future of the RIPE Database and the public registry. You all know the process will take years to complete. Kicking the can down the road is not a professional approach. I am close to 70. I don't intend to be around much longer. (Hurray, I hear many of you saying.) You have to decide what you want to do.... and the final word: id nunc, aliquo tempore postea fit numquam or: do it now, sometime later becomes never cheers denis co-chair DB-WG ======================================================== DISCLAIMER Everything I said above is my personal, professional opinion. It is what I believe to be honest and true to the best of my knowledge. No one in this industry pays me anything. I have nothing to gain or lose by any decision. I push for what I believe is for the good of the Internet, in some small way. Nothing I say is ever intended to be offensive or a personal attack. Even if I strongly disagree with you or question your motives. Politicians question each other's motives all the time. RIPE discussion is often as much about politics and self interest as it is technical. I have a style of writing that some may not be familiar with, others sometimes use it against me. I also have OCD. It makes me see the world slightly differently to others. It drives my mind's obsessive need for detail. I can not change the way I express my detailed opinions. People may choose how to interpret them. ========================================================

Cheers, Sander

Hi Denis, Classic Tech Debt / delayed refactoring scenario. What we should seek to understand is the following:- - Maintainability - Platform: Is this genuinely becoming harder and costly to achieve? Is availability and security at risk? - Maintainability - Users: Is it easy for users to fulfil their objectives / obligations? - Purpose: Why does the database exist? - Effectiveness: Is it fulfilling the set objectives? Are datasets missing or even obsolete? Hopefully with answers to this we arrive at the next 1-2 questions:- - Does something need to be done? - Are we going to do anything about it? I recognise to some, maybe even all of these questions will be really obvious and probably already have answers brining up thoughts of "oh no, not this again" or "we've done this to death already, see previous analysis", however based on recent discussion across WGs, there does appear to be some big questions which need to be asked on the fundamentals. Things change and testing what that means is a good thing. Many thanks, Brian Brian Storey IP Network Capacity Planning Manager Tel: 0333 240 3481 Mobile: 07436 101 378 Email: Brian.Storey@gamma.co.uk This is an email from Gamma Telecom Ltd, trading as “Gamma”. The contents of this email are confidential to the ordinary user of the email address to which it was addressed. This email is not intended to create any legal relationship. No one else may place any reliance upon it, or copy or forward all or any of it in any form (unless otherwise notified). If you receive this email in error, please accept our apologies, we would be obliged if you would telephone our postmaster on +44 (0) 808 178 9652 or email postmaster@gamma.co.uk Gamma Telecom Limited, a company incorporated in England and Wales, with limited liability, with registered number 04340834, and whose registered office is at The Scalpel, 18th Floor, 52 Lime Street, London EC3M 7AF and whose principal place of business is at Kings House, Kings Road West, Newbury, Berkshire, RG14 5BY. -----Original Message----- From: db-wg <db-wg-bounces@ripe.net> On Behalf Of denis walker via db-wg Sent: Thursday, March 21, 2024 3:04 AM To: Sander Steffann <sander@steffann.nl> Cc: db-wg@ripe.net Subject: Re: [db-wg] Complying with Mail Provider Requirements When Sending Mail from Whois Hi Sander and others On Wed, 20 Mar 2024 at 23:06, Sander Steffann <sander@steffann.nl> wrote:

Hi Denis,

...

So we just stay where we are, yet again, with the NCC sending out tens of thousands of emails every day to some of the almost a million email addresses in the database and you all continue to get spam sent to those harvested email addresses.

You seem to perceive a problem that not everybody else is perceiving. Maybe we should first see what the users of the database think of this instead of assuming there is a problem.

Not everybody has OCD. I know this is a long email. I know few people will read it. I don't expect anyone to respond to it as I talk about issues that no one else in this community will talk about, no matter how important they are. But as I say in my disclaimer, my mind is driven by detail. Things I see need to be said and archived, even if nothing is done. I would LOVE to hear what users of the database think. A 'perceived' problem, HHmmm. -The database design and data model are about 30 years old -the current java based software was first written about 12 years ago -the data model was built around storing unedited, partially un-formatted, human readable text blocks, with all indexed and machine parsable data a fudge on top of the text blocks -few centralised, distributed tools for processing data from the database, it's mostly diy -everytime someone asks for a new feature, like assigning an allocation, it needs open heart surgery -2023-04 has highlighted that the community has collectively forgotten what some of the data means or why rules were put in place -2023-04 has also shown that we have lost track of what a public registry is, who it serves and in what way -we have no documented business requirements for either the RIPE Database or a public registry -there is still a mindset problem with many users still seeing it as an operators tool and denying the need to serve other stakeholders -it has security issues -it has privacy issues -it has commercial confidentiality issues -geofeed references to external data (itself not covered by policy) allows any policy constraints on the database to be undermined -there is no simple way for data users to even query their own data, never mind manage it -2023-04 has also stressed that it is a very manual process to update the database which is not convenient for operators -it is not possible to have a full audit trail of all changes to your own data, despite the notif type attributes -the data model does not handle different business models adopted by some resource holders now, especially where resource holders are financial investors and some brokers operate as commercial IRs below the RIPE NCC -it operates in an evolving environment increasingly subject to regulatory control that is outside the control of this community, while the database itself remains static -no possibility of introducing new concepts like customised authentication or verified query users -the lack of simple concepts like inheritance results in massive duplication of data across millions of objects .... Just a quick list from memory of some problems surrounding the database. 'Hear from users'...Whenever I bring up any issues like this, I either hit a total wall of silence or I am hit with familiar negativity. NO one ever talks about the future of the RIPE Database in a positive way. Daniel said at the BOFF in Iceland, "It's time to stop tinkering around the edges of the RIPE Database". I thought that would be the trigger to start this positive discussion. At the BOFF there were lots of heads nodding in agreement. Any thought of a positive discussion was lost as quickly as people left the BOFF heading for the bar. A Task Force was set up, I thought, to document the business requirements for the RIPE Database and public registry. All they did was look backwards and write a document on the history of the database...many of us already knew that. The RIPE NCC has a mandate to operate a public registry. We don't even know what that means. We don't know what a public registry is in 2024. We don't know what civil society or regulators (that John Curran talks about in his speech at NANOG) expect from a public registry today or tomorrow. All we know is what we gave them yesterday. 2023-04 shows that some people don't care what they need tomorrow. The virtually static RIPE Database operates in an evolving environment that we have no control over. Assigning an allocation and new anti-spam rules have shown how difficult it is to manage change with this database. We are one feature, one policy change, one regulatory change away from this database breaking. Gaffa tape will not always fix it. We had two options on the table for assigning an allocation. No matter which option we chose, many of those diy tools people use will break. That is how fragile this system is. If we were to decide today to document the business requirements for the RIPE Database and public registry, consult with major stakeholders, have a conversation about how to iteratively redevelop the current system, then do it, we are talking several years before we have a new system resilient enough to operate in today's and tomorrow's environment and satisfy the requirements. You all know that this process will take years!!! But you will not even start the conversation. I was a database engineer and analyst, not an operator. You are right that I don't know what an LIR does on a daily basis. I have no idea how many staff you have, who does what, how much time is spent working with the database, how important (or not) it is to you. I know how the database works. I wrote the specifications and test suites for the whole system when we changed the software from C to Java. (Something else we had never documented. It was just in the collective minds of those who had worked on the software.) When I see problems and throw out ideas like having a centralised, complete audit trail, it is from my perspective of understanding how the database works. I hope that it starts a positive and professional conversation about options and possibilities. Of course the usual option of doing nothing is generally on the table (until the database really does break). But all I ever get back is total silence or sarcasm and negativity or sometimes even personal attacks and insults. Being the type of co-chair that I am, where I try to promote constructive conversation, sets me up to be a fall guy. There are some well known members of this community who take every opportunity to attack me personally. Sometimes publicly, sometimes privately. They love to highlight anything I say about internet operations that is wrong. They know that is not my area of expertise, but it doesn't stop them shooting at me. These are people who should know better. (And of course when I genuinely question someone's motivation I get banned from a mailing list for a month. There are double standards in this community.) So finally the bottom line: It really is time to start this conversation about the future of the RIPE Database and the public registry. You all know the process will take years to complete. Kicking the can down the road is not a professional approach. I am close to 70. I don't intend to be around much longer. (Hurray, I hear many of you saying.) You have to decide what you want to do.... and the final word: id nunc, aliquo tempore postea fit numquam or: do it now, sometime later becomes never cheers denis co-chair DB-WG ======================================================== DISCLAIMER Everything I said above is my personal, professional opinion. It is what I believe to be honest and true to the best of my knowledge. No one in this industry pays me anything. I have nothing to gain or lose by any decision. I push for what I believe is for the good of the Internet, in some small way. Nothing I say is ever intended to be offensive or a personal attack. Even if I strongly disagree with you or question your motives. Politicians question each other's motives all the time. RIPE discussion is often as much about politics and self interest as it is technical. I have a style of writing that some may not be familiar with, others sometimes use it against me. I also have OCD. It makes me see the world slightly differently to others. It drives my mind's obsessive need for detail. I can not change the way I express my detailed opinions. People may choose how to interpret them. ========================================================

Cheers, Sander

-- To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg

Hi Denis,

On 21 Mar 2024, at 04:03, denis walker via db-wg <db-wg@ripe.net> wrote:

Hi Sander and others

On Wed, 20 Mar 2024 at 23:06, Sander Steffann <sander@steffann.nl> wrote:

...

Hi Denis,

...

So we just stay where we are, yet again, with the NCC sending out tens of thousands of emails every day to some of the almost a million email addresses in the database and you all continue to get spam sent to those harvested email addresses.

You seem to perceive a problem that not everybody else is perceiving. Maybe we should first see what the users of the database think of this instead of assuming there is a problem.

Not everybody has OCD.

I know this is a long email. I know few people will read it. I don't expect anyone to respond to it as I talk about issues that no one else in this community will talk about, no matter how important they are. But as I say in my disclaimer, my mind is driven by detail. Things I see need to be said and archived, even if nothing is done.

I would LOVE to hear what users of the database think. A 'perceived' problem, HHmmm.

-The database design and data model are about 30 years old -the current java based software was first written about 12 years ago -the data model was built around storing unedited, partially un-formatted, human readable text blocks, with all indexed and machine parsable data a fudge on top of the text blocks -few centralised, distributed tools for processing data from the database, it's mostly diy -everytime someone asks for a new feature, like assigning an allocation, it needs open heart surgery -2023-04 has highlighted that the community has collectively forgotten what some of the data means or why rules were put in place -2023-04 has also shown that we have lost track of what a public registry is, who it serves and in what way -we have no documented business requirements for either the RIPE Database or a public registry -there is still a mindset problem with many users still seeing it as an operators tool and denying the need to serve other stakeholders -it has security issues -it has privacy issues -it has commercial confidentiality issues -geofeed references to external data (itself not covered by policy) allows any policy constraints on the database to be undermined -there is no simple way for data users to even query their own data, never mind manage it -2023-04 has also stressed that it is a very manual process to update the database which is not convenient for operators -it is not possible to have a full audit trail of all changes to your own data, despite the notif type attributes -the data model does not handle different business models adopted by some resource holders now, especially where resource holders are financial investors and some brokers operate as commercial IRs below the RIPE NCC -it operates in an evolving environment increasingly subject to regulatory control that is outside the control of this community, while the database itself remains static -no possibility of introducing new concepts like customised authentication or verified query users -the lack of simple concepts like inheritance results in massive duplication of data across millions of objects ....

I'd like to make a clarification, in that any functional issues with the RIPE database should be kept separate from the technical implementation, which is the responsibility of the Database team. The current Java based software was written to make the database more maintainable, including adding new features, and this has been done very successfully over the past 12 years. The DB team is productive with the current software, i.e. we deliver features in a reasonable amount of time with a good quality level. Our code is open source so can be inspected and re-used. The other Software Engineering teams also use Java, so we can cooperate easily. We keep Java and MariaDB updated to the latest LTS (Long Term Support) versions so we can make use of latest features combined with a stable release. Operationally we deliver excellent uptime of the Whois service. We are very familiar with how the software behaves in production, in particular monitoring and handling failures. There may be *functional* issues with the RIPE database, but if the community agrees on changes to the data model, we are confident that we can deliver any resulting *technical* changes with the software we already have. Regards Ed Shryane RIPE NCC