Dear Elvis, Your email touches on multiple issues. I‘d like to address some of them now, with the understanding that we will share more detailed information soon. For now, I can provide a general overview on what the situation with wait times is, and how we plan to tackle this issue in the coming weeks. First, a quick historical comparison for context. Back in 2013, the RIPE NCC had around 9,000 LIRs, and we processed a mere 150 policy transfers in the whole year, i.e. approximately 12-13 transfer requests a month. We currently have about 23,700 LIRs, which is 2.5x the LIRs in 2013. Last month, we processed over 400 policy transfers. This is 30x the number of transfer requests we processed when you were part of the Registry Services team. And remember that there are all of the other requests on top of this. In 2020, we averaged around 1,600 resources-related requests per month. However, from November to December there was a threefold increase compared to the rest of the year. It is hard to avoid an increased wait time when this happens, and that’s why we prefer to be upfront when we expect delays, so members can plan around them. While these statistics provide a glimpse into how things have changed over the past few years, I would like to reiterate that it is not just the number of tickets but also the complexity of requests that has increased. We report on this regularly at RIPE Meetings. Despite these challenges, Registry Services strives to handle incoming requests efficiently. In order to improve our ticket handling times, we have: - Hired new staff, who are still being trained, but will help address the workload once they are up to speed. - Taken on temporary staff to enter registration details of resource holders into our system which will allow us to automate certain due diligence checks. We will publish a RIPE Labs article about these changes in the coming weeks. Some additional clarifications regarding internal targets: - Every ticket that doesn't get a response by 18:00 (UTC +2) the following working day is counted as a ‘miss’ against our internal targets. - Our notification emails do not count against these targets (we don’t see much point in gaming our internal KPIs). - A gentle reminder that Saturdays and Sundays are not working days, so requests received on a Friday are likely to be addressed the next business day, i.e. the Monday after the weekend. We can appreciate that it is frustrating to wait on tickets. However, while we’re still training additional staff and automating systems, we need to rely on your patience and ask that you allow us time to ensure requests are not just handled quickly, but also accurately. Since we know from experience that the number of requests increases dramatically in the last quarter of the year, I would like to repeat our request for members to kindly not wait until the last minute to send in a request, and to expect longer handling times in the last quarter of 2021. Once I have a more detailed overview of our ticket history, I’ll share it on this list, along with how we plan to further improve our services. I am aware that I have not addressed all the concerns raised in your email, Elvis, but I plan to continue this conversation in the coming days. Kind regards, Marco Schmidt Registry Services Assistant Manager RIPE NCC On 31/08/2021 06:06, Elvis Daniel Velea wrote:

Dear Marco,

thank you for the update below. I'd like to ask you to provide a few more details and to detail your plans to fix the SLA violation by the Registration Services department, please see my questions inline.

On 8/4/21 3:43 AM, Marco Schmidt wrote:

...

Dear colleagues,

We would like to give you an update on the ticket wait times in Registry Services, after discussions at RIPE 82 [1] and on the mailing list [2].

We have been addressing the issues causing delays. As the number of incoming requests continue to exceed previous years, we have taken on additional staff to help with the workload. We are also automating some of our due diligence processes which are currently time consuming.

You say that you receive more requests than in previous years, can you please provide some statistics for the past 10 years showing the number of employees vs number of requests/replies received/sent daily? - please try, if possible, to show us how many of these tickets were related to transfers, how many to billing or closures, etc...

Can you provide some stats showing how many replies are necessary for a ticket to be resolved? Also, let me know what is being done so that requests get evaluated and resolved in one reply.

Also, please let us know how exactly have you addressed the constant SLA violations, what part of which process has been automated and whether you believe that with the additional staff you will resolve the issues swiftly.

As you probably remember, I used to work in the RS department between 2007 and 2013. Back then, a day when the SLA was not respected was almost a tragedy... it seems that now it's the norm and not the exception any longer. What happened that your department ended in this situation?

...

You should start to see improvements in our response times towards the end of this year once our new colleagues are trained and our automation is ready. In the meantime, you may experience slower response times. In other words, we will be slower before we can get up to speed properly, but we think this will be worth it over the long term.

Things are slower and slower. For example, last week I had a ticket receive 1 reply after ~24 hours saying that because of time constraints, the ticket will be evaluated later. Then Friday, Saturday and Sunday passed before I got a reply on Monday.

None of the tickets I or my customers opened last month (August) have been replied to within the SLA. NONE! We're talking about more than 10 tickets in a month, none receiving a reply within the SLA.

You are saying that waiting will be worth it over the long term while right now all tickets are being delayed from a day to another and sometimes a ticket that should take 10 minutes to evaluate is delayed for 4 days or more... This is not acceptable and that is why I'd like to ask you for more details to prove to us that the wait will be worth it. As things are getting worse and not better, I hope you understand why I can't take your word for it :)

When you generate those stats, please let us know if the replies saying 'Due to a very high workload, we were unable to process your request at this time and will process it as soon as possible.' are still counted as a violation of the SLA or not.

Also, when you generate the stats I ask for, please also tell us how many working days in a month (for the past year or two) the Registration Services dept. has violated the SLA.

I've been trying to help the RS department evaluate all the requests received from V4Escrow and it's customers quicker. The help I offered was refused while the SLA violations got worse and worse.. maybe it is time you accept the help when it is offered to you.

I've had several conversation with RS management and higher up, all the way up to MD. I talked to several teams to explain the way I wanted to help RS and our customers get all the tickets resolved swiftly (at least the ones that are for transfers of resources). I proposed the introduction of an API for tickets/requests so we can try to automate a large part of the transfer ticket, this automation could help RS evaluate requests faster, would help customers provide all the required information in one go; everyone would be happy. I was told my request will be looked at and maybe added to plan no earlier than Q1-2022. Please tell me if that is still the plan or whether, in the light of these constant SLA violation, you could enable the API faster so we, brokers, can help you receive complete and easy to evaluate transfer requests.

...

We also know from experience that we receive far more requests towards the end of the year which results in a bottleneck. We will shift resources to counter this, but we would also appreciate it if you could get some of these requests in earlier.

Is this a warning that requests will take weeks to process towards the end of the year? They already take days...

Please clarify the bottlenecks you have and let's see if there's any way we can help you with them. I am sure that automating a lot of steps in your overly complicated processes will help but I am looking forward to receiving more details from you about this.

Elvis

PS: Note that I wanted to also send an e-mail about the RIPE NCC's ticketing system (zendesk) to the members-discuss mailing list and to the board members. I will wait for an answer to the e-mail I sent to HPH one week ago (which he promised he'll forward to the relevant team for a reply).

...

Kind regards, Marco Schmidt Registry Services Assistant Manager RIPE NCC

[1] RIPE NCC Operational Update (slides 4-10): https://ripe82.ripe.net/archives/video/584/

[2] Discussion on Members Discuss mailing list: https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html

_______________________________________________ members-discuss mailing list members-discuss@ripe.net https://lists.ripe.net/mailman/listinfo/members-discuss Unsubscribe: https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net

Dear Elvis, Thank you for your feedback. You are correct in pointing out that it isn’t straightforward to compare the work of the Registry Services in 2013 with the work we do in 2021. That’s why I’m preparing a deeper analysis that looks at recent changes and developments, as well as our approach to improving our services. A lot has changed since you left eight years ago, and you might be surprised to learn about the added complexity for even seemingly easy requests. To share some more numbers on this situation, in the first eight months of 2021, the amount of resource-related requests has increased by almost 25% compared to the same period last year. Looking at just August of this year, 62% more requests were created than in August 2020. At the same time, our due diligence activities have increased due to sanction checks. This is the main reason it has been challenging to provide the fast response time that our members are used to. We are confident that with the steps taken, as outlined in my previous email, we will soon be handling this increased workload more quickly. As soon as my overview is ready, I will share it on this list. Kind regards, Marco Schmidt Registry Services Assistant Manager RIPE NCC On 03/09/2021 12:39, Elvis Daniel Velea wrote:

Hi Marco,

thank you for your reply. Please see further comments below. Maybe some will help you with your next reply.

On Fri, Sep 3, 2021 at 01:57 Marco Schmidt <mschmidt@ripe.net <mailto:mschmidt@ripe.net>> wrote:

Dear Elvis,

Your email touches on multiple issues. I‘d like to address some of them now, with the understanding that we will share more detailed information soon. For now, I can provide a general overview on what the situation with wait times is, and how we plan to tackle this issue in the coming weeks.

ok

First, a quick historical comparison for context. Back in 2013, the RIPE NCC had around 9,000 LIRs, and we processed a mere 150 policy transfers in the whole year, i.e. approximately 12-13 transfer requests a month.

Please do not compare apples to oranges. In 2013 the RIPE NCC was processing hundreds of requests for initial and additional allocations, requests that were way more complex than some of the current resource requests (which no longer require any need based justification or evaluation) or transfer requests that only require the RIPE NCC to verify (i) if an organization is still active in their country and (ii) the document you received is signed by the people with PoA.

We currently have about 23,700 LIRs, which is 2.5x the LIRs in 2013. Last month, we processed over 400 policy transfers. This is 30x the number of transfer requests we processed when you were part of the Registry Services team. And remember that there are all of the other requests on top of this.

That is why I was asking you to provide total number of tickets handled, back then, now and every year in between. A ticket is a ticket, some are simple, some are more complicated. Same was then as it is now…

You are right, you did not process hundreds of transfer requests in 2013, but there were many other types of requests that were way more complex than some of current transfer requests. So saying that you process 30x transfer requests is a bit misleading as you are not showing the complete picture.

The NCC RS dept. even succeeded to respect the SLA when it was evaluating requests in pairs, towards the end of the runout. At that time you were receiving maaany requests, more than usually. If the department was able to cope with that situation, why can’t it now? I still have not seen the answer to this question.

In 2020, we averaged around 1,600 resources-related requests per month. However, from November to December there was a threefold increase compared to the rest of the year. It is hard to avoid an increased wait time when this happens, and that’s why we prefer to be upfront when we expect delays, so members can plan around them.

I await the numbers as requested.  The numbers that show the whole picture, not just the parts you want to show, please.

Remember when we had a ‘quota’ of 20-30 requests/replies per day. Multiply that with 20-30 employees and at minimum you should be able to process 2000 requests per week, not per month. There were employees leaving the NCC and new ones (that needed training) joining all the time. That did not stop the RS management to require the department to respect the SLA daily. Again, and you remember this perfectly as we used to work in the same department, a day missing the SLA was a tragedy, why is it now the norm?

While these statistics provide a glimpse into how things have changed over the past few years, I would like to reiterate that it is not just the number of tickets but also the complexity of requests that has increased. We report on this regularly at RIPE Meetings.

Complexity of tickets before the RIPE NCC reached the last /8 (actually, until the requirements for additional allocations were simplified in policy) was much higher than the complexity of current transfer tickets. Indeed, back then the amount of attempts of fraud were way lower.. but that doesn’t mean complexity has increased, au contraire. Complexity has just changed from complex evaluations of networks, deployment plans, equipment lists and network diagrams to anti-fraud. The RIPE NCC’s RS dept. philosophy has also changed from ‘we trust our members’ to ‘we need to make sure this is not fraud’. Maybe this change has caused the increased complexity you are talking about?

Despite these challenges, Registry Services strives to handle incoming requests efficiently. In order to improve our ticket handling times, we have: - Hired new staff, who are still being trained, but will help address the workload once they are up to speed. - Taken on temporary staff to enter registration details of resource holders into our system which will allow us to automate certain due diligence checks. We will publish a RIPE Labs article about these changes in the coming weeks.

Not sure exactly what you mean by ‘enter registration details of resource holders into our system’ but I look forward to reading more details in the article.

Some additional clarifications regarding internal targets: - Every ticket that doesn't get a response by 18:00 (UTC +2) the following working day is counted as a ‘miss’ against our internal targets. - Our notification emails do not count against these targets (we don’t see much point in gaming our internal KPIs). - A gentle reminder that Saturdays and Sundays are not working days, so requests received on a Friday are likely to be addressed the next business day, i.e. the Monday after the weekend.

See ticket #398137, for example. Ticket opened Aug 26 at 4AM, received an automated message the same day saying that due to workload… That was on Thursday. Do you agree with me that this ticket should have been handled on Friday? If yes, why was it resolved only next Monday? Marco, this is one of many tickets that have been delayed for no reason. For some I have called the NCC to see what’s the problem, we even talked a few times about some of the tickets that were being delayed for no reason.. You know about them.

See the other e-mail sent by Elina. Their situation is even worse…

So, Marco, please tell us how many working days in the past few months/years has the RS dept. missed the SLA? - actually, what may be easier to calculate is how many working days you haven’t missed the SLA

Around 2015-2017 (I don’t remember when exactly) the RS dept. decided to stop showing how many requests it received and handled daily. Before that it used to have a page showing the workload and how long it was taking for a ticket to receive a reply. Although at that time I asked for that functionality to be restored, the NCC preferred to hide it’s head in the sand and decided that too much transparency was not so good. - any plans to become transparent again about your workload and SLA?

We can appreciate that it is frustrating to wait on tickets. However, while we’re still training additional staff and automating systems, we need to rely on your patience and ask that you allow us time to ensure requests are not just handled quickly, but also accurately.

Since we know from experience that the number of requests increases dramatically in the last quarter of the year, I would like to repeat our request for members to kindly not wait until the last minute to send in a request, and to expect longer handling times in the last quarter of 2021.

I think this warning was and is a bad idea. Instead of warning the members that you will miss the SLA even more in Q4, you should work on fixing the bottlenecks. This situation has been ongoing for months/years now… you can’t come now and say ‘we know we are slow, we will be even slower in the future’, this should have been solved already so you don’t have a bigger problem next Q.

Once I have a more detailed overview of our ticket history, I’ll share it on this list, along with how we plan to further improve our services.

I am aware that I have not addressed all the concerns raised in your email, Elvis, but I plan to continue this conversation in the coming days.

I look forward to your next e-mails. So far this one has had almost no new information, you just repeated the same info you sent last month, just worded it differently. You also tried to say you are handling more requests now than years ago, you haven’t convinced me, instead you tried to trick us with some numbers that were misleading.

Looking forward to have an actual conversation about this.

Elvis

Kind regards, Marco Schmidt Registry Services Assistant Manager RIPE NCC

On 31/08/2021 06:06, Elvis Daniel Velea wrote: > Dear Marco, > > thank you for the update below. I'd like to ask you to provide a few > more details and to detail your plans to fix the SLA violation by the > Registration Services department, please see my questions inline. > > On 8/4/21 3:43 AM, Marco Schmidt wrote: >> Dear colleagues, >> >> We would like to give you an update on the ticket wait times in >> Registry Services, after discussions at RIPE 82 [1] and on the >> mailing list [2]. >> >> We have been addressing the issues causing delays. As the number of >> incoming requests continue to exceed previous years, we have taken on >> additional staff to help with the workload. We are also automating >> some of our due diligence processes which are currently time consuming. > > You say that you receive more requests than in previous years, can you > please provide some statistics for the past 10 years showing the > number of employees vs number of requests/replies received/sent daily? > - please try, if possible, to show us how many of these tickets were > related to transfers, how many to billing or closures, etc... > > Can you provide some stats showing how many replies are necessary for > a ticket to be resolved? Also, let me know what is being done so that > requests get evaluated and resolved in one reply. > > Also, please let us know how exactly have you addressed the constant > SLA violations, what part of which process has been automated and > whether you believe that with the additional staff you will resolve > the issues swiftly. > > As you probably remember, I used to work in the RS department between > 2007 and 2013. Back then, a day when the SLA was not respected was > almost a tragedy... it seems that now it's the norm and not the > exception any longer. What happened that your department ended in this > situation? > >> >> You should start to see improvements in our response times towards >> the end of this year once our new colleagues are trained and our >> automation is ready. In the meantime, you may experience slower >> response times. In other words, we will be slower before we can get >> up to speed properly, but we think this will be worth it over the >> long term. > > Things are slower and slower. For example, last week I had a ticket > receive 1 reply after ~24 hours saying that because of time > constraints, the ticket will be evaluated later. Then Friday, Saturday > and Sunday passed before I got a reply on Monday. > > None of the tickets I or my customers opened last month (August) have > been replied to within the SLA. NONE! We're talking about more than 10 > tickets in a month, none receiving a reply within the SLA. > > You are saying that waiting will be worth it over the long term while > right now all tickets are being delayed from a day to another and > sometimes a ticket that should take 10 minutes to evaluate is delayed > for 4 days or more... This is not acceptable and that is why I'd like > to ask you for more details to prove to us that the wait will be worth > it. As things are getting worse and not better, I hope you understand > why I can't take your word for it :) > > > When you generate those stats, please let us know if the replies > saying 'Due to a very high workload, we were unable to process your > request at this time and will process it as soon as possible.' are > still counted as a violation of the SLA or not. > > Also, when you generate the stats I ask for, please also tell us how > many working days in a month (for the past year or two) the > Registration Services dept. has violated the SLA. > > > I've been trying to help the RS department evaluate all the requests > received from V4Escrow and it's customers quicker. The help I offered > was refused while the SLA violations got worse and worse.. maybe it is > time you accept the help when it is offered to you. > > I've had several conversation with RS management and higher up, all > the way up to MD. I talked to several teams to explain the way I > wanted to help RS and our customers get all the tickets resolved > swiftly (at least the ones that are for transfers of resources). I > proposed the introduction of an API for tickets/requests so we can try > to automate a large part of the transfer ticket, this automation could > help RS evaluate requests faster, would help customers provide all the > required information in one go; everyone would be happy. I was told my > request will be looked at and maybe added to plan no earlier than > Q1-2022. Please tell me if that is still the plan or whether, in the > light of these constant SLA violation, you could enable the API faster > so we, brokers, can help you receive complete and easy to evaluate > transfer requests. > >> We also know from experience that we receive far more requests >> towards the end of the year which results in a bottleneck. We will >> shift resources to counter this, but we would also appreciate it if >> you could get some of these requests in earlier. > > Is this a warning that requests will take weeks to process towards the > end of the year? They already take days... > > Please clarify the bottlenecks you have and let's see if there's any > way we can help you with them. I am sure that automating a lot of > steps in your overly complicated processes will help but I am looking > forward to receiving more details from you about this. > > > Elvis > > PS: Note that I wanted to also send an e-mail about the RIPE NCC's > ticketing system (zendesk) to the members-discuss mailing list and to > the board members. I will wait for an answer to the e-mail I sent to > HPH one week ago (which he promised he'll forward to the relevant team > for a reply). > >> >> Kind regards, >> Marco Schmidt >> Registry Services Assistant Manager >> RIPE NCC >> >> [1] RIPE NCC Operational Update (slides 4-10): >> https://ripe82.ripe.net/archives/video/584/ <https://ripe82.ripe.net/archives/video/584/> >> >> [2] Discussion on Members Discuss mailing list: >> https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html <https://www.ripe.net/ripe/mail/archives/members-discuss/2021-May/004353.html>

>> >> >> >> _______________________________________________ >> members-discuss mailing list >> members-discuss@ripe.net <mailto:members-discuss@ripe.net> >> https://lists.ripe.net/mailman/listinfo/members-discuss <https://lists.ripe.net/mailman/listinfo/members-discuss> >> Unsubscribe: >> https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net <https://lists.ripe.net/mailman/options/members-discuss/elvis%40v4escrow.net> >

-- This message was sent from a mobile device. Some typos may be possible.