Re: [db-wg] So what happened last night to the DB?
Dear colleagues,
We are aware of an incident affecting Whois updates yesterday evening, which we are continuing to investigate. We will keep the DB-WG informed.
Regards Ed Shryane RIPE NCC There is still ongoing activity with perhaps malicious whois updates. See attached.
Regards, Hank
Hank, On 21/06/2024 06.36, Hank Nussbacher via db-wg wrote:
We are aware of an incident affecting Whois updates yesterday evening, which we are continuing to investigate. We will keep the DB-WG informed.
There is still ongoing activity with perhaps malicious whois updates. See attached.
This is `mnt-nfy:` working as intended right? You can contact the folks at IFAT group and let them know that their resources might have been hijacked, and you can contact the RIPE DBM and have them investigate. Is there some specific action that you think that the RIPE DB working group can or should do? Cheers, -- Shane
Dear RIPE DB-WG, Hope this email finds you in good health! Please see my comments below, inline... Thanks. Le vendredi 21 juin 2024, Shane Kerr via db-wg <db-wg@ripe.net> a écrit :
Hank,
On 21/06/2024 06.36, Hank Nussbacher via db-wg wrote:
We are aware of an incident affecting Whois updates yesterday evening, which we are continuing to investigate. We will keep the DB-WG informed.
There is still ongoing activity with perhaps malicious whois updates. See attached.
This is `mnt-nfy:` working as intended right? You can contact the folks at IFAT group and let them know that their resources might have been hijacked, and you can contact the RIPE DBM and have them investigate.
Is there some specific action that you think that the RIPE DB working group can or should do?
Hi Shane, Thanks for your email, brother. It seems as its actions are also oriented as allowed by the objectives of this very working group. Please see the below quoted: " *[...]The Database Working Group is open to anyone with an interest in the RIPE Database. The group discusses changes to existing database objects, the creation of new objects and features, distribution of databases _***and security issues***_. To post a message to the list, send an email to * db-wg@ripe.net *. Please note that only subscribers can post messages.[...]*" __ https://www.ripe.net/membership/mail/ripe-mailing-lists/db-wg/#:~:text=The%2... . Blessed day! Shalom, --sb.
Cheers,
-- Shane
-- Best Regards ! baya.sylvain [AT cmNOG DOT cm] | [[https://www.cmnog.cm/dokuwiki/Structure |cmNOG's Structure]] | [[https://survey2.cmnog.cm/ |cmNOG's Surveys]] | [[ https://lists.cmnog.cm/mailman/listinfo/cmnog |Subscribe to cmNOG's Mailing List]] | [[https://tools.std.douala-ix.net/lg |DIX's LookingGlass]] | __ #LASAINTEBIBLE|#Colossiens2:14,13-17«14 IL a effacé l'acte dont les ordonnances nous condamnaient et qui subsistait contre nous, et IL l'a détruit en le clouant à la croix;»#AMEN,#Maranatha,#MerciJÉSUS! #MaPrière est que tu naisses de nouveau.#Chrétiennement
Dear RIPE DB-WG, Please, find a comment below, inline... Thanks. Le vendredi 21 juin 2024, Hank Nussbacher via db-wg <db-wg@ripe.net> a écrit :
Dear colleagues,
We are aware of an incident affecting Whois updates yesterday evening, which we are continuing to investigate. We will keep the DB-WG informed.
Regards Ed Shryane RIPE NCC
There is still ongoing activity with perhaps malicious whois updates. See attached.
Hi Hank, Thanks for your valuable notification, brother. It seems as the patch [*] is not working as expected :-/ Please see below: " *[...]Jun 16, 2024Whois Update Service Incident_***Resolved***_ - Between approximately 20:00 UTC on Sunday 16th June and 01:00 UTC on Monday 17th June, there was an intermittent interruption to the Whois Update service, due to a number of unusual update messages. In addition, _***many spurious update notification emails were generated to uninvolved maintainers***_. We have taken steps to mitigate the impact of these updates, and will publish a more detailed _***post-mortem later* **_.Jun 16, 20:00 UTC[...]*" __ [*]: https://status.ripe.net/#:~:text=Jun%2016%2C%202024,%2C%2020%3A00%20UTC Hope this helps! Shalom, --sb.
Regards,
Hank
-- Best Regards ! baya.sylvain [AT cmNOG DOT cm] | [[https://www.cmnog.cm/dokuwiki/Structure |cmNOG's Structure]] | [[https://survey2.cmnog.cm/ |cmNOG's Surveys]] | [[ https://lists.cmnog.cm/mailman/listinfo/cmnog |Subscribe to cmNOG's Mailing List]] | [[https://tools.std.douala-ix.net/lg |DIX's LookingGlass]] | __ #LASAINTEBIBLE|#Colossiens2:14,13-17«14 IL a effacé l'acte dont les ordonnances nous condamnaient et qui subsistait contre nous, et IL l'a détruit en le clouant à la croix;»#AMEN,#Maranatha,#MerciJÉSUS! #MaPrière est que tu naisses de nouveau.#Chrétiennement
Dear colleagues,
We are aware of an incident affecting Whois updates yesterday evening, which we are continuing to investigate. We will keep the DB-WG informed.
Regards Ed Shryane RIPE NCC
Hi. It has been 2 weeks. Did I miss an email on the matter? Regards, Hank
Hi Hank! Apologies for the delayed response. I'd like to provide you with an update on the current status. On one hand, we have deployed features to mitigate issues related to making updates with multiple references (#1486). This prevents the creation of an RPSL with a lot of references. We are continuing to investigate ways to further improve this process. On the other hand, the second round of notification you received was due to some notifications bouncing back and being incorrectly handled as mail updates instead of bounced messages. We have addressed this issue by correctly handling these messages as Multipart/mixed Bounced and Auto-submitted Messages (#1490). We are actively monitoring the situation to ensure the issue is resolved and to determine if additional features are necessary. We will publish a post-mortem once we are confident the issues raised have been mitigated. Thank you for your patience and understanding. Regards Miguel Herrán RIPE NCC On Mon, 1 Jul 2024 at 18:39, Hank Nussbacher via db-wg <db-wg@ripe.net> wrote:
Dear colleagues,
We are aware of an incident affecting Whois updates yesterday evening, which we are continuing to investigate. We will keep the DB-WG informed.
Regards Ed Shryane RIPE NCC
Hi.
It has been 2 weeks. Did I miss an email on the matter?
Regards, Hank
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
On 02/07/2024 10:44, Miguel Mosquera via db-wg wrote: Thanks. The initial incident happened on June 17 and a subsequent incident on June 24. Were all your fixes as listed below, performed post June 24? Regards, Hank
Hi Hank! Apologies for the delayed response.
I'd like to provide you with an update on the current status.
On one hand, we have deployed features to mitigate issues related to making updates with multiple references (#1486). This prevents the creation of an RPSL with a lot of references. We are continuing to investigate ways to further improve this process.
On the other hand, the second round of notification you received was due to some notifications bouncing back and being incorrectly handled as mail updates instead of bounced messages. We have addressed this issue by correctly handling these messages as Multipart/mixed Bounced and Auto-submitted Messages (#1490).
We are actively monitoring the situation to ensure the issue is resolved and to determine if additional features are necessary. We will publish a post-mortem once we are confident the issues raised have been mitigated.
Thank you for your patience and understanding.
Regards Miguel Herrán RIPE NCC
On Mon, 1 Jul 2024 at 18:39, Hank Nussbacher via db-wg <db-wg@ripe.net <mailto:db-wg@ripe.net>> wrote:
> Dear colleagues, > > We are aware of an incident affecting Whois updates yesterday > evening, which we are continuing to investigate. We will keep > the DB-WG informed. > > Regards > Ed Shryane > RIPE NCC
Hi.
It has been 2 weeks. Did I miss an email on the matter?
Regards, Hank
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg <https://lists.ripe.net/mailman/listinfo/db-wg>
Hello Hank,
On 2 Jul 2024, at 14:24, Hank Nussbacher via db-wg <db-wg@ripe.net> wrote:
On 02/07/2024 10:44, Miguel Mosquera via db-wg wrote:
Thanks. The initial incident happened on June 17 and a subsequent incident on June 24. Were all your fixes as listed below, performed post June 24?
We implemented two fixes in response to the incidents in June. The fix for multiple references was deployed on June 24th in release 1.112.1. The fix for handling bounced messages was deployed on Monday July 1st in release 1.113.2. Regards Ed Shryane RIPE NCC
Hi Ed On Wed, 3 Jul 2024 at 11:09, Edward Shryane via db-wg <db-wg@ripe.net> wrote:
Hello Hank,
On 2 Jul 2024, at 14:24, Hank Nussbacher via db-wg <db-wg@ripe.net> wrote:
On 02/07/2024 10:44, Miguel Mosquera via db-wg wrote:
Thanks. The initial incident happened on June 17 and a subsequent incident on June 24. Were all your fixes as listed below, performed post June 24?
We implemented two fixes in response to the incidents in June.
The fix for multiple references was deployed on June 24th in release 1.112.1.
What is the fix that you have deployed? Does it have any operational impact? For example a limit on the number of MNTNER objects that can be referenced or changes to the way MNTNERs are referenced. cheers denis
The fix for handling bounced messages was deployed on Monday July 1st in release 1.113.2.
Regards Ed Shryane RIPE NCC
--
To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/db-wg
denis walker via db-wg wrote on 03/07/2024 11:24:
What is the fix that you have deployed? Does it have any operational impact? For example a limit on the number of MNTNER objects that can be referenced or changes to the way MNTNERs are referenced.
Denis, the fix is public:
https://github.com/RIPE-NCC/whois/commit/8738f0b32796f2631a2c611527fbbd5bc8f...
i.e. configurable number of max references, currently set to 100 in the config file. Also, a unit test added. All looks entirely reasonable. Nick
Hi, On Tue, Jul 02, 2024 at 09:44:33AM +0200, Miguel Mosquera via db-wg wrote:
On one hand, we have deployed features to mitigate issues related to making updates with multiple references (#1486). This prevents the creation of an RPSL with a lot of references. We are continuing to investigate ways to further improve this process.
Has there been some insight on *why* the original change has been attempted? Has this been a breach of account, or a sort-of reasonable explanation by the SSO user? Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard, Ingo Lalla, Karin Schuler, Sebastian Cler Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Hello Gert,
On 2 Jul 2024, at 14:50, Gert Doering via db-wg <db-wg@ripe.net> wrote:
Hi,
On Tue, Jul 02, 2024 at 09:44:33AM +0200, Miguel Mosquera via db-wg wrote:
On one hand, we have deployed features to mitigate issues related to making updates with multiple references (#1486). This prevents the creation of an RPSL with a lot of references. We are continuing to investigate ways to further improve this process.
Has there been some insight on *why* the original change has been attempted? Has this been a breach of account, or a sort-of reasonable explanation by the SSO user?
I can't speculate on why it was attempted, but this is being investigated separately by the RIPE NCC. The updates referenced every maintainer in the database, which caused the spike in mail notifications and delays to other updates. There has been no data leak or security breach as a result of this incident. The DB team are focused on mitigating the operational impact of these updates. As Miguel said, we will publish a full post-mortem once we are confident the vulnerabilities have been addressed. Regards Ed Shryane RIPE NCC
participants (8)
-
denis walker -
Edward Shryane -
Gert Doering -
Hank Nussbacher -
Miguel Mosquera -
Nick Hilliard -
Shane Kerr -
Sylvain Baya