RPKI validator 3: initial sync regression?
Hi, lately with validator 3 version > 370 it felt like rpki-validator 3 is less reliable and taking longer to download RPKI repository data (especially RIPE's) when starting for the first time when compared to older version (313) where it probably took less than 5 minutes after starting it to have the data for all regions. I reproduced this consistently. Can you think of anything that caused this? Does anyone else see this as well? Unfortunately I didn't find any changelogs and it is not obvious what version relates to what git commit since there are no git tags in the github repo. It would be great to have changelogs or git tags. thank you, nusenu -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Hi nusenu, We are looking into the situation with the validator as it seems to be problematic since release 378. Here are our findings so far: 1 Version Jar creation time Loading from scratch 2 ——————————————————————————————————— 3 4 366 Jan 28, 11:47 5 minute’ish 5 370 Jan 28, 18:40 5 minute’ish 6 376 Jan 31, 14:15 5 minute’ish 7 377 Feb 01, 13:30 5 minute'ish 8 378 Mar 06, 14:23 Stuck after 10 mins, only pending APNIC 9 381 Mar 06, 18:30 Stuck after 10 mins, AFRINIC, LACNIC, APNIC done For now, I recommend to revert back to release 377 while we work on fixing the issue. Unfortunately, this build did not include the https fix for the RIS dumps. There is an issue in GitHub: https://github.com/RIPE-NCC/rpki-validator-3/issues/77 <https://github.com/RIPE-NCC/rpki-validator-3/issues/77> Regarding the git tags and/or change logs, you are right, we will include them in next releases. We’ll keep the mailinglist and GitHub posted. Thanks, Nathalie Trenaman RIPE NCC
Op 18 mrt. 2019, om 18:17 heeft nusenu <nusenu-lists@riseup.net> het volgende geschreven:
Hi,
lately with validator 3 version > 370 it felt like rpki-validator 3 is less reliable and taking longer to download RPKI repository data (especially RIPE's) when starting for the first time when compared to older version (313) where it probably took less than 5 minutes after starting it to have the data for all regions. I reproduced this consistently.
Can you think of anything that caused this? Does anyone else see this as well?
Unfortunately I didn't find any changelogs and it is not obvious what version relates to what git commit since there are no git tags in the github repo.
It would be great to have changelogs or git tags.
thank you, nusenu
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Hi all, I can confirm that 377 was the last release that was... usable. I tried 394 and yesterday's 397. Version 397 had only managed to load RIPE + APNIC after 30 minutes, with no indication when the other repositories would finish. Kind regards, Job On Tue, Mar 19, 2019 at 04:12:07PM +0100, Nathalie Trenaman wrote:
Hi nusenu,
We are looking into the situation with the validator as it seems to be problematic since release 378. Here are our findings so far:
1 Version Jar creation time Loading from scratch 2 ——————————————————————————————————— 3 4 366 Jan 28, 11:47 5 minute’ish 5 370 Jan 28, 18:40 5 minute’ish 6 376 Jan 31, 14:15 5 minute’ish 7 377 Feb 01, 13:30 5 minute'ish 8 378 Mar 06, 14:23 Stuck after 10 mins, only pending APNIC 9 381 Mar 06, 18:30 Stuck after 10 mins, AFRINIC, LACNIC, APNIC done
For now, I recommend to revert back to release 377 while we work on fixing the issue. Unfortunately, this build did not include the https fix for the RIS dumps. There is an issue in GitHub: https://github.com/RIPE-NCC/rpki-validator-3/issues/77 <https://github.com/RIPE-NCC/rpki-validator-3/issues/77>
Regarding the git tags and/or change logs, you are right, we will include them in next releases.
We’ll keep the mailinglist and GitHub posted.
Thanks, Nathalie Trenaman RIPE NCC
Op 18 mrt. 2019, om 18:17 heeft nusenu <nusenu-lists@riseup.net> het volgende geschreven:
Hi,
lately with validator 3 version > 370 it felt like rpki-validator 3 is less reliable and taking longer to download RPKI repository data (especially RIPE's) when starting for the first time when compared to older version (313) where it probably took less than 5 minutes after starting it to have the data for all regions. I reproduced this consistently.
Can you think of anything that caused this? Does anyone else see this as well?
Unfortunately I didn't find any changelogs and it is not obvious what version relates to what git commit since there are no git tags in the github repo.
It would be great to have changelogs or git tags.
thank you, nusenu
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
FYI, as7018 has locked our rpki-validator-3 and rpki-rtr-server at their version 333. I had difficulty in my lab deployments moving beyond that version. In production, even that version 333 requires occasional manual restarts at unpredictable intervals to get rpki-validator-3 working again. Also, could RIPE resume displaying the version number on the validator's web pages? That had been the case for the version 2 validators. Thanks. Jay B. Job Snijders writes:
Hi all,
I can confirm that 377 was the last release that was... usable.
I tried 394 and yesterday's 397. Version 397 had only managed to load RIPE + APNIC after 30 minutes, with no indication when the other repositories would finish.
Kind regards,
Job
On Tue, Mar 19, 2019 at 04:12:07PM +0100, Nathalie Trenaman wrote:
Hi nusenu,
We are looking into the situation with the validator as it seems to be problematic since release 378. Here are our findings so far:
1 Version Jar creation time Loading from scratch 2 ——————————————————————————————————— 3 4 366 Jan 28, 11:47 5 minute’ish 5 370 Jan 28, 18:40 5 minute’ish 6 376 Jan 31, 14:15 5 minute’ish 7 377 Feb 01, 13:30 5 minute'ish 8 378 Mar 06, 14:23 Stuck after 10 mins, only pending APNIC 9 381 Mar 06, 18:30 Stuck after 10 mins, AFRINIC, LACNIC, APNIC done
For now, I recommend to revert back to release 377 while we work on fixing the issue. Unfortunately, this build did not include the https fix for the RIS dumps. There is an issue in GitHub: https://github.com/RIPE-NCC/rpki-validator-3/issues/77 <https://github.com/RIPE-NCC/rpki-validator-3/issues/77>
Regarding the git tags and/or change logs, you are right, we will include them in next releases.
We’ll keep the mailinglist and GitHub posted.
Thanks, Nathalie Trenaman RIPE NCC
Op 18 mrt. 2019, om 18:17 heeft nusenu <nusenu-lists@riseup.net> het volgende geschreven:
Hi,
lately with validator 3 version > 370 it felt like rpki-validator 3 is less reliable and taking longer to download RPKI repository data (especially RIPE's) when starting for the first time when compared to older version (313) where it probably took less than 5 minutes after starting it to have the data for all regions. I reproduced this consistently.
Can you think of anything that caused this? Does anyone else see this as well?
Unfortunately I didn't find any changelogs and it is not obvious what version relates to what git commit since there are no git tags in the github repo.
It would be great to have changelogs or git tags.
thank you, nusenu
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
Hello, we also observe problems on our installations with never versions. After very long period (hour+), all repositories loaded in 397. But after few hours of running, database file quickly grew up (13 GB total size, with more than 9 GB alocated within ~half hour). This problem was introduced somewhere between versions 386 and 397. Based on local observations, we returned to 377 for now. It seems, there're multiple regressions in new versions... - Daniel On 3/20/19 12:54 PM, Job Snijders wrote:
Hi all,
I can confirm that 377 was the last release that was... usable.
I tried 394 and yesterday's 397. Version 397 had only managed to load RIPE + APNIC after 30 minutes, with no indication when the other repositories would finish.
Kind regards,
Job
On Tue, Mar 19, 2019 at 04:12:07PM +0100, Nathalie Trenaman wrote:
Hi nusenu,
We are looking into the situation with the validator as it seems to be problematic since release 378. Here are our findings so far:
1 Version Jar creation time Loading from scratch 2 ——————————————————————————————————— 3 4 366 Jan 28, 11:47 5 minute’ish 5 370 Jan 28, 18:40 5 minute’ish 6 376 Jan 31, 14:15 5 minute’ish 7 377 Feb 01, 13:30 5 minute'ish 8 378 Mar 06, 14:23 Stuck after 10 mins, only pending APNIC 9 381 Mar 06, 18:30 Stuck after 10 mins, AFRINIC, LACNIC, APNIC done
For now, I recommend to revert back to release 377 while we work on fixing the issue. Unfortunately, this build did not include the https fix for the RIS dumps. There is an issue in GitHub: https://github.com/RIPE-NCC/rpki-validator-3/issues/77 <https://github.com/RIPE-NCC/rpki-validator-3/issues/77>
Regarding the git tags and/or change logs, you are right, we will include them in next releases.
We’ll keep the mailinglist and GitHub posted.
Thanks, Nathalie Trenaman RIPE NCC
Op 18 mrt. 2019, om 18:17 heeft nusenu <nusenu-lists@riseup.net> het volgende geschreven:
Hi,
lately with validator 3 version > 370 it felt like rpki-validator 3 is less reliable and taking longer to download RPKI repository data (especially RIPE's) when starting for the first time when compared to older version (313) where it probably took less than 5 minutes after starting it to have the data for all regions. I reproduced this consistently.
Can you think of anything that caused this? Does anyone else see this as well?
Unfortunately I didn't find any changelogs and it is not obvious what version relates to what git commit since there are no git tags in the github repo.
It would be great to have changelogs or git tags.
thank you, nusenu
-- https://twitter.com/nusenu_ https://mastodon.social/@nusenu
participants (5)
-
Daniel Suchy -
Jay Borkenhagen -
Job Snijders -
Nathalie Trenaman -
nusenu