Mojn guys
I had a server crash last week, and when I got it up and running again, the database was not able to receive updates - the whois-server core dumped. I've been there before, so I kept the old pseudo defect database running and rebuild a new DB during the night (downloading latest ripe.db.gz and RIPE.CURRENTSERIAL)
Rebuilding ended up with an error. I can't remember seeing this before, but my first thought was that it something to do with the already running database.
*** ERROR: Loader failure [139]. Exiting\n
************** ERROR ***************
*** 04:56:54 Error loading database=WHOISDB for source=RIPE, /usr/local/whoisserver/bin/make_db exiting
************************************
I've started NRTM through the console, and everything looks fine, but apparently my current serial is 0 (zero) and not 28071882(+) as I would expect.
20140227 14:31:32 whois_rip-25050/2356136848 INFO connecting to NRTM server [whois.ripe.net:4444] (current serial=0)
20140227 14:31:32 whois_rip-25050/2356136848 INFO ******** report **********
20140227 14:31:32 whois_rip-25050/2356136848 INFO 0 objects OK ( 0.0000 obj/s)
20140227 14:31:32 whois_rip-25050/2356136848 INFO 0 objects failed
20140227 14:31:32 whois_rip-25050/2356136848 INFO average processing time 0.0000 obj/s ( 0.00 obj/min)
20140227 14:31:32 whois_rip-25050/2356136848 INFO forwarded to serial:0
The old database (table serials) contains some 5051203 rows ending with serial=28072454
The new database (table serials) is pretty empty:
mysql> desc serials;
+-------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------------+------+-----+---------+----------------+
| thread_id | int(10) unsigned | NO | MUL | 0 | |
| serial_id | int(11) | NO | PRI | NULL | auto_increment |
| object_id | int(10) unsigned | NO | MUL |
0 | |
| sequence_id | int(10) unsigned | NO | | 0 | |
| atlast | tinyint(4) unsigned | NO | | 0 | |
| operation | tinyint(4) unsigned | NO | | 0 | |
+-------------+---------------------+------+-----+---------+----------------+
6 rows in set (0.00 sec)
mysql> select serial_id fr
om serials;
Empty set (0.00 sec)
Digging deeper into the the problem I found this - this is the actual error when it crashed during make_db the first night:
WARNING - 'Feb 27 04:56:54 snowblind kernel: [67641.487347] loader[22291]: segfault at 0 ip 00000000 sp bfb7fddc error 4 in loader[8048000+1a6000]' (1 matches for 'WARNING,[Ww]arning,ERROR,[Ee]rror') found in '/var/log/messages'
I wonder if the ripe.db.gz is broken ("ip 00000000") ?
So, I ran another make_db - but with the same result. Then I recovered an mysqldump from the start of the week, and it started up OK. Then I activated NRTM and updates started to roll in ...
Happy I was for some hours; the server crash again saturday night :-(
Today even startup of whois-server fails completely with core dumps, so I've picked another mysql backup and it's curently recovering.
It seems me like one of the updates are causing problems - it's just a feeling - not a solid "Hey, here's the culprit!" fact.
I've attached the error log (from today - core dumping (crash recovery mode + roll back))
Anyone ?
Best Regards,
Peter Landbo