Ticket Number: - Ticket Status: open Ticket Type: unscheduled Ticket Source: RIPE NCC Ticket Scope: RIPE NCC Site/Line: RIPE NCC Ticket Owner: ops@ripe.net Problem Fixer: Geert Jan de Groot Ticket Opened: 950217 21:47 UTC Problem Started: 950217 21:00 UTC Closed: Ended: Problem Description: Due to the failure of 2 (two) harddisks at the RIPE NCC on the same day, it seems we may have lost mail sent to the various role accounts on Friday. Also, whois.ripe.net has been offline on Friday for an hour (sorry). Affected: RIPE NCC role accounts: mail which arrived between 950217 00:00 UTC and 950217 22:00 UTC may be lost (we still have the sendmail logs) RIPE auto-dbm updates were deliberately delayed until machines are in full working order, but will be processed shortly. If you have not received an ack by Monday morning, please re-submit the update. We have been unable to send this ticker earlier, because the mail aliases were also on one of the disks involved.. Actions: Install replacement harddisk on whois.ripe.net on Monday (downtime is expected to be < 5 minutes). Restore disk contents of 950217 0:00 (last dump date) for NCC role accounts. Investigate sendmail logs for possible lost mail, and send individual messages, if possible. Restore role account transaction records for 950217 as far as possible. LOCAL-REGISTRIES: if possible, re-send your mail sent on Friday with RESEND in the subject line. Install some garlic in the engine room ;-) While we make dayly backups (of course!), the crash happened 21 hours after the last backup, and those 21 hours includes a full working day. I can't think of a better way to guard against these exceptional circumstances. Time to fix: RIPE NCC operations should be back to normal by the time you read this, but us staff has to determine what mail has been lost. As a result, we might not respond immediately to messages on Monday. I expect to be able to do the harddisk swaps with minimal disruption.