We have just published the time-line of events that led to the rDNS service issues last week:
https://labs.ripe.net/Members/dfk/timeline-of-reverse-dns-events
I realise that our operational performance was not up to the standard you expect from us and I apologise again for the considerable inconvenience we have caused some of you. Another report will follow once we have fully analysed what happened.
Daniel
Dear colleagues,
During the DNS Working Group session at RIPE 64, Olaf Kolkman asked the
RIPE NCC to provide graphs of DNSKEY queries arriving at K-root. These
graphs, especially the yearly one, are a good indicator about the level
of DNSSEC validation happening for the root zone.
We have enabled automatic generation of these graphs. They are available at:
http://k.root-servers.org/statistics/ROOT/dnskey_queries.html
If you have any questions, please email <dns(a)ripe.net>.
Regards,
Anand Buddhdev
RIPE NCC
Dear colleagues,
As of Friday, 15 June 2012 at 07:30 UTC, reverse DNS operations are
fully back to normal.
The service update has been posted at:
http://www.ripe.net/lir-services/service-announcements/reverse-dns-services…
However, if you are still experiencing abnormal reverse DNS responses,
please email us at <dns-help(a)ripe.net>.
========================
Summary Report and Incident Analysis
========================
We are analysing the incident and preparing a summary report on the
facts, cause and effects of the outage. We will send this report out
early next week.
We will follow up with a detailed incident analysis.
==============
Service Updates
==============
All RIPE NCC-related service announcements will be posted on
<ncc-announce(a)ripe.net> and published at:
http://www.ripe.net/lir-services/service-announcements
We will also alert our Twitter and Facebook followers that an update has
been posted. Social media updates are posted and maintained during
regular business hours.
Please note that the RIPE NCC does not use the
<members-discuss(a)ripe.net> mailing list to post service announcements or
updates.
===================
Reporting Procedures Review
===================
We strive to provide you with accurate updates on outages. We understand
that for incidents of this nature, timely and regular updates to our
stakeholders is of the utmost importance. We are analysing feedback from
the incident and using it to improve our communication procedures.
Kind regards,
Serge Radovcic
Chief Communications Officer
RIPE NCC
Dear colleagues,
As of 20:00 UTC, Thursday 14 June 2012, all regular zone information is again complete and up-to-date. However, we are still processing some imported ERX zones from the APNIC region.
If you are still experiencing any abnormal responses, other than related to the above mentioned ERX zone, you can contact us atdns-help(a)ripe.net. Provide as much detail as possible.
This update is available online at:
http://www.ripe.net/lir-services/service-announcements/reverse-dns-services…
Kind regards,
Romeo Zwart
GII Services Manager
RIPE NCC
Dear colleagues,
We'd like to update you on the reverse DNS outage situation:
As of 13:00 UTC on Thursday, 14 June 2012 the zones listed in the previous update have been restored to the state of 13:30 UTC, Wednesday, 13 June 2012. So all zones we maintain are now available and zones in the list are no longer special. There may be a very few ERX zones maintained by other RIRs that are still not fully restored.
The update has been posted online at:
https://www.ripe.net/lir-services/service-announcements/reverse-dns-service… <http://www.ripe.net/lir-services/service-announcements>
All updates since 13:30 UTC yesterday are still pending and are not reflected in the zones. However, no updates were lost. Any updates you will submit or you have already submitted will eventually be reflected in the zones once we complete our restoration procedure.
=============
Next actions
=============
We will now create up-to-date zones that include all the updates since yesterday 13:30 UTC. We will also recreate all ERX zones with current data. We expect to complete this by this evening. Once that is complete, we will re-enable regular provisioning and everything will be back to normal.
If you experience any abnormal responses beyond the residual inconsistencies described here, please let us know atdns-help(a)ripe.net with as much detail as possible.
Kind regards,
Romeo Zwart
GII Services Manager
RIPE NCC
Dear colleagues,
We'd like to update you on the reverse DNS service outage situation.
As of 10:45 UTC on Thursday, 14 June 2012, all reverse DNS zones we publish and that are not on the list below are correct and reflect the state of 13:30 UTC yesterday, 13 June. If you continue to see problems after this time, please contactdns-help(a)ripe.net with details of what responses you are getting. Please consider (negative) caching TTLs and restart your caching servers if possible.
We've posted this update online at:
https://www.ripe.net/internet-coordination/news/announcements/update-14-jun…
===============
Outage Details
===============
We know that some zones not mentioned in the list of the previous update (20:45 UTC, 13 June) were not correctly propagated to some authoritative servers after the roll-back last night. We do not know the extent of this at the present time. However we are sure that as of 10:45 UTC today all authoritative servers answer correctly with the state of 13:30 UTC 13 June for all zones but the few on the list in the previous update.
We are working hard to restore the zones listed. We are busy adding all the updates since 13:30 UTC 13 June to all zones. We will report on how this work is progressing.
The zones known to be still affected are:
0.4.1.0.0.2.ip6.arpa
185.in-addr.arpa
4.1.1.0.0.2.ip6.arpa
5.1.0.0.2.ip6.arpa
6.1.1.0.0.2.ip6.arpa
7.0.1.0.0.2.ip6.arpa
7.1.1.0.0.2.ip6.arpa
7.4.1.0.0.2.ip6.arpa
8.0.1.0.0.2.ip6.arpa
a.0.1.0.0.2.ip6.arpa
a.1.1.0.0.2.ip6.arpa
a.4.1.0.0.2.ip6.arpa
b.0.1.0.0.2.ip6.arpa
b.1.1.0.0.2.ip6.arpa
b.4.1.0.0.2.ip6.arpa
We will publish a full report once we've restored all the zones and have had sufficient time to fully analyse what went wrong.
We have been working hard over the past 18 hours to resolve this problem and we apologise for the considerable inconvenience.
Kind regards,
Romeo Zwart
GII Services Manager
RIPE NCC
