Re: [dns-wg] Followup to IANA TLD delegation problem
On Friday, 10 Jun 2005, Doug Barton writes: Dear Doug Thanks for responding to Jim's questions. Reading your mail I still have difficulties understanding two issues: a) You state in [1] that after editing the root zone, VerSign experienced an "unexpected result". What exactly was the result what caused it ? Whould also appreciate if you could elaborate on how VerSign was able to fix this "unexpected result". b) When this "unexpected result" was regarded by IANA staff as temporary and as having minimal operational impact, why was AFNIC then sent this response to their application: "IANA is no longer adding additional examples of host names in delegation records where multiple host names point to the same IP address". The "no longer" suggests a policy change. Why did IANA not just say: "Folks we have a problem here, wait a moment until we've fixed it" ? Marcel
Thanks again for the opportunity to discuss these issues. I hope that the group finds these answers satisfactory. We are of course happy to discuss this in further detail if desired.
[1] What was the nature of the technical problem that prevented multiple names in for an IP address and how was it resolved?
In November of 2004 AFNIC asked to have several hostnames added to the root zone that resolved to the same IPv4 addresses as hostnames that already existed. When VeriSign edited the root zone to include these new names, an unexpected result occurred. In some of the TLDs that had the pre-existing hostnames for these servers, the new hostnames appeared in the delegations inappropriately. This problem was identified within several hours of the zone going live by VeriSign staff, as well as at least one member of the ccTLD community. The VeriSign and IANA staffs communicated immediately about this issue, and the specific problem related to this request was fixed manually by VeriSign in the next zone revision after it was identified.
Because in any pair of hostnames affected by this result the IPv4 address was the same, the operational impact of this problem was minimal, if it was identifiable at all.
[2] Why was there no announcement that this problem existed?
The specific problem applied only to new instances of duplicate names, it did not affect existing duplicate names for which no change was in process. The total number of these duplicate hostnames is very small relative to the rest of the zone (less than 3%), and requests to modify domains which have these hostnames in their delegations are rare. There were no instances of such a request from November of 2004 through March of 2005, when AFNIC submitted the request for the RE ccTLD.
Because the issue affected only a small number of TLDs and name server operators, IANA staff chose to address the issue with those individuals directly. Often TLD managers appreciate such specific cases being dealt with on a personal level.
[3] Are safeguards now in place to prevent this sort of problem recurring?
Yes. VeriSign has informed us that the general problem which produced the original unexpected result has now been corrected. Additionally, IANA staff have enhanced efforts to monitor the "live" root zone from several different servers, and use that data to proactively monitor the results of change requests.
[4] What procedures does IANA (or ICANN?) have to make sure that changes to the TLD delegation process or problems with that process are properly communicated to its stakeholders?
The procedure that IANA uses to communicate changes in the TLD delegation process follows the typical ICANN formula of involving key stakeholders at early stages, opening proposals up for public comment, incorporating appropriate comments into the final version, and posting the procedure for its stakeholders. A relevant example of this process is the "Delegation Data Procedure," which is posted at http://www.iana.org/procedures/delegation-data.html.
Specific problems in the root management process are evaluated on a case by case basis. When necessary, there is a mailing list composed of the contact addresses for each of the TLD managers that can be utilized for any issue of wide interest to the TLD manager community. As noted above, cases which affect one, or a few TLDs are often dealt with on a one-to-one basis, with the appreciation of the relevant TLD manager(s).
[5] Were those procedures followed for this incident? If not, why not?
The procedure for communicating changes to the TLD delegation process was not followed in this specific case because there was no formal change of process.
The specific problem which produced the delay in processing AFNIC's request for RE was always understood to be temporary. As discussed above, rather than widely disseminate the information about this issue to the TLD manager community, IANA staff chose to communicate directly with those affected by the issue.
ICANN's goal, as always, is to cooperate with the TLD managers in implementing the changes they feel are appropriate for their TLDs, while also maintaining the stability of the DNS. We regret if our cautious handling of this particular request, or the timing of our responses to the group's reasonable questions, has caused any undue confusion in the DNS community about ICANN's dedication to that goal. We welcome comment and discussion on these topics.
I am enclosing below some information about the current (at this writing, 2005061000) root zone that may be relevant to this discussion. The first part is the complete list of IPv4 addresses that have multiple hostnames associated. There are no IPv6 addresses with multiple hostnames. The second part is a compilation of statistics about the current zone.
Regards,
Doug
128.112.129.15: dns.princeton.edu c.ext.nic.fr 192.134.0.49: c.nic.fr ns3.nic.fr 192.36.144.116: slave.sth.netnod.se slave1.sth.netnod.se 192.93.0.1: a.nic.fr ns1.nic.fr 192.93.0.4: b.nic.fr ns2.nic.fr 193.176.144.6: e.ext.nic.fr ns3.domain-registry.nl 193.51.208.13: dns.inria.fr a.ext.nic.fr 204.152.184.64: ns-ext.vix.com ns-ext.isc.org 204.74.112.1: tld1.basicfusion.net tld1.ultradns.net ud1.ns.net.ua 204.74.113.1: tld2.basicfusion.net tld2.ultradns.net
Statistics ========== Number of gTLDs: 15 Number of ccTLDs: 245 Total number of TLDs: 260
Number of IPv4 hosts: 798 Number of IPv4 addresses: 798
Number of IPv6 hosts: 40 Number of IPv6 addresses: 41 TLDs with IPv6 glue: 50
Total TLD name server hosts: 797
-- Doug Barton General Manager, The Internet Assigned Numbers Authority
Marcel, Thanks for these questions. I've responded in line below to (hopefully) make the responses more clear. Marcel Schneider wrote:
On Friday, 10 Jun 2005, Doug Barton writes:
Dear Doug
Thanks for responding to Jim's questions. Reading your mail I still have difficulties understanding two issues:
a) You state in [1] that after editing the root zone, VeriSign experienced an "unexpected result". What exactly was the result
As described in the first paragraph of my answers below, the wrong hostname appeared in a few TLD delegations. For example, if there was an existing hostname ns1.foo.example which pointed to 10.0.0.1, and this hostname appeared in the delegation of the .aa ccTLD; then a new hostname was introduced with the same IP address, the problem manifested itself as the new hostname appearing in the delegation for .aa, instead of the correct (old) hostname. This error was caught by the VeriSign staff during their change management process for root zone changes, and brought to the attention of the IANA staff before that version of the zone was published. A new root zone with the correct changes was then published, ans verified by the ICANN and VeriSign staffs. Unfortunately, subsequent production of the root zone that should not have had changes to the delegations with the duplicate hostnames reflected the initial problem, and this result was not observed before the new zone went live. The unplanned changes to the delegations were noticed both by IANA staff, and at least one member of the ccTLD community, and when brought to VeriSign's attention, was corrected as described below.
what caused it ? Whould also appreciate if you could elaborate on how VeriSign was able to fix this "unexpected result".
ICANN was not informed of the details of what caused the problem, or how it was fixed. We have asked VeriSign to provide us, and the community with additional details about this incident.
b) When this "unexpected result" was regarded by IANA staff as temporary and as having minimal operational impact, why was AFNIC then sent this response to their application:
"IANA is no longer adding additional examples of host names in delegation records where multiple host names point to the same IP address".
The "no longer" suggests a policy change. Why did IANA not just say: "Folks we have a problem here, wait a moment until we've fixed it" ?
AFNIC had requested a clear statement about why an alternative solution was being discussed to fulfill their operational goals. My statement to AFNIC was, "Due to problems encountered in the past, we are no longer adding additional examples of hostnames in delegation records where multiple hostnames point to the same IP address." In retrospect, my statement was made in more absolute terms than actually intended, or needed. I would once again like to offer my apologies for the confusion caused by this incident. My message to AFNIC was not intended to convey any kind of policy change on the part of the IANA, and I appreciate this opportunity to clarify the situation. For those attending the CENTR meeting in Trondheim this week I would love to chat with you more in person about this, or any other issue of interest. Warmest regards, Doug -- Doug Barton General Manager, The Internet Assigned Numbers Authority
Thanks again for the opportunity to discuss these issues. I hope that the group finds these answers satisfactory. We are of course happy to discuss this in further detail if desired.
[1] What was the nature of the technical problem that prevented multiple names in for an IP address and how was it resolved?
In November of 2004 AFNIC asked to have several hostnames added to the root zone that resolved to the same IPv4 addresses as hostnames that already existed. When VeriSign edited the root zone to include these new names, an unexpected result occurred. In some of the TLDs that had the pre-existing hostnames for these servers, the new hostnames appeared in the delegations inappropriately. This problem was identified within several hours of the zone going live by VeriSign staff, as well as at least one member of the ccTLD community. The VeriSign and IANA staffs communicated immediately about this issue, and the specific problem related to this request was fixed manually by VeriSign in the next zone revision after it was identified.
Because in any pair of hostnames affected by this result the IPv4 address was the same, the operational impact of this problem was minimal, if it was identifiable at all.
[2] Why was there no announcement that this problem existed?
The specific problem applied only to new instances of duplicate names, it did not affect existing duplicate names for which no change was in process. The total number of these duplicate hostnames is very small relative to the rest of the zone (less than 3%), and requests to modify domains which have these hostnames in their delegations are rare. There were no instances of such a request from November of 2004 through March of 2005, when AFNIC submitted the request for the RE ccTLD.
Because the issue affected only a small number of TLDs and name server operators, IANA staff chose to address the issue with those individuals directly. Often TLD managers appreciate such specific cases being dealt with on a personal level.
[3] Are safeguards now in place to prevent this sort of problem recurring?
Yes. VeriSign has informed us that the general problem which produced the original unexpected result has now been corrected. Additionally, IANA staff have enhanced efforts to monitor the "live" root zone from several different servers, and use that data to proactively monitor the results of change requests.
[4] What procedures does IANA (or ICANN?) have to make sure that changes to the TLD delegation process or problems with that process are properly communicated to its stakeholders?
The procedure that IANA uses to communicate changes in the TLD delegation process follows the typical ICANN formula of involving key stakeholders at early stages, opening proposals up for public comment, incorporating appropriate comments into the final version, and posting the procedure for its stakeholders. A relevant example of this process is the "Delegation Data Procedure," which is posted at http://www.iana.org/procedures/delegation-data.html.
Specific problems in the root management process are evaluated on a case by case basis. When necessary, there is a mailing list composed of the contact addresses for each of the TLD managers that can be utilized for any issue of wide interest to the TLD manager community. As noted above, cases which affect one, or a few TLDs are often dealt with on a one-to-one basis, with the appreciation of the relevant TLD manager(s).
[5] Were those procedures followed for this incident? If not, why not?
The procedure for communicating changes to the TLD delegation process was not followed in this specific case because there was no formal change of process.
The specific problem which produced the delay in processing AFNIC's request for RE was always understood to be temporary. As discussed above, rather than widely disseminate the information about this issue to the TLD manager community, IANA staff chose to communicate directly with those affected by the issue.
ICANN's goal, as always, is to cooperate with the TLD managers in implementing the changes they feel are appropriate for their TLDs, while also maintaining the stability of the DNS. We regret if our cautious handling of this particular request, or the timing of our responses to the group's reasonable questions, has caused any undue confusion in the DNS community about ICANN's dedication to that goal. We welcome comment and discussion on these topics.
I am enclosing below some information about the current (at this writing, 2005061000) root zone that may be relevant to this discussion. The first part is the complete list of IPv4 addresses that have multiple hostnames associated. There are no IPv6 addresses with multiple hostnames. The second part is a compilation of statistics about the current zone.
Regards,
Doug
128.112.129.15: dns.princeton.edu c.ext.nic.fr 192.134.0.49: c.nic.fr ns3.nic.fr 192.36.144.116: slave.sth.netnod.se slave1.sth.netnod.se 192.93.0.1: a.nic.fr ns1.nic.fr 192.93.0.4: b.nic.fr ns2.nic.fr 193.176.144.6: e.ext.nic.fr ns3.domain-registry.nl 193.51.208.13: dns.inria.fr a.ext.nic.fr 204.152.184.64: ns-ext.vix.com ns-ext.isc.org 204.74.112.1: tld1.basicfusion.net tld1.ultradns.net ud1.ns.net.ua 204.74.113.1: tld2.basicfusion.net tld2.ultradns.net
Statistics ========== Number of gTLDs: 15 Number of ccTLDs: 245 Total number of TLDs: 260
Number of IPv4 hosts: 798 Number of IPv4 addresses: 798
Number of IPv6 hosts: 40 Number of IPv6 addresses: 41 TLDs with IPv6 glue: 50
Total TLD name server hosts: 797
-- Doug Barton General Manager, The Internet Assigned Numbers Authority
Dear Doug, I share the questions of Marcel Schneider: - what was the unexpected result? - what did cause it? - what did permit to decide it could be addressed in small committee? - what did permit to immediately understand the problem was temporary? - how was it fixed? - how to be sure it will not repeat? - why was the mail received by AFNIC phrased that way? - I have some concerns learning that Verisign has the capacity to enter manually in the root? Why was the problem corrected in changing the root file and not the entry in the IANA file? I am sorry for beeing formal about this. But I frankly do not understand anything here. We tested that a distributed multiple automated building of the root from TLD Manager controled multiple on-line copies of their data would be the most secure, interference free solution and most easy for mutual control. Depending on your responses, this case could contradict or confirm this result. As I am writing a Draft to include it into a community test bed along ICP-3 guidelines, I would really want to understand, not propose testing a wrong approach. I thank you for your comments. Sincerely yours. jfc morfin At 14:02 13/06/2005, Marcel Schneider wrote:
On Friday, 10 Jun 2005, Doug Barton writes:
Dear Doug
Thanks for responding to Jim's questions. Reading your mail I still have difficulties understanding two issues:
a) You state in [1] that after editing the root zone, VerSign experienced an "unexpected result". What exactly was the result what caused it ? Whould also appreciate if you could elaborate on how VerSign was able to fix this "unexpected result".
b) When this "unexpected result" was regarded by IANA staff as temporary and as having minimal operational impact, why was AFNIC then sent this response to their application:
"IANA is no longer adding additional examples of host names in delegation records where multiple host names point to the same IP address".
The "no longer" suggests a policy change. Why did IANA not just say: "Folks we have a problem here, wait a moment until we've fixed it" ?
Marcel
Thanks again for the opportunity to discuss these issues. I hope that the group finds these answers satisfactory. We are of course happy to discuss this in further detail if desired.
[1] What was the nature of the technical problem that prevented multiple names in for an IP address and how was it resolved?
In November of 2004 AFNIC asked to have several hostnames added to the root zone that resolved to the same IPv4 addresses as hostnames that already existed. When VeriSign edited the root zone to include these new names, an unexpected result occurred. In some of the TLDs that had the pre-existing hostnames for these servers, the new hostnames appeared in the delegations inappropriately. This problem was identified within several hours of the zone going live by VeriSign staff, as well as at least one member of the ccTLD community. The VeriSign and IANA staffs communicated immediately about this issue, and the specific problem related to this request was fixed manually by VeriSign in the next zone revision after it was identified.
Because in any pair of hostnames affected by this result the IPv4 address was the same, the operational impact of this problem was minimal, if it was identifiable at all.
[2] Why was there no announcement that this problem existed?
The specific problem applied only to new instances of duplicate names, it did not affect existing duplicate names for which no change was in process. The total number of these duplicate hostnames is very small relative to the rest of the zone (less than 3%), and requests to modify domains which have these hostnames in their delegations are rare. There were no instances of such a request from November of 2004 through March of 2005, when AFNIC submitted the request for the RE ccTLD.
Because the issue affected only a small number of TLDs and name server operators, IANA staff chose to address the issue with those individuals directly. Often TLD managers appreciate such specific cases being dealt with on a personal level.
[3] Are safeguards now in place to prevent this sort of problem recurring?
Yes. VeriSign has informed us that the general problem which produced the original unexpected result has now been corrected. Additionally, IANA staff have enhanced efforts to monitor the "live" root zone from several different servers, and use that data to proactively monitor the results of change requests.
[4] What procedures does IANA (or ICANN?) have to make sure that changes to the TLD delegation process or problems with that process are properly communicated to its stakeholders?
The procedure that IANA uses to communicate changes in the TLD delegation process follows the typical ICANN formula of involving key stakeholders at early stages, opening proposals up for public comment, incorporating appropriate comments into the final version, and posting the procedure for its stakeholders. A relevant example of this process is the "Delegation Data Procedure," which is posted at http://www.iana.org/procedures/delegation-data.html.
Specific problems in the root management process are evaluated on a case by case basis. When necessary, there is a mailing list composed of the contact addresses for each of the TLD managers that can be utilized for any issue of wide interest to the TLD manager community. As noted above, cases which affect one, or a few TLDs are often dealt with on a one-to-one basis, with the appreciation of the relevant TLD manager(s).
[5] Were those procedures followed for this incident? If not, why not?
The procedure for communicating changes to the TLD delegation process was not followed in this specific case because there was no formal change of process.
The specific problem which produced the delay in processing AFNIC's request for RE was always understood to be temporary. As discussed above, rather than widely disseminate the information about this issue to the TLD manager community, IANA staff chose to communicate directly with those affected by the issue.
ICANN's goal, as always, is to cooperate with the TLD managers in implementing the changes they feel are appropriate for their TLDs, while also maintaining the stability of the DNS. We regret if our cautious handling of this particular request, or the timing of our responses to the group's reasonable questions, has caused any undue confusion in the DNS community about ICANN's dedication to that goal. We welcome comment and discussion on these topics.
I am enclosing below some information about the current (at this writing, 2005061000) root zone that may be relevant to this discussion. The first part is the complete list of IPv4 addresses that have multiple hostnames associated. There are no IPv6 addresses with multiple hostnames. The second part is a compilation of statistics about the current zone.
Regards,
Doug
128.112.129.15: dns.princeton.edu c.ext.nic.fr 192.134.0.49: c.nic.fr ns3.nic.fr 192.36.144.116: slave.sth.netnod.se slave1.sth.netnod.se 192.93.0.1: a.nic.fr ns1.nic.fr 192.93.0.4: b.nic.fr ns2.nic.fr 193.176.144.6: e.ext.nic.fr ns3.domain-registry.nl 193.51.208.13: dns.inria.fr a.ext.nic.fr 204.152.184.64: ns-ext.vix.com ns-ext.isc.org 204.74.112.1: tld1.basicfusion.net tld1.ultradns.net ud1.ns.net.ua 204.74.113.1: tld2.basicfusion.net tld2.ultradns.net
Statistics ========== Number of gTLDs: 15 Number of ccTLDs: 245 Total number of TLDs: 260
Number of IPv4 hosts: 798 Number of IPv4 addresses: 798
Number of IPv6 hosts: 40 Number of IPv6 addresses: 41 TLDs with IPv6 glue: 50
Total TLD name server hosts: 797
-- Doug Barton General Manager, The Internet Assigned Numbers Authority
On Jun 22, 2005, at 00:01, JFC (Jefsey) Morfin wrote:
- what was the unexpected result? - what did cause it? - what did permit to decide it could be addressed in small committee? - what did permit to immediately understand the problem was temporary? - how was it fixed? - how to be sure it will not repeat? - why was the mail received by AFNIC phrased that way? - I have some concerns learning that Verisign has the capacity to enter manually in the root? Why was the problem corrected in changing the root file and not the entry in the IANA file?
Jefsey, these questions have already been adequately answered in the responses from Doug Barton on June 11th and 13th. Matt Larson's clarification on the 15th was also helpful. If those replies were not clear enough for you, please take it up directly with the authors of those messages. I'd appreciate it if you did not carry out that dialogue on the DNS WG list. I see no need to continue this discussion any further in this mailing list. Alexander Gall made some comments on the processes surrounding the to-be-replaced template. Aside from his observations, the list has been silent. So the WG should now consider this discussion closed. There may well be some questions on layer-9 issues about transparency and process at IANA that have arisen as a result of this incident. If these exist, they should be taken somewhere else, presumably ICANN. This WG is for operational and technical discussions about the DNS. It does not do layer-9 stuff.
As I am writing a Draft to include it into a community test bed along ICP-3 guidelines, I would really want to understand, not propose testing a wrong approach.
Hmm. I would have thought the approach to take would be to write the draft, see if the ideas that were proposed had any merit and then implement and test them. That's normally how things get done at the IETF. [Authors of drafts will typically circulate them to colleagues for feedback before they get published.] Feel free to complete your draft and submit it to the IETF in the usual manner. Or if you'd prefer to have the DNS WG discuss your draft, post it to the list where I'm sure it will get the attention it deserves. BTW Jefsey, please fix your mail client or at least trim your postings. There's no need for everyone to see everything that's been posted on this thread over and over again by appending the whole of the message you're replying to.
At 02:38 22/06/2005, Jim Reid wrote:
On Jun 22, 2005, at 00:01, JFC (Jefsey) Morfin wrote:
- what was the unexpected result? - what did cause it? - what did permit to decide it could be addressed in small committee? - what did permit to immediately understand the problem was temporary? - how was it fixed? - how to be sure it will not repeat? - why was the mail received by AFNIC phrased that way? - I have some concerns learning that Verisign has the capacity to enter manually in the root? Why was the problem corrected in changing the root file and not the entry in the IANA file?
Jefsey, these questions have already been adequately answered in the responses from Doug Barton on June 11th and 13th. Matt Larson's clarification on the 15th was also helpful. If those replies were not clear enough for you, please take it up directly with the authors of those messages.
Dear Jim, I do not think they answer the concerns I have.
I'd appreciate it if you did not carry out that dialogue on the DNS WG list. I see no need to continue this discussion any further in this mailing list. Alexander Gall made some comments on the processes surrounding the to-be-replaced template. Aside from his observations, the list has been silent. So the WG should now consider this discussion closed.
It is not silent since I rose these points. Political decisions may result from this, this is why I would have prefered to be exhaustive and fair. But I do not understand your political layer: now we discuss source code and operational procedures.
[Authors of drafts will typically circulate them to colleagues for feedback before they get published.]
This is a BCP based upon three years of work and test by an independent organisation (IETF was not interested). What is reported here is more experimental than operational. Hence my interest.
Feel free to complete your draft and submit it to the IETF in the usual manner. Or if you'd prefer to have the DNS WG discuss your draft, post it to the list where I'm sure it will get the attention it deserves.
Certainly.
BTW Jefsey, please fix your mail client or at least trim your postings. There's no need for everyone to see everything that's been posted on this thread over and over again by appending the whole of the message you're replying to.
Apologies. jfc
I do not think they answer the concerns I have. As Jim has said: >Aside from his observations, the >list has been silent. So the WG should now consider this discussion closed. It is not silent since I rose these points. Political decisions may result from this, this is why I would have prefered to be exhaustive and fair. But I do not understand your political layer: now we discuss source code and operational procedures. There has been no discussion. >[Authors of drafts will typically circulate them to colleagues for >feedback before they get published.] This is a BCP based upon three years of work and test by an independent organisation (IETF was not interested). What is reported here is more experimental than operational. Hence my interest. Why don't you just publish it? I only see it mentioned by you on various mailing lists, but the actual thing never surfaced. jaap
participants (5)
-
Doug Barton -
Jaap Akkerhuis -
JFC (Jefsey) Morfin -
Jim Reid -
Marcel Schneider