Extension Headers / Impact on Security Devices
Hi, hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs! With regard to the strong position on ext hdrs that I expressed in the session yesterday pls allow to add some material/sources: - this is a paper laying out how we could circumvent some major (both commercial and FOSS) IDPS solutions in their at the time of testing latest versions, by various combinations of extension headers and fragmentation: https://www.ernw.de/download/eu-14-Atlasis-Rey-Schaefer-briefings-Evasion-of.... - here's some thoughts and preliminary results of tests performed to circumvent stateless ACLs: http://www.insinuator.net/2015/01/evasion-of-cisco-acls-by-abusing-ipv6-disc.... http://www.insinuator.net/2015/01/the-persistent-problem-of-state-in-ipv6-se.... We have a (somewhat) ongoing research project looking more closely on the interaction of ACLs and extension headers. For the moment I can only state that it's not just Cisco who are "affected". More results will be available in some months. If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion. We could even bring some devices to Bucharest to do some practical testing and demos "in some side room" during #RIPE71, for those interested. We can even do the latter without any official part in a session, if there's interest. As for the topic itself I'd like to summarize our position as follows: - it has not happened in the past 17 yrs (since publication of RFC2460) that compelling, Internet-scale use cases of extension headers have been brought up. - we're hence quite sceptical as for the "we might see cool use cases in 15-20 yrs" position someone expressed at the mic. - from a security perspective they turn out to be a nightmare for (a number of) current security architectures and controls. it is hence understandable (and we actually advise to do so) that they are blocked at the _border_ of networks that have not yet managed to identify a compelling use case. - looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order, options, fragmentable vs. unfragmentable part etc.) we do not expect that type of security issues to disappear soon (the main reason why we did not continue the IDPS research was that the involved student eventually delivered this thesis. one can probably find much more issues provided time & resources). Adding more parsing cycles & intelligence (read: silicon) is not an option, at least not for sth that doesn't have a use case. - the results of this (blocking) approach have been observed and documented by Jen & Fernando and others (Tim Chown). - now that "this vicious circle" has gained sufficient ground it will be even less incentivized to develop a compelling use case. which is why we do not expect to see one in the future. === Everybody have a great weekend and for those still in AMS: have a safe trip home! Best Enno -- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
Hi Enno and list, Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1: When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order: followed on the next page by Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header). Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such. The impact here is actually at least twofold: - It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router. - As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered. The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes. Cheers, Benedikt -- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/ Business Grade IPv6 --- Consulting, Training, Projects BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
Hi I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it. Couldn't we update RFC2460 and make this list a strict order? I would want my firewall to notify me if the EHs in a packet do not follow the list. And limiting the number of possible EHs per packet might be a good idea. Silvia -----Ursprüngliche Nachricht----- Von: ipv6-wg [mailto:ipv6-wg-bounces@ripe.net] Im Auftrag von Benedikt Stockebrand Gesendet: Sonntag, 17. Mai 2015 18:39 An: ipv6-wg@ripe.net Betreff: Re: [ipv6-wg] Extension Headers / Impact on Security Devices Hi Enno and list, Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1: When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order: followed on the next page by Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header). Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such. The impact here is actually at least twofold: - It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router. - As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered. The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes. Cheers, Benedikt -- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/ Business Grade IPv6 --- Consulting, Training, Projects BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
Hi Silvia, On Sun, May 17, 2015 at 06:43:11PM +0000, Silvia Hagen wrote:
Hi
I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it.
Couldn't we update RFC2460 and make this list a strict order?
good luck bringing this suggestion to IETF 6man... ;-)
I would want my firewall to notify me if the EHs in a packet do not follow the list.
actually when we tested a number of commercial firewalls in late 2013 (results here: https://www.ernw.de/download/TR14_IPv6SecSummit_AAtlasis-RISC-project_result...) it turned out that pretty much all of them follow a (from our perspective "reasonable approach", that is dropping "unusal combinations or number of ext_hdrs" by default. (which, of course, violates RFC 2460. but apparently all major vendors follow a line of reasoning similar to ours). so, in short, firewalls "are not affected".
And limiting the number of possible EHs per packet might be a good idea.
yes, that _would actually be_ a good idea. I'm cross-posting this to v6ops as there's a related discussion going on right now. pls forgive this potential Usenet etiquette violation. best Enno
Silvia
-----Urspr?ngliche Nachricht----- Von: ipv6-wg [mailto:ipv6-wg-bounces@ripe.net] Im Auftrag von Benedikt Stockebrand Gesendet: Sonntag, 17. Mai 2015 18:39 An: ipv6-wg@ripe.net Betreff: Re: [ipv6-wg] Extension Headers / Impact on Security Devices
Hi Enno and list,
Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1:
When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order:
followed on the next page by
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such.
The impact here is actually at least twofold:
- It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router.
- As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered.
The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes.
Cheers,
Benedikt
-- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/
Business Grade IPv6 --- Consulting, Training, Projects
BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
-- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
I'm of course missing the preceding email. No problem with the cross-posting, but let's get the question on the table? Is this section 4.1's statement that "it is recommended that those headers appear in the following order"? I have a hunch that an internet draft requiring headers to be in the listed order would be looked at with approbation. The issue, if any, would likely have regard to repeated Destination Option headers. However Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header). seems to me fairly definitive. Fernando would like to see a specification of how long the entire list of extension headers may be, in terms of "how large of a hardware buffer has to be implemented in a switch?" That has a couple of issues related to "why does one have to ask the question?" The hardware should definitely be large enough to read in the IPv6 header, the HBH Header, and the Routing Header. After that, the only host that SHOULD need to read it all in is the host itself, and the host can do its processing from RAM. There is one "gotcha" case, which is when someone uses an ACL (disallowing Fragmentation Headers, perhaps, or looking for the TCP port, meaning one has to ask whether a given packet is TCP and what its port number is). The firewall is going to have to chase down the chain of extension headers. How big is a HBH header? How big is a routing option such as the Segment Routing option? If that's a list of 27 services, each with a 16 byte IPv6 address...
On May 17, 2015, at 12:18 PM, Enno Rey <erey@ernw.de> wrote:
Hi Silvia,
On Sun, May 17, 2015 at 06:43:11PM +0000, Silvia Hagen wrote:
Hi
I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it.
Couldn't we update RFC2460 and make this list a strict order?
good luck bringing this suggestion to IETF 6man... ;-)
I would want my firewall to notify me if the EHs in a packet do not follow the list.
actually when we tested a number of commercial firewalls in late 2013 (results here: https://www.ernw.de/download/TR14_IPv6SecSummit_AAtlasis-RISC-project_result...) it turned out that pretty much all of them follow a (from our perspective "reasonable approach", that is dropping "unusal combinations or number of ext_hdrs" by default. (which, of course, violates RFC 2460. but apparently all major vendors follow a line of reasoning similar to ours). so, in short, firewalls "are not affected".
And limiting the number of possible EHs per packet might be a good idea.
yes, that _would actually be_ a good idea.
I'm cross-posting this to v6ops as there's a related discussion going on right now. pls forgive this potential Usenet etiquette violation.
best
Enno
Silvia
-----Urspr?ngliche Nachricht----- Von: ipv6-wg [mailto:ipv6-wg-bounces@ripe.net] Im Auftrag von Benedikt Stockebrand Gesendet: Sonntag, 17. Mai 2015 18:39 An: ipv6-wg@ripe.net Betreff: Re: [ipv6-wg] Extension Headers / Impact on Security Devices
Hi Enno and list,
Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1:
When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order:
followed on the next page by
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such.
The impact here is actually at least twofold:
- It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router.
- As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered.
The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes.
Cheers,
Benedikt
-- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/
Business Grade IPv6 --- Consulting, Training, Projects
BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
-- Enno Rey
ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902
Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey
======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
Fred, All, On Wed, Jun 17, 2015 at 07:41:33AM +0000, Fred Baker (fred) wrote:
I'm of course missing the preceding email. No problem with the cross-posting, but let's get the question on the table? Is this section 4.1's statement that "it is recommended that those headers appear in the following order"?
I have a hunch that an internet draft requiring headers to be in the listed order would be looked at with approbation. The issue, if any, would likely have regard to repeated Destination Option headers. However
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
seems to me fairly definitive.
I'm not so sure about this. First probably a capital "SHOULD" would have been more appropriate. Second - and this is our main point/observation - there's too much space of interpretation for actual implementation. To underline this I will just cite two somewhat random sections from the study we performed on evading security controls (IDPS systems) by means of extension headers, combined with fragmentation (https://www.ernw.de/download/eu-14-Atlasis-Rey-Schaefer-briefings-Evasion-of...). One of the devices (all of them latest code incl. some high-end gear) could be evaded by the following: "Case 1: - An IPv6 Routing Header Type 0 in the unfragmentable part is used. - AND a IPv6 Destination Option header is part of the fragmentable piece of the IPv6 datatagram. - AND The IPv6 Destinations Option header is padded with six (6) octets of bytes (at least). - AND The IPv6 datagram is fragmented in 2 fragments. Case 2: - An IPv6 Hop-by-Hop Option Header and Routing Header Type 0 in the unfragmentable part is used - AND a IPv6 Destination Option header is part of the fragmentable piece of the IPv6 datatagram. - AND The IPv6 datagram is fragmented in 2 fragments. Case 3: - An IPv6 Destination Option Header and Routing Header Type 0 in the unfragmentable part is used. - AND a IPv6 Destination Option header is part of the fragmentable piece of the IPv6 datatagram. - AND The IPv6 datagram is fragmented in 2 fragments. Case 4: - An IPv6 Hop-by-Hop, Destination Option Header and Routing Header Type 0 in the unfragmentable part is used - AND a IPv6 Destination Option header is part of the fragmentable piece of the IPv6 datatagram. - AND The IPv6 datagram is fragmented in 2 fragments." Another one by the following: "a. The unfragmentable part consists of three (3) Destination Option headers b. The fragmentable part consists of two (2) Destination Option headers plus the layer 4 header. c. The aforementioned datagram is split in two fragments." It should be noted that most operating systems happily accept and process (incl., ofc, reassembly) packets crafted in such ways. [see results section in https://www.ernw.de/download/Advanced%20Attack%20Techniques%20against%20IPv6...]. It should further be noted that some of the techniques we used would even have worked in settings where RFC 7112 would have been strictly applied.
Fernando would like to see a specification of how long the entire list of extension headers may be, in terms of "how large of a hardware buffer has to be implemented in a switch?" That has a couple of issues related to "why does one have to ask the question?" The hardware should definitely be large enough to read in the IPv6 header, the HBH Header, and the Routing Header. After that, the only host that SHOULD need to read it all in is the host itself, and the host can do its processing from RAM. There is one "gotcha" case, which is when someone uses an ACL (disallowing Fragmentation Headers, perhaps, or looking for the TCP port, meaning one has to ask whether a given packet is TCP and what its port number is). The firewall is going to have to chase down the chain of extension headers.
Which might be a quite difficult task (not least performance-wise) once this chain, by definition, is abitrarily long. best Enno
How big is a HBH header? How big is a routing option such as the Segment Routing option? If that's a list of 27 services, each with a 16 byte IPv6 address...
On May 17, 2015, at 12:18 PM, Enno Rey <erey@ernw.de> wrote:
Hi Silvia,
On Sun, May 17, 2015 at 06:43:11PM +0000, Silvia Hagen wrote:
Hi
I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it.
Couldn't we update RFC2460 and make this list a strict order?
good luck bringing this suggestion to IETF 6man... ;-)
I would want my firewall to notify me if the EHs in a packet do not follow the list.
actually when we tested a number of commercial firewalls in late 2013 (results here: https://www.ernw.de/download/TR14_IPv6SecSummit_AAtlasis-RISC-project_result...) it turned out that pretty much all of them follow a (from our perspective "reasonable approach", that is dropping "unusal combinations or number of ext_hdrs" by default. (which, of course, violates RFC 2460. but apparently all major vendors follow a line of reasoning similar to ours). so, in short, firewalls "are not affected".
And limiting the number of possible EHs per packet might be a good idea.
yes, that _would actually be_ a good idea.
I'm cross-posting this to v6ops as there's a related discussion going on right now. pls forgive this potential Usenet etiquette violation.
best
Enno
Silvia
-----Urspr?ngliche Nachricht----- Von: ipv6-wg [mailto:ipv6-wg-bounces@ripe.net] Im Auftrag von Benedikt Stockebrand Gesendet: Sonntag, 17. Mai 2015 18:39 An: ipv6-wg@ripe.net Betreff: Re: [ipv6-wg] Extension Headers / Impact on Security Devices
Hi Enno and list,
Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1:
When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order:
followed on the next page by
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such.
The impact here is actually at least twofold:
- It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router.
- As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered.
The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes.
Cheers,
Benedikt
-- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/
Business Grade IPv6 --- Consulting, Training, Projects
BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
-- Enno Rey
ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902
Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey
======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
-- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
On Jun 17, 2015, at 1:14 AM, Enno Rey <erey@ernw.de> wrote:
Fred, All,
On Wed, Jun 17, 2015 at 07:41:33AM +0000, Fred Baker (fred) wrote:
I'm of course missing the preceding email. No problem with the cross-posting, but let's get the question on the table? Is this section 4.1's statement that "it is recommended that those headers appear in the following order"?
I have a hunch that an internet draft requiring headers to be in the listed order would be looked at with approbation. The issue, if any, would likely have regard to repeated Destination Option headers. However
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
seems to me fairly definitive.
I'm not so sure about this. First probably a capital "SHOULD" would have been more appropriate.
RFC 2460 doesn't use the RFC 2119 conventions. Yes, it's hard to remember now, but RFC 2119 conventions weren't always the rule; they derived from similar usage in RFCs 1122/1123 and 1812. History lesson in the presence of adult beverages if that's interesting.
Second - and this is our main point/observation - there's too much space of interpretation for actual implementation. To underline this I will just cite two somewhat random sections from the study we performed on evading security controls (IDPS systems) by means of extension headers, combined with fragmentation (https://www.ernw.de/download/eu-14-Atlasis-Rey-Schaefer-briefings-Evasion-of...). One of the devices (all of them latest code incl. some high-end gear) could be evaded by the following:
Cutting to the chase, on the surface it looks like most of the cases you specified conform to the sequence rule, with the exception of one that has three Destination Option Headers. What I *think* you're pointing out is (from your PDF) that this is part of a sequence of messages, one of which is fragmented, accepted by the intrusion detector, and not accepted by the ultimate host (perhaps a TCP checksum failed?), and as a result bypasses the signature that the intrusion detector is looking for. The issue wasn't that the sequence was incorrect, in most cases; it was that the intrusion detector made a different choice than the end host. Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
Fred, On Wed, Jun 17, 2015 at 04:46:03PM +0000, Fred Baker (fred) wrote:
Cutting to the chase, on the surface it looks like most of the cases you specified conform to the sequence rule, with the exception of one that has three Destination Option Headers. What I *think* you're pointing out is (from your PDF) that this is part of a sequence of messages, one of which is fragmented, accepted by the intrusion detector, and not accepted by the ultimate host (perhaps a TCP checksum failed?), and as a result bypasses the signature that the intrusion detector is looking for. The issue wasn't that the sequence was incorrect, in most cases; it was that the intrusion detector made a different choice than the end host.
Let's put it slightly different: the intrusion detector is supposed to detect/block certain application layer attacks. Which it does as long as those come in/pass by without extension headers/fragmentation. Once the attack packets are crafted in certain ways, the intrusion detector let's them through, the end host happily reassembles and gets hit. The end host does so as the overall datagram does not violate RFC 2460's (dubious) liberty. The detector is "blind" as parsing a single datagram split into an arbitrary number of fragments with an arbitrary number of extension headers in a (mostly) arbitrary order is a, well, complex task. It's not only intrusion detectors facing this type of difficulty, it's other "black list approach" security controls (like so-called First Hop Security features or Infrastructure ACLs), too. [see, for example, http://www.insinuator.net/2015/01/dhcpv6-guard-do-it-like-ra-guard-evasion/).
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)?
'm tempted to reply that this would make the Internet world a safer place (without losing any current functionality) and might speeden up overall IPv6 deployment, as some organizations had one (for some of them: severe) concern less. So it's not about my personal happiness; though the factors mentioned might actually contribute to that as well ;-).
From that angle RFC 7112 is certainly a step into the right direction. It's just not widely implemented - while IPv6 is "finally here" - and it doesn't "solve" certain cases (see the link above). That's why I think that a major revision of the whole extension header concept is needed.
Best Enno -- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
Hi, On Wed, Jun 17, 2015 at 04:46:03PM +0000, Fred Baker (fred) wrote:
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It would certainly make *me* happier... Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
On 6/18/2015 3:00 PM, Gert Doering wrote:
Hi,
On Wed, Jun 17, 2015 at 04:46:03PM +0000, Fred Baker (fred) wrote:
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It makes sense to require all of the HBH and routing headers in the first fragment. The rest is impossible to mandate and irrelevant. Anything that looks far enough into a packet to need to find the transport header might be looking several layers of encapsulation in; that's acting as an endpoint, and ought to reassemble the packet at that point. Joe
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It would certainly make *me* happier…
done. RFC7112. cheers, Ole
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It would certainly make *me* happier$,1s&(B
done. RFC7112.
As I wrote in another mail,
It may be relevant to ask for RFC 7112 support next time we're doing an equipment RFQ (in a few years). ... But until RFC 7112 support is available, I believe we will see a significant amount of breakage for IPv6 extension headers - and header chains will be limited to significantly less than 1280 bytes.
And until such support is available, we have to deal with the current mess. Which may imply more filtering than some people would like. Steinar Haug, AS 2116
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It would certainly make *me* happier�$,1s&
done. RFC7112.
As I wrote in another mail,
It may be relevant to ask for RFC 7112 support next time we're doing an equipment RFQ (in a few years). ... But until RFC 7112 support is available, I believe we will see a significant amount of breakage for IPv6 extension headers - and header chains will be limited to significantly less than 1280 bytes.
And until such support is available, we have to deal with the current mess. Which may imply more filtering than some people would like.
I don’t think that follows. cheers, Ole
Hi, On Fri, Jun 19, 2015 at 09:56:20AM +0200, Ole Troan wrote:
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It would certainly make *me* happier???$,1s&
done. RFC7112.
As I wrote in another mail,
It may be relevant to ask for RFC 7112 support next time we're doing an equipment RFQ (in a few years). ... But until RFC 7112 support is available, I believe we will see a significant amount of breakage for IPv6 extension headers - and header chains will be limited to significantly less than 1280 bytes.
And until such support is available, we have to deal with the current mess. Which may imply more filtering than some people would like.
I don???t think that follows.
I would second the observation that this (subsequent action) actually happens. Not least because many consider it a reasonable approach not to process and/or to drop something that induces complexity & insecurity and which at the same time is not needed by any service or application (read: all EHs except ESP and, maybe in some corner cases, AH+FH). thanks Enno
cheers, Ole
-- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
From: Enno Rey <erey@ernw.de> To: Ole Troan <otroan@employees.org> Cc: v6ops@ietf.org; ipv6-wg@ripe.net Sent: Friday, 19 June 2015, 18:34 Subject: Re: [v6ops] [ipv6-wg] Extension Headers / Impact on Security Devices Hi, On Fri, Jun 19, 2015 at 09:56:20AM +0200, Ole Troan wrote:
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It would certainly make *me* happier???$,1s&
done. RFC7112.
As I wrote in another mail,
It may be relevant to ask for RFC 7112 support next time we're doing an equipment RFQ (in a few years). ... But until RFC 7112 support is available, I believe we will see a significant amount of breakage for IPv6 extension headers - and header chains will be limited to significantly less than 1280 bytes.
And until such support is available, we have to deal with the current mess. Which may imply more filtering than some people would like.
I don???t think that follows.
I would second the observation that this (subsequent action) actually happens. Not least because many consider it a reasonable approach not to process and/or to drop something that induces complexity & insecurity and which at the same time is not needed by any service or application (read: all EHs except ESP and, maybe in some corner cases, AH+FH). / I think you're only expressing the view of many security engineers, not the many network engineers or the many more users of networks. At an ISP, if you block traffic your customer wants send, all you get is angry customers. They're paying you to deliver any and all of their packets as well as possible, with as minimum restrictions as possible, ideally none, rather than the opposite. This is why I don't think you'll ever get consensus on deprecating EHs, because I think you're only coming the position of trying to solve problem (3) I described in my last email. / I'd also observe that deprecating EHs, other than ESP, AH+FH, TCP and UDP, would also prevent them from being used behind the ESP EH - where you won't be able to see what is in them, so you won't be able to care what is in them. You'll also be preventing people from other transport layer protocols, such as SCTP and DCCP, that are feasible to deploy over the IPv6 Internet that couldn't be over the IPv4 network because of IPv4 NAT and other middle boxes. thanks Enno
cheers, Ole
-- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator ======================================================= _______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
On 19/06/2015 21:13, Mark ZZZ Smith wrote:
From: Enno Rey <erey@ernw.de> To: Ole Troan <otroan@employees.org> Cc: v6ops@ietf.org; ipv6-wg@ripe.net Sent: Friday, 19 June 2015, 18:34 Subject: Re: [v6ops] [ipv6-wg] Extension Headers / Impact on Security Devices
Hi,
On Fri, Jun 19, 2015 at 09:56:20AM +0200, Ole Troan wrote:
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It would certainly make *me* happier???$,1s&
done. RFC7112.
As I wrote in another mail,
It may be relevant to ask for RFC 7112 support next time we're doing an equipment RFQ (in a few years). ... But until RFC 7112 support is available, I believe we will see a significant amount of breakage for IPv6 extension headers - and header chains will be limited to significantly less than 1280 bytes.
And until such support is available, we have to deal with the current mess. Which may imply more filtering than some people would like.
I don???t think that follows.
I would second the observation that this (subsequent action) actually happens. Not least because many consider it a reasonable approach not to process and/or to drop something that induces complexity & insecurity and which at the same time is not needed by any service or application (read: all EHs except ESP and, maybe in some corner cases, AH+FH).
/ I think you're only expressing the view of many security engineers, not the many network engineers or the many more users of networks. At an ISP, if you block traffic your customer wants send, all you get is angry customers. They're paying you to deliver any and all of their packets as well as possible, with as minimum restrictions as possible, ideally none, rather than the opposite. This is why I don't think you'll ever get consensus on deprecating EHs, because I think you're only coming the position of trying to solve problem (3) I described in my last email.
I don't think that's what he was saying. I think he was saying "enforce RFC 7112." Discarding packets whose headers exceed the first fragment strikes me as very reasonable, even before host stacks are updated to enforce RFC 7112 at the source. What isn't reasonable is to say "laugh at RFC 7045". Anybody who purports to sell IPv6 firewalls needs to support RFC 7045 in their next release. The next order of business in the IETF (6man, not here) is to decide whether we go further in fixing the standard, which I think you and Fred are both advocating. Brian
/ I'd also observe that deprecating EHs, other than ESP, AH+FH, TCP and UDP, would also prevent them from being used behind the ESP EH - where you won't be able to see what is in them, so you won't be able to care what is in them. You'll also be preventing people from other transport layer protocols, such as SCTP and DCCP, that are feasible to deploy over the IPv6 Internet that couldn't be over the IPv4 network because of IPv4 NAT and other middle boxes. thanks
Enno
cheers, Ole
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
On Fri, Jun 19, 2015 at 5:13 AM, Mark ZZZ Smith <markzzzsmith@yahoo.com.au> wrote:
________________________________ From: Enno Rey <erey@ernw.de> To: Ole Troan <otroan@employees.org> Cc: v6ops@ietf.org; ipv6-wg@ripe.net Sent: Friday, 19 June 2015, 18:34 Subject: Re: [v6ops] [ipv6-wg] Extension Headers / Impact on Security Devices
Hi,
On Fri, Jun 19, 2015 at 09:56:20AM +0200, Ole Troan wrote:
Tell me this. Would you be happier if the fragmentation rule said that the first fragment had to contain the entire IPv6 header, plus the transport layer header (for ACL support)? I think Fernando would support such a statement (I think I have "heard" him make such a statement).
It would certainly make *me* happier???$,1s&
done. RFC7112.
As I wrote in another mail,
It may be relevant to ask for RFC 7112 support next time we're doing an equipment RFQ (in a few years). ... But until RFC 7112 support is available, I believe we will see a significant amount of breakage for IPv6 extension headers - and header chains will be limited to significantly less than 1280 bytes.
And until such support is available, we have to deal with the current mess. Which may imply more filtering than some people would like.
I don???t think that follows.
I would second the observation that this (subsequent action) actually happens. Not least because many consider it a reasonable approach not to process and/or to drop something that induces complexity & insecurity and which at the same time is not needed by any service or application (read: all EHs except ESP and, maybe in some corner cases, AH+FH).
/ I think you're only expressing the view of many security engineers, not the many network engineers or the many more users of networks. At an ISP, if you block traffic your customer wants send, all you get is angry customers.
... and at an ISP if your customer calls you up because they are being DDoSed out of existence with traffic at some port that they don't use, and you tell them at that you cannot filter UDP port 80 because, well, you cannot find it, you also end up with an angry customer.
They're paying you to deliver any and all of their packets as well as possible, with as minimum restrictions as possible, ideally none, rather than the opposite.
Perhaps. I figure they are /actually/ paying you to provide them with good service. Sometimes that includes filtering for them, because you have bigger pipes than them... W
This is why I don't think you'll ever get consensus on deprecating EHs, because I think you're only coming the position of trying to solve problem (3) I described in my last email.
/ I'd also observe that deprecating EHs, other than ESP, AH+FH, TCP and UDP, would also prevent them from being used behind the ESP EH - where you won't be able to see what is in them, so you won't be able to care what is in them. You'll also be preventing people from other transport layer protocols, such as SCTP and DCCP, that are feasible to deploy over the IPv6 Internet that couldn't be over the IPv4 network because of IPv4 NAT and other middle boxes.
thanks
Enno
cheers, Ole
-- Enno Rey
ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902
Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey
======================================================= Blog: || Conference: www.troopers.de Twitter: @Enno_Insinuator
=======================================================
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
Hi Warren, From: Warren Kumari <warren@kumari.net> To: Mark ZZZ Smith <markzzzsmith@yahoo.com.au> Cc: Enno Rey <erey@ernw.de>; Ole Troan <otroan@employees.org>; "v6ops@ietf.org" <v6ops@ietf.org>; "ipv6-wg@ripe.net" <ipv6-wg@ripe.net> Sent: Saturday, 20 June 2015, 6:19 Subject: Re: [v6ops] [ipv6-wg] Extension Headers / Impact on Security Devices On Fri, Jun 19, 2015 at 5:13 AM, Mark ZZZ Smith <markzzzsmith@yahoo.com.au> wrote:
________________________________ From: Enno Rey <erey@ernw.de> To: Ole Troan <otroan@employees.org> Cc: v6ops@ietf.org; ipv6-wg@ripe.net Sent: Friday, 19 June 2015, 18:34 Subject: Re: [v6ops] [ipv6-wg] Extension Headers / Impact on Security Devices
<snip>
/ I think you're only expressing the view of many security engineers, not the many network engineers or the many more users of networks. At an ISP, if you block traffic your customer wants send, all you get is angry customers.
... and at an ISP if your customer calls you up because they are being DDoSed out of existence with traffic at some port that they don't use, and you tell them at that you cannot filter UDP port 80 because, well, you cannot find it, you also end up with an angry customer. / If you can't find it, that may be because it isn't actually there. / Here's what I wrote in my earlier email what problem/problems being solved email about network capacity DoS: / "(1) I think TCP/UDP port level filtering for this purpose is certainly a nice to have but not a necessary to have, because network capacity DoSs will use some other protocol for their DoS if mitigating DoSs using TCP/UDP traffic becomes too effective. Completely dropping DoS traffic identified just by src and/or dst address, or plonking that traffic into a scavenger QoS class for a while if complete dropping is too severe is a general solution that will work regardless of the type of DoS traffic. Using TCP/UDP ports if they're easily available would allow the dropping to be more granular, but is not required to mitigate the network capacity DoS." / although I'd reword that last sentence and say something like "Using TCP/UDP ports if they're easily available would allow the dropping to be more granular, but cannot be relied on to mitigate all network capacity DoSes." / If the motivation of the DoS attacker is to exhaust network capacity, then they don't really care what type of traffic it is. Using UDP or TCP is probably convenient, but if that becomes too easy to block, they'll use something else. / OTOH, if their motivation is to DoS a particular application, then you're going to have to let some of it through because some of it is legitimate via something like the policed/shaped QoS class, as well as possibly by dropping traffic from a set of sources that appear to be DoS traffic origins rather than legitimate sources. / Perhaps wanting the TCP and UDP headers in a known location in the packet so that simple ACLs can filter them is a more specific case of being able to filter packets using a bit mask in an arbitrary specified location within the packet. I'm not aware of any routers where this is an option in ACLs (although it is ringing a very small bell), so perhaps that is what is missing from routers today (I don't know much about OpenFlow, although I'd have expected OF switches to be able to do this. However, quickly downloading and looking at the OF specs, it doesn't seem that specifying an arbitrary bit range and mask to match on is supported). Being able to do that would not only make ACLs more general purpose, but would also allow more accurate filtering on non-TCP/UDP packets too, rather than just using source and/or destination addresses for this traffic. I think that would better alleviate the desire of people to restrict transport layer protocols to only TCP/UDP. / A DoS attacker could overcome that by randomising packet contents, however that is more work for them, and I think would classify the DoS attack as a network capacity exhaustion attack, meaning that the methods such as dropping or policing/shaping high volume sources using just IP addresses would be more effective.
They're paying you to deliver any and all of their packets as well as possible, with as minimum restrictions as possible, ideally none, rather than the opposite.
Perhaps. I figure they are /actually/ paying you to provide them with good service. Sometimes that includes filtering for them, because you have bigger pipes than them... / I think sanitising traffic is a different and much harder problem to solve than mitigating a DoS attack. I think mitigating DoSes is enough of a service to provide to customers as part of their Internet access, because going any "deeper" means getting involved in what protocols and therefore what applications the customer is choosing to use and may choose to use in the future. That's a separate service. / Regards,/ Mark. W
This is why I don't think you'll ever get consensus on deprecating EHs, because I think you're only coming the position of trying to solve problem (3) I described in my last email.
/ I'd also observe that deprecating EHs, other than ESP, AH+FH, TCP and UDP, would also prevent them from being used behind the ESP EH - where you won't be able to see what is in them, so you won't be able to care what is in them. You'll also be preventing people from other transport layer protocols, such as SCTP and DCCP, that are feasible to deploy over the IPv6 Internet that couldn't be over the IPv4 network because of IPv4 NAT and other middle boxes.
thanks
Enno
cheers, Ole
-- Enno Rey
ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902
Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey
======================================================= Blog: || Conference: www.troopers.de
Twitter: @Enno_Insinuator
=======================================================
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
-- I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf
On Wed, Jun 17, 2015 at 9:41 AM, Fred Baker (fred) <fred@cisco.com> wrote:
Fernando would like to see a specification of how long the entire list of extension headers may be, in terms of "how large of a hardware buffer has to be implemented in a switch?" That has a couple of issues related to "why does one have to ask the question?" The hardware should definitely be large enough to read in the IPv6 header, the HBH Header, and the Routing Header. After that, the only host that SHOULD need to read it all in is the host itself, and the host can do its processing from RAM. There is one "gotcha" case, which is when someone uses an ACL (disallowing Fragmentation Headers, perhaps, or looking for the TCP port, meaning one has to ask whether a given packet is TCP and what its port number is). The firewall is going to have to chase down the chain of extension headers.
I agree. There is a definitely a big difference in requirements for "traditional routers" and devices which are going to implement ACLs (and other types of "multifields classifications" (as QoS based other data than IP header). IMHO it's reasonable to assume that one might need different hardware for "just routing" and enhanced QoS/ACL services (it's a case nowadays anyway). For the latter case the device has to either, as you said, chaise the whole chain down (which might be as long as the whole packet) or have a way to deal with packets it could not parse (applying a specific policy to such packets).
How big is a HBH header? How big is a routing option such as the Segment Routing option? If that's a list of 27 services, each with a 16 byte IPv6 address...
I'd like to point out that the problem is not specific to IPv6 at all. How deep is MPLS label stack? Where are TCP flags or port number in the packet (so I can match 'tcp established' or 'tcp 443')? oops, we don't know....it depends...so some linecards do not copy enough data, some (newer ones) do. We do not know (could not know) exact numbers, the requirements are changing, so does the hardware we deploy in out network. It's a kind of race ;) and as long as there is a way to predict/configure the behavior of the device when it could not fully parse the packet - I'm fine with it..
On May 17, 2015, at 12:18 PM, Enno Rey <erey@ernw.de> wrote:
Hi Silvia,
On Sun, May 17, 2015 at 06:43:11PM +0000, Silvia Hagen wrote:
Hi
I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it.
Couldn't we update RFC2460 and make this list a strict order?
good luck bringing this suggestion to IETF 6man... ;-)
I would want my firewall to notify me if the EHs in a packet do not follow the list.
actually when we tested a number of commercial firewalls in late 2013 (results here: https://www.ernw.de/download/TR14_IPv6SecSummit_AAtlasis-RISC-project_result...) it turned out that pretty much all of them follow a (from our perspective "reasonable approach", that is dropping "unusal combinations or number of ext_hdrs" by default. (which, of course, violates RFC 2460. but apparently all major vendors follow a line of reasoning similar to ours). so, in short, firewalls "are not affected".
And limiting the number of possible EHs per packet might be a good idea.
yes, that _would actually be_ a good idea.
I'm cross-posting this to v6ops as there's a related discussion going on right now. pls forgive this potential Usenet etiquette violation.
best
Enno
Silvia
-----Urspr?ngliche Nachricht----- Von: ipv6-wg [mailto:ipv6-wg-bounces@ripe.net] Im Auftrag von Benedikt Stockebrand Gesendet: Sonntag, 17. Mai 2015 18:39 An: ipv6-wg@ripe.net Betreff: Re: [ipv6-wg] Extension Headers / Impact on Security Devices
Hi Enno and list,
Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1:
When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order:
followed on the next page by
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such.
The impact here is actually at least twofold:
- It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router.
- As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered.
The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes.
Cheers,
Benedikt
-- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/
Business Grade IPv6 --- Consulting, Training, Projects
BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
-- Enno Rey
ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902
Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey
======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
-- SY, Jen Linkova aka Furry
On Wed, Jun 17, 2015 at 9:41 AM, Fred Baker (fred) <fred@cisco.com> wrote:
Fernando would like to see a specification of how long the entire list of extension headers may be, in terms of "how large of a hardware buffer has to be implemented in a switch?" That has a couple of issues related to "why does one have to ask the question?" The hardware should definitely be large enough to read in the IPv6 header, the HBH Header, and the Routing Header. After that, the only host that SHOULD need to read it all in is the host itself, and the host can do its processing from RAM. There is one "gotcha" case, which is when someone uses an ACL (disallowing Fragmentation Headers, perhaps, or looking for the TCP port, meaning one has to ask whether a given packet is TCP and what its port number is). The firewall is going to have to chase down the chain of extension headers.
You may call it a firewall. I call it a "border router" - and the difference between this and a firewall is striking. Most importantly, my border routers don't keep state for the sessions passing through them. However, I *do* expect them to handle basic stateless ACLs, including L4 info (port numbers). For IPv4 and IPv6.
I agree. There is a definitely a big difference in requirements for "traditional routers" and devices which are going to implement ACLs (and other types of "multifields classifications" (as QoS based other data than IP header). IMHO it's reasonable to assume that one might need different hardware for "just routing" and enhanced QoS/ACL services (it's a case nowadays anyway).
You may feel it is reasonable. Not everybody agrees. If we compare with IPv4: All modern routers I know of (including high speed boxes with multiple 10G and 100G ports) are able to handle stateless ACLs based on IPv4 addresses and port numbers. The boxes with multiple 10G and 100G ports process these ACLs at line rate. I don't pay extra for this functionality - possibly because a box *without* such functionality would have a limited market.
For the latter case the device has to either, as you said, chaise the whole chain down (which might be as long as the whole packet) or have a way to deal with packets it could not parse (applying a specific policy to such packets).
How big is a HBH header? How big is a routing option such as the Segment Routing option? If that's a list of 27 services, each with a 16 byte IPv6 address...
A small but possibly relevant digression here: My border routers (which happen to be Juniper routers) are configured to disallow source routing. This has been the Juniper *default* since around 2008, see http://www.juniper.net/techpubs/software/junos/junos85/swconfig85-routing/fr... I expect (but do not know with certainty) that Juniper changed this source routing default behavior based on customer feedback. Back to IPv6: I might allow "interesting" IPv6 extension headers within my own AS - because in such cases I have much more control. There is no way I'm going to allow IPv6 packets with long chains of "interesting" IPv6 header chains to pass my border routers. Either they have short enough header chains that my border routers can inspect the L4 info at line rate - or they get dropped.
I'd like to point out that the problem is not specific to IPv6 at all. How deep is MPLS label stack? Where are TCP flags or port number in the packet (so I can match 'tcp established' or 'tcp 443')? oops, we don't know....it depends...so some linecards do not copy enough data, some (newer ones) do.
The MPLS label stack is normally used with a single provider - thus the provider is in control. I agree that the IPv4 packet may have options, making it variable length. However the length is still limited by the IHL field, which has a max value of 15 (60 bytes). Steinar Haug, AS 2116
On Wed, Jun 17, 2015 at 2:02 PM, <sthaug@nethelp.no> wrote:
IMHO it's reasonable to assume that one might need different hardware for "just routing" and enhanced QoS/ACL services (it's a case nowadays anyway).
You may feel it is reasonable. Not everybody agrees. If we compare with IPv4: All modern routers I know of (including high speed boxes with multiple 10G and 100G ports) are able to handle stateless ACLs based on IPv4 addresses and port numbers. The boxes with multiple 10G and 100G ports process these ACLs at line rate. I don't pay extra for this functionality - possibly because a box *without* such functionality would have a limited market.
[skip]
I agree that the IPv4 packet may have options, making it variable length. However the length is still limited by the IHL field, which has a max value of 15 (60 bytes).
I'm glad you mentioned 60 bytes ;) Because there are a lot of reasonably modern hardware around which copies 64 bytes on-chip. Which means if you happen such hardware in your network and your stateless ACL have 'match tcp flags' rules, you might get quite unexpected results processing packets with 60 bytes IPv4 header....So, while it might be perfectly fine to have such cars in the core, I'd expect people not to install then at the border routers which are supposed to perform enhanced ACL services. It was my point. So we all agree that 'variable length is OK as long as our hardware can look deep enough'? And what people are complaining about is exact number? Which we do not know yet for IPv6 EHs? -- SY, Jen Linkova aka Furry
So we all agree that 'variable length is OK as long as our hardware can look deep enough'? And what people are complaining about is exact number? Which we do not know yet for IPv6 EHs?
Agreed, variable length *by itself* is not the problem. I see *large* variable length headers, in combination with complex parsing rules, as the problem. Steinar Haug, AS 2116
Hi, On Wed, Jun 17, 2015 at 03:27:50PM +0200, sthaug@nethelp.no wrote:
So we all agree that 'variable length is OK as long as our hardware can look deep enough'? And what people are complaining about is exact number? Which we do not know yet for IPv6 EHs?
Agreed, variable length *by itself* is not the problem.
I see *large* variable length headers, in combination with complex parsing rules, as the problem.
(*large* variable headers) exactly, plus fragmentation and "ambiguities" wrt fragmentable vs. unfragmentable part and how headers point to the next one once there's a cut between them due to fragmentation. the, what we consider, "problem space" is much larger, unfortunately. thanks Enno
Steinar Haug, AS 2116
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
-- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
On Wed, Jun 17, 2015 at 3:33 PM, Enno Rey <erey@ernw.de> wrote:
I see *large* variable length headers, in combination with complex parsing rules, as the problem.
(*large* variable headers) exactly, plus fragmentation
Fragmentation is not specific to IPv6, is it? And, as the fragment header itself is just 8 bytes (AFAIR, too lazy to check ;)) - I'd not classify it as *large* variable header.
and "ambiguities" wrt fragmentable vs. unfragmentable part and how headers point to the next one once there's a cut between them due to fragmentation. the, what we consider, "problem space" is much larger, unfortunately.
Has not RFC7112 addressed this particular problem? -- SY, Jen Linkova aka Furry
Hi, On Wed, Jun 17, 2015 at 03:41:01PM +0200, Jen Linkova wrote:
On Wed, Jun 17, 2015 at 3:33 PM, Enno Rey <erey@ernw.de> wrote:
I see *large* variable length headers, in combination with complex parsing rules, as the problem.
(*large* variable headers) exactly, plus fragmentation
Fragmentation is not specific to IPv6, is it?
right [even though I'm in the camp of "deprecate fragmentation as a whole" ;-), but let's not open this can of worms here]. BUT: fragmentation becomes a (then somewhat IPv6 specific) huge problem space once you have a datagram containing, say, 20 extension headers of partially variable lenght & pretty much arbitrary order, split into, say, 50 fragments. [again, see https://www.ernw.de/download/eu-14-Atlasis-Rey-Schaefer-briefings-Evasion-of...]. Such a datagram would be fully valid as of today/RFC2460. Yes, we're aware of RFC7112. It's just: no OS we know and no devices we're aware of (feel free to provide pointers) implement RFC 7112 as of today. but many attack tools implement the techniques mentioned above. Which is why quite some operators (in particular, but not only) from enterprise and managed service provider/cloud space drop all EHs except, maybe, AH+ESP. thanks Enno And, as the fragment
header itself is just 8 bytes (AFAIR, too lazy to check ;)) - I'd not classify it as *large* variable header.
and "ambiguities" wrt fragmentable vs. unfragmentable part and how headers point to the next one once there's a cut between them due to fragmentation. the, what we consider, "problem space" is much larger, unfortunately.
Has not RFC7112 addressed this particular problem?
-- SY, Jen Linkova aka Furry
-- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
Catching up on a few points in this thread that went crazy while I was sleeping... On 18/06/2015 02:00, Enno Rey wrote: ...
Yes, we're aware of RFC7112. It's just: no OS we know and no devices we're aware of (feel free to provide pointers) implement RFC 7112 as of today.
No, it's too new. But I suggest that it gives you license to drop packets with fragmented header chains, and tell anyone who complains that they don't conform to the IPv6 standard.
but many attack tools implement the techniques mentioned above. Which is why quite some operators (in particular, but not only) from enterprise and managed service provider/cloud space drop all EHs except, maybe, AH+ESP.
Whereas dropping *all* EHs breaks the IPv6 standard. On 18/06/2015 03:11, Ca By wrote:
For the folks looking for extension header innovation, would you be willing to work on IP version X instead of IPv6? Or perhaps you can use the Class E IPv4 space for your innovation?
Now that's a polemic, not an argument. But since you ask: of course not.
Serious. IPv6 is not a place for innovation at the Network / Internet layer.
EHs as an extension mechanism are *not* innovation. They've been in the design for 20 years. I'm actually with Fred on this: it's time for the hardware designers to step up. With RFC 7112, we've told them that the maximum packet size they need to parse is 1280 (after removing tunneling overhead). Brian
Yes, we're aware of RFC7112. It's just: no OS we know and no devices we're aware of (feel free to provide pointers) implement RFC 7112 as of today.
No, it's too new. But I suggest that it gives you license to drop packets with fragmented header chains, and tell anyone who complains that they don't conform to the IPv6 standard.
It may be relevant to ask for RFC 7112 support next time we're doing an equipment RFQ (in a few years).
but many attack tools implement the techniques mentioned above. Which is why quite some operators (in particular, but not only) from enterprise and managed service provider/cloud space drop all EHs except, maybe, AH+ESP.
Whereas dropping *all* EHs breaks the IPv6 standard.
Obviously. But until RFC 7112 support is available, I believe we will see a significant amount of breakage for IPv6 extension headers - and header chains will be limited to significantly less than 1280 bytes.
EHs as an extension mechanism are *not* innovation. They've been in the design for 20 years. I'm actually with Fred on this: it's time for the hardware designers to step up. With RFC 7112, we've told them that the maximum packet size they need to parse is 1280 (after removing tunneling overhead).
IPv6 extension header processing is sufficiently complex that I'm not at all sure that "time for the hardware designers to step up" will be enough. My prediction is that we'll see security alerts from hardware manufacturers for many years to come, due to the complexity. Steinar Haug, AS 2116
I am aware of one implementation that is almost RFC 7112 compliant. It drops the packet, but fails to send an ICMP message. Ron
Yes, we're aware of RFC7112. It's just: no OS we know and no devices we're aware of (feel free to provide pointers) implement RFC 7112 as of today.
* sthaug@nethelp.no
Back to IPv6: I might allow "interesting" IPv6 extension headers within my own AS - because in such cases I have much more control. There is no way I'm going to allow IPv6 packets with long chains of "interesting" IPv6 header chains to pass my border routers. Either they have short enough header chains that my border routers can inspect the L4 info at line rate - or they get dropped.
Hi Steinar, I wouldn't react to the above if you were operating an enterprise network, but considering you're an ISP and transit provider, I find the above rather surprising (and I do not mean that in a good way). First, your customers might have a perfectly valid reason to send or receive IPv6 headers with IPv6 extension header chains you apparantly will drop at your border. FWIW, if I found out that my upstream arbitrarily dropped packets because they found them "interesting", breaking my applications in the process, I would not remain a customer of theirs for long. Second, the packets might be encrypted using ESP. In that case, you have absolutely no way of knowing if the extension header chain is long enough to be "interesting enough to drop", or if the ESP header is the only extension header there is ("short enough to forward"). What do you do then? Third, your border routers obviously cannot inspect the L4 info in an ESP-encrypted packet at all, line rate or not. Does that mean you drop all ESP packets at your AS borders? I really hope not. Tore
Hi Tore, On Wed, Jun 17, 2015 at 08:18:09PM +0200, Tore Anderson wrote:
* sthaug@nethelp.no
Back to IPv6: I might allow "interesting" IPv6 extension headers within my own AS - because in such cases I have much more control. There is no way I'm going to allow IPv6 packets with long chains of "interesting" IPv6 header chains to pass my border routers. Either they have short enough header chains that my border routers can inspect the L4 info at line rate - or they get dropped.
Hi Steinar,
I wouldn't react to the above if you were operating an enterprise network, but considering you're an ISP and transit provider, I find the above rather surprising (and I do not mean that in a good way).
First, your customers might have a perfectly valid reason to send or receive IPv6 headers with IPv6 extension header chains you apparantly will drop at your border. FWIW, if I found out that my upstream arbitrarily dropped packets because they found them "interesting", breaking my applications
that brings us directly to the core of the debate: break "exactly which application?". there's no single application/service using EHs other than AH/ESP and, maybe in a few corner cases, FH today and I doubt we'll see some tomorrow (given developing such a thing is heavily de-incentivized by the growing number of operators mostly dropping EHs). Taking into account that stateless ACLs of all router vendors we tested (results tb published soon) can be avoided/evaded by adding ~5 extension headers to datagrams I fully understand any operator who does not want SSH on its devices to be reachable from the Internet (over v6 with extension headers) and hence acts in a way similar to the one Steinar described. I doubt Steinar loses many customers (due to "application breakage") by taking that path. In contrary I expect many of his customers valueing the increased level of device & network availability gained by eliminating an entire class of attacks. best Enno -- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
Hi Enno, * Enno Rey
On Wed, Jun 17, 2015 at 08:18:09PM +0200, Tore Anderson wrote:
First, your customers might have a perfectly valid reason to send or receive IPv6 headers with IPv6 extension header chains you apparantly will drop at your border. FWIW, if I found out that my upstream arbitrarily dropped packets because they found them "interesting", breaking my applications
that brings us directly to the core of the debate: break "exactly which application?"
Well, ESP at least. And, by extension, any protocol that might be carried inside ESP, so pretty much all of them.
Taking into account that stateless ACLs of all router vendors we tested (results tb published soon) can be avoided/evaded by adding ~5 extension headers to datagrams I fully understand any operator who does not want SSH on its devices to be reachable from the Internet (over v6 with extension headers) and hence acts in a way similar to the one Steinar described.
There is a big difference between an operator dropping all packets with EHs that are destined for *his own devices/routers* (I have no problem with that - your devices, your rules), and an operator that drops *transit* traffic destined for his customers because his routers cannot understand/parse/filter its L4/EH payload. In my opinion, an ISP/IP transit network shouldn't even attempt to parse the L4/EH payload in customer traffic (except if the customer asks for it of course), it should just deliver the packets.
I doubt Steinar loses many customers (due to "application breakage") by taking that path. In contrary I expect many of his customers valueing the increased level of device & network availability gained by eliminating an entire class of attacks.
The first operator I mentioned above won't lose any customers because his filtering activities doesn't impact customer traffic. The second operator would lose my business, at least. And probably others' too, as business customers might want their site2site IPSEC tunnels to work, residental customers might want their Xbox One online gaming to work, and so on. Tore
Taking into account that stateless ACLs of all router vendors we tested (results tb published soon) can be avoided/evaded by adding ~5 extension headers to datagrams I fully understand any operator who does not want SSH on its devices to be reachable from the Internet (over v6 with extension headers) and hence acts in a way similar to the one Steinar described.
There is a big difference between an operator dropping all packets with EHs that are destined for *his own devices/routers* (I have no problem with that - your devices, your rules), and an operator that drops *transit* traffic destined for his customers because his routers cannot understand/parse/filter its L4/EH payload.
I certainly appreciate this distinction. However, problems arise in practice when customers *ask* for various types of protection - which are most sensibly implemented on border routers. Just to be clear - we don't drop IPsec traffic today (neither IPv6 nor IPv4).
In my opinion, an ISP/IP transit network shouldn't even attempt to parse the L4/EH payload in customer traffic (except if the customer asks for it of course), it should just deliver the packets.
The problem is - the customer *does* ask for it, in many cases.
I doubt Steinar loses many customers (due to "application breakage") by taking that path. In contrary I expect many of his customers valueing the increased level of device & network availability gained by eliminating an entire class of attacks.
The first operator I mentioned above won't lose any customers because his filtering activities doesn't impact customer traffic. The second operator would lose my business, at least. And probably others' too, as business customers might want their site2site IPSEC tunnels to work, residental customers might want their Xbox One online gaming to work, and so on.
I agree - IPSEC tunnels and Xbox One online gaming need to work. That doesn't mean *all* extension header combinations should be expected to work - and it appears that they often don't. I expect we'll see much more of these discussions in the coming years, as IPv6 traffic grows. And I certainly also expect significant size IPv6 DDoS attacks - at least some of which will probably be based on extension header manipulation. Steinar Haug, AS 2116
Let me chime in ;-) On this topic, I am suffering of schizophrenia with a double personality... - security person: I hate when there are too many options and sub-options and my usual recommendation is to use white list approach at the destination (being plain layer-4 ACL or more content filtering) and at the source (let's try to cut down covert channels): block unused ports, block unused protocols (remember IP protocol 41), block unused extension headers and drop everything which is not a normal IP packet (from address spoofing to invalid EH chain). If your firewall cannot implement this policy, then it is time to change or to use a combination of FW & IPS or whatever. - network person: as I will live with IPv6 for the rest of my life, then I want a way to extend it and I need any packet to travel to my source to my destination. Any IPv6 packet should be able to traverse the Internet/private network as long as its layer-3 header is valid (filtering/dropping can be done exceptionally for good operational reason). Did we remember the IPv4 teardrop attack? It is blocked by default by most routers for years ;-) Bugs will plague us for ever... And make our life more complex than the above description of course ;-) -éric On 17/05/15 20:43, "Silvia Hagen" <silvia.hagen@sunny.ch> wrote:
Hi
I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it.
Couldn't we update RFC2460 and make this list a strict order?
I would want my firewall to notify me if the EHs in a packet do not follow the list. And limiting the number of possible EHs per packet might be a good idea.
Silvia
-----Ursprüngliche Nachricht----- Von: ipv6-wg [mailto:ipv6-wg-bounces@ripe.net] Im Auftrag von Benedikt Stockebrand Gesendet: Sonntag, 17. Mai 2015 18:39 An: ipv6-wg@ripe.net Betreff: Re: [ipv6-wg] Extension Headers / Impact on Security Devices
Hi Enno and list,
Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1:
When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order:
followed on the next page by
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such.
The impact here is actually at least twofold:
- It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router.
- As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered.
The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes.
Cheers,
Benedikt
-- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/
Business Grade IPv6 --- Consulting, Training, Projects
BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
Eric, All, On Mon, May 18, 2015 at 06:07:45AM +0000, Eric Vyncke (evyncke) wrote:
Let me chime in ;-)
On this topic, I am suffering of schizophrenia with a double personality...
- security person: I hate when there are too many options and sub-options and my usual recommendation is to use white list approach at the destination (being plain layer-4 ACL or more content filtering) and at the source (let's try to cut down covert channels): block unused ports, block unused protocols (remember IP protocol 41), block unused extension headers and drop everything which is not a normal IP packet (from address spoofing to invalid EH chain).
the problem here is the definition of "normal IP packet" as of RFC2460. To illustrate this I just quote from today's Cisco advisory (Cisco IOS XR Software Crafted IPv6 Packet Denial of Service Vulnerability) on packets potentially crashing CRS-3 line cards: "The vulnerability is due to incorrect processing of an IPv6 packet carrying IPv6 extension headers that are valid but unlikely to be seen during normal operation. An attacker could exploit this vulnerability by sending such an IPv6 packet to an affected device that is configured to process IPv6 traffic. An exploit could allow the attacker to cause a reload of the line card, resulting in a DoS condition." two question come to mind here: - is a "valid but unlikely" extension header chain "normal"? - what ("combination of FW & IPS or whatever") would you put in front of a CRS? my (sad) expectation is that we'll see much more of these (types of) issues in the future. given the current level of freedom that the RFC2460 leaves (see also discussion/picture in http://www.insinuator.net/2015/06/is-ipv6-more-secure-than-ipv4-or-less/) "properly parsing an IPv6 packet, let alone in wire speed" seems a pretty much unsolvable task to me. That said I still fail to see any useful extension header besides AH&ESP (not even FH) to be transported in the Internet. best Enno If your firewall cannot implement this policy, then
it is time to change or to use a combination of FW & IPS or whatever.
- network person: as I will live with IPv6 for the rest of my life, then I want a way to extend it and I need any packet to travel to my source to my destination. Any IPv6 packet should be able to traverse the Internet/private network as long as its layer-3 header is valid (filtering/dropping can be done exceptionally for good operational reason). Did we remember the IPv4 teardrop attack? It is blocked by default by most routers for years ;-)
Bugs will plague us for ever... And make our life more complex than the above description of course ;-)
-??ric
On 17/05/15 20:43, "Silvia Hagen" <silvia.hagen@sunny.ch> wrote:
Hi
I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it.
Couldn't we update RFC2460 and make this list a strict order?
I would want my firewall to notify me if the EHs in a packet do not follow the list. And limiting the number of possible EHs per packet might be a good idea.
Silvia
-----Urspr??ngliche Nachricht----- Von: ipv6-wg [mailto:ipv6-wg-bounces@ripe.net] Im Auftrag von Benedikt Stockebrand Gesendet: Sonntag, 17. Mai 2015 18:39 An: ipv6-wg@ripe.net Betreff: Re: [ipv6-wg] Extension Headers / Impact on Security Devices
Hi Enno and list,
Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1:
When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order:
followed on the next page by
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such.
The impact here is actually at least twofold:
- It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router.
- As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered.
The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes.
Cheers,
Benedikt
-- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/
Business Grade IPv6 --- Consulting, Training, Projects
BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
-- Enno Rey ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902 Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey ======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
Enno, As you probably know, Cisco handles PSIRT issues with care, so, I have no clue on this specific advisory. My own (educated) guess is that the packet causing a LC reload is fully RFC 2460 compliant but unusual, so, probably (a guess again) a combination of extension headers triggering a bug. In this specific case, I would not blame IPv6 or RFC 2460 but rather my own company (even if the affected versions were quite old as new IOS-XR versions appear to be immune to this bug). And I appreciate the humor in your rhetorical question about which FW could protect a CRS... Air gap probably :-) More generally, regarding the 'parsing of the header' and performance, my understanding (and I am NOT a HW engineer) is that there are multiple ways to parse the header chain: - purely software in low-end devices, should have a marginal performance impact - specific ASIC in middle-end devices (read high end enterprise), probably some limitations and heavy performance impact (but again within one organization, so, easier to fix the root cause) - a lot of specialized network processors in high-end devices (SP gears), which is a similar case as the 'low-end device', performance impact but marginal as parsing the chain of headers is not that hard after all _when_ coded correctly As the extension headers are specific to IPv6, this is an area when we will all learn (and have learned already) a lot of things... I respectfully disagree with your last sentence. Of course, destination AS/host SHOULD only accept useful, known and legit ext headers; but, why would an ISP filter on ext headers if they do not harm their own infra or a customer infra? Do you envision an Internet where only TCP/443 and UDP/43 would be accepted? Last point, the Internet will have to use IPv6 for 10 or 20 years at least. If the IETF/ISP block all new _options_ in _existing_ extension headers, then we will be stuck in 1997 for network innovation. While I do not like taking security risks, I even fear more the lack of innovation. Let's talk on Thursday at Silvia's IPv6 Conference in Zurich ;-) -éric On 11/06/15 11:58, "Enno Rey" <erey@ernw.de> wrote:
the problem here is the definition of "normal IP packet" as of RFC2460. To illustrate this I just quote from today's Cisco advisory (Cisco IOS XR Software Crafted IPv6 Packet Denial of Service Vulnerability) on packets potentially crashing CRS-3 line cards:
"The vulnerability is due to incorrect processing of an IPv6 packet carrying IPv6 extension headers that are valid but unlikely to be seen during normal operation. An attacker could exploit this vulnerability by sending such an IPv6 packet to an affected device that is configured to process IPv6 traffic. An exploit could allow the attacker to cause a reload of the line card, resulting in a DoS condition."
two question come to mind here:
- is a "valid but unlikely" extension header chain "normal"? - what ("combination of FW & IPS or whatever") would you put in front of a CRS?
my (sad) expectation is that we'll see much more of these (types of) issues in the future. given the current level of freedom that the RFC2460 leaves (see also discussion/picture in http://www.insinuator.net/2015/06/is-ipv6-more-secure-than-ipv4-or-less/) "properly parsing an IPv6 packet, let alone in wire speed" seems a pretty much unsolvable task to me.
That said I still fail to see any useful extension header besides AH&ESP (not even FH) to be transported in the Internet.
best
Enno
If your firewall cannot implement this policy, then
it is time to change or to use a combination of FW & IPS or whatever.
- network person: as I will live with IPv6 for the rest of my life, then I want a way to extend it and I need any packet to travel to my source to my destination. Any IPv6 packet should be able to traverse the Internet/private network as long as its layer-3 header is valid (filtering/dropping can be done exceptionally for good operational reason). Did we remember the IPv4 teardrop attack? It is blocked by default by most routers for years ;-)
Bugs will plague us for ever... And make our life more complex than the above description of course ;-)
-??ric
On 17/05/15 20:43, "Silvia Hagen" <silvia.hagen@sunny.ch> wrote:
Hi
I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it.
Couldn't we update RFC2460 and make this list a strict order?
I would want my firewall to notify me if the EHs in a packet do not follow the list. And limiting the number of possible EHs per packet might be a good idea.
Silvia
-----Urspr??ngliche Nachricht----- Von: ipv6-wg [mailto:ipv6-wg-bounces@ripe.net] Im Auftrag von Benedikt Stockebrand Gesendet: Sonntag, 17. Mai 2015 18:39 An: ipv6-wg@ripe.net Betreff: Re: [ipv6-wg] Extension Headers / Impact on Security Devices
Hi Enno and list,
Enno Rey <erey@ernw.de> writes:
hope everybody had a great #RIPE70 meeting. We did! Many thanks to the organizers and chairs!
and thanks to the actual speakers as well the speakers we had to turn down due to time constraints, too:-)
If the chairs consider this appropriate we will happily give a presentation on this stuff in Bucharest or at another occasion.
Sounds good to me!
- looking at the "liberty" RFC2460 provides as for ext_hdrs (wrt to their number, order[...]
Actually, as far as I'm concerned that's the real core of the problem. Or more specifically, the first two lines of RFC 2460, section 4.1:
When more than one extension header is used in the same packet, it is recommended that those headers appear in the following order:
followed on the next page by
Each extension header should occur at most once, except for the Destination Options header which should occur at most twice (once before a Routing header and once before the upper-layer header).
Note in particular that these are not even RFC 2119 "SHOULD" or "RECOMMENDED" and such.
The impact here is actually at least twofold:
- It is impossible to implement this as a simple pipeline architecture in hardware; at least for cases deviating from the "recommendations" above this effectively becomes either excessively complex to implement in hardware or an invitation to DoS when implemented in software on an otherwise hardware router.
- As I understand it, at least some of the issues you have found are effectively based on violating the second paragraph quoted, making it impossible to come up with a lower bound on how long the header chain can actually get and therefore leading to the fragmentation related attacks and similar you have discovered.
The original idea was that the extension headers are processed strictly in the order they occur, so one question to ask is if there is any valid reason to violate these "recommendations" for other than malicious purposes.
Cheers,
Benedikt
-- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/
Business Grade IPv6 --- Consulting, Training, Projects
BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
-- Enno Rey
ERNW GmbH - Carl-Bosch-Str. 4 - 69115 Heidelberg - www.ernw.de Tel. +49 6221 480390 - Fax 6221 419008 - Cell +49 173 6745902
Handelsregister Mannheim: HRB 337135 Geschaeftsfuehrer: Enno Rey
======================================================= Blog: www.insinuator.net || Conference: www.troopers.de Twitter: @Enno_Insinuator =======================================================
_______________________________________________ v6ops mailing list v6ops@ietf.org https://www.ietf.org/mailman/listinfo/v6ops
Hi, On Mon, Jun 15, 2015 at 10:15:36PM +0000, Eric Vyncke (evyncke) wrote:
Let's talk on Thursday at Silvia's IPv6 Conference in Zurich ;-)
Shall I bait you on the panel with that...? ;-) Gert Doering -- NetMaster -- have you enabled IPv6 on something today...? SpaceNet AG Vorstand: Sebastian v. Bomhard Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann D-80807 Muenchen HRB: 136055 (AG Muenchen) Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
On Thu, Jun 11, 2015 at 6:58 PM, Enno Rey <erey@ernw.de> wrote:
the problem here is the definition of "normal IP packet" as of RFC2460.
The problem here is what one might mean by "normal" (from Oxford dictionary): 1) conforming to a standard; 2) usual, typical, or expected;
To illustrate this I just quote from today's Cisco advisory (Cisco IOS XR Software Crafted IPv6 Packet Denial of Service Vulnerability) on packets potentially crashing CRS-3 line cards:
"The vulnerability is due to incorrect processing of an IPv6 packet carrying IPv6 extension headers that are valid but unlikely to be seen during normal operation. An attacker could exploit this vulnerability by sending such an IPv6 packet to an affected device that is configured to process IPv6 traffic. An exploit could allow the attacker to cause a reload of the line card, resulting in a DoS condition."
two question come to mind here:
- is a "valid but unlikely" extension header chain "normal"?
It is "normal" if you define "normal" as "conforming to a standard", but it's not "normal" if you define "normal" as "usual/typical".
- what ("combination of FW & IPS or whatever") would you put in front of a CRS?
Nothing. There is smth to put *on* CRS however - the image which contains the fix... I do not think we should blame a protocol for software bugs. I've seen router crashes caused by ICMP and BGP packets. Shall we go ahead and start discussing deprecation of the two above mentioned protocols? ;) Especially taking into account than some people like filtering ICMP on the edge of their networks? ;)
my (sad) expectation is that we'll see much more of these (types of) issues in the future. given the current level of freedom that the RFC2460 leaves (see also discussion/picture in http://www.insinuator.net/2015/06/is-ipv6-more-secure-than-ipv4-or-less/) "properly parsing an IPv6 packet, let alone in wire speed" seems a pretty much unsolvable task to me.
(shameless plug) a group of enthusiasts have just submitted a new version of document which discusses exactly this problem: https://tools.ietf.org/html/draft-wkumari-long-headers-03 Comments are appreciated... -- SY, Jen Linkova aka Furry
On 6/16/2015 12:02 PM, Jen Linkova wrote:
On Thu, Jun 11, 2015 at 6:58 PM, Enno Rey <erey@ernw.de> wrote:
the problem here is the definition of "normal IP packet" as of RFC2460.
The problem here is what one might mean by "normal" (from Oxford dictionary): 1) conforming to a standard; 2) usual, typical, or expected;
That's the trouble with extensions. You can't expect them until they're developed AND deployed, so initially they're never "expected" in terms of actual traffic. Except that we SHOULD expect *everything* that's in a spec. Joe
On 17/06/2015 07:02, Jen Linkova wrote: ...
(shameless plug) a group of enthusiasts have just submitted a new version of document which discusses exactly this problem:
https://tools.ietf.org/html/draft-wkumari-long-headers-03
Comments are appreciated...
In REQ-2 on HbH headers, you say:
The forwarder MUST process each option as specified in Section 4.2 of [RFC2460].
That aspect of RFC 2460 was fundamentally changed by RFC 7045. And of course this is the issue addressed by draft-baker-6man-hbh-header-handling. Personally I still think RFC 7045 is the most realistic on this point, but Fred would like things to get better ;-). I'm sure other things in the long-headers draft need revising as a result of RFC 7045, since its whole topic is the handling of extension headers ("This document updates RFC 2460 to clarify how intermediate nodes should deal with such extension headers and with any that are defined in the future.") Y'all also need to take account of RFC 7112, which forbids fragmented header chains. Brian
On Jun 16, 2015, at 6:24 PM, Brian E Carpenter <brian.e.carpenter@gmail.com> wrote:
Personally I still think RFC 7045 is the most realistic on this point, but Fred would like things to get better ;-).
And I haven't finished with Dennis Ferguson's comment. Bottom line, if one accepts the present status quo as the state forever, then we should stop with RFC 7045, and (with Fernando) agree to deprecate all extension headers. I'd like to not do that, and the only way I see to not do that is to not accept the status quo.
On 06/17/2015 01:45 AM, Fred Baker (fred) wrote:
On Jun 16, 2015, at 6:24 PM, Brian E Carpenter <brian.e.carpenter@gmail.com> wrote:
Personally I still think RFC 7045 is the most realistic on this point, but Fred would like things to get better ;-).
And I haven't finished with Dennis Ferguson's comment.
Bottom line, if one accepts the present status quo as the state forever, then we should stop with RFC 7045, and (with Fernando) agree to deprecate all extension headers. I'd like to not do that, and the only way I see to not do that is to not accept the status quo.
Not sure if that's simply a matter of an error in punctuation (or in my interpretation)... but for the record, I'm not arguing in favor of deprecating IPv6 EHs. Actually, I've worked to figured out what's the status quo, and working on stuff that may help to change that (e.g., RFC7112). Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@si6networks.com PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
On Wed, Jun 17, 2015 at 3:24 AM, Brian E Carpenter <brian.e.carpenter@gmail.com> wrote:
On 17/06/2015 07:02, Jen Linkova wrote:
https://tools.ietf.org/html/draft-wkumari-long-headers-03
Comments are appreciated...
In REQ-2 on HbH headers, you say:
The forwarder MUST process each option as specified in Section 4.2 of [RFC2460].
That aspect of RFC 2460 was fundamentally changed by RFC 7045.
Oh, very good point. Thanks for pointing this out. Looks like it does cover our REQ2 (but requires different behavior). So, Section 2.1 RFC7045 says about *all* EHs: "If a forwarding node discards a packet containing a standard IPv6 extension header, it MUST be the result of a configurable policy and not just the result of a failure to recognise such a header. " and then, in Section 2.1 for HbH: "The IPv6 Hop-by-Hop Options header SHOULD be processed by intermediate forwarding nodes as described in [RFC2460]. However, it is to be expected that high-performance routers will either ignore it or assign packets containing it to a slow processing path. ". I read this as 'if a router does not recognize/parse the HbH header, it MUST not discard it unless there is an explicit policy configured. It may just ignore it or send for "special processing" via slow path. So it assumes that it is always safe to ignore HbH header (as if all options in the header had highest-order two bits set to '00') while our draft proposed quite different approach ("drop and send ICMP back"). Ignoring HbH header seems to be reasonable and safe if we are sure that every single existing or to be developed HbH option can be safely ignored (or options which could be ignored must be in the very beginning of the header...). In the light of RFC7045 it looks to me that one possible approach for REQ2 might be: - if a forwarder can not parse the HbH header and there is no explicitly configurable policy, it SHOULD either: -- if the forwarder can not parse any of options or if all parsed options have highest-order two bits set to '00') - ignore the header; -- if the forwarder was able to parse some of options and at least one of the options has highest-order two bits set to smth except '00' - discard the packet and send ICMPv6 message if Section 4.2 of RFC 2460 requires so (sending ICMPv6 MAY be subject to a configurable policy) or send it to slow path for full header processing (subject to a configurable policy). How does that sound?
I'm sure other things in the long-headers draft need revising as a result of RFC 7045, since its whole topic is the handling of extension headers ("This document updates RFC 2460 to clarify how intermediate nodes should deal with such extension headers and with any that are defined in the future.")
Yes, we'll revise it, thanks for the comment.
Y'all also need to take account of RFC 7112, which forbids fragmented header chains.
We do mention 7112 but I agree, we should explicitly mention that header chain is limited by a packet size...Will update the doc. -- SY, Jen Linkova aka Furry
On 18/06/2015 04:54, Jen Linkova wrote:
On Wed, Jun 17, 2015 at 3:24 AM, Brian E Carpenter <brian.e.carpenter@gmail.com> wrote:
On 17/06/2015 07:02, Jen Linkova wrote:
https://tools.ietf.org/html/draft-wkumari-long-headers-03
Comments are appreciated...
In REQ-2 on HbH headers, you say:
The forwarder MUST process each option as specified in Section 4.2 of [RFC2460].
That aspect of RFC 2460 was fundamentally changed by RFC 7045.
Oh, very good point. Thanks for pointing this out. Looks like it does cover our REQ2 (but requires different behavior).
So, Section 2.1 RFC7045 says about *all* EHs:
"If a forwarding node discards a packet containing a standard IPv6 extension header, it MUST be the result of a configurable policy and not just the result of a failure to recognise such a header. "
and then, in Section 2.1 for HbH: "The IPv6 Hop-by-Hop Options header SHOULD be processed by intermediate forwarding nodes as described in [RFC2460]. However, it is to be expected that high-performance routers will either ignore it or assign packets containing it to a slow processing path. ".
I read this as 'if a router does not recognize/parse the HbH header, it MUST not discard it unless there is an explicit policy configured. It may just ignore it or send for "special processing" via slow path.
So it assumes that it is always safe to ignore HbH header (as if all options in the header had highest-order two bits set to '00') while our draft proposed quite different approach ("drop and send ICMP back").
Right. And we can have that discussion, of course, but IMNSHO a firewall should either let stuff through or drop it for a congigured reason; dropping it because the product development manager made a lazy choice is not OK.
Ignoring HbH header seems to be reasonable and safe if we are sure that every single existing or to be developed HbH option can be safely ignored (or options which could be ignored must be in the very beginning of the header...).
Well, we can't know today what sort of crazy option might be invented in future; so the thinking behind draft-ietf-opsec-ipv6-eh-filtering seems right to me. Maybe that's a draft you need to be tracking. To be honest what I was thinking when we added discussion of HbH to RFC 7045 was not so much fixing the slow routers, but warning the community that "hop-by-hop" does not mean every hop. Fred would like to fix that with his 6man draft, and I don't mean to attack that idea.
In the light of RFC7045 it looks to me that one possible approach for REQ2 might be: - if a forwarder can not parse the HbH header and there is no explicitly configurable policy,
AND if the forwarder is trying to be a firewall or otherwise wants to interfere. If not, just forward the packet, already!
it SHOULD either: -- if the forwarder can not parse any of options or if all parsed options have highest-order two bits set to '00') - ignore the header; -- if the forwarder was able to parse some of options and at least one of the options has highest-order two bits set to smth except '00' - discard the packet and send ICMPv6 message if Section 4.2 of RFC 2460 requires so (sending ICMPv6 MAY be subject to a configurable policy) or send it to slow path for full header processing (subject to a configurable policy).
How does that sound?
As I said: if the forwarder insists on doing something, OK. But we have to think for a typical use case: does it matter if a forwarder simply ignores the HbH header? For example, for the IntServ/RSVP case, it just makes that router transparent to RSVP. That doesn't break RSVP; discarding the packet does. That was the thinking behind RFC 7045. Brian
I'm sure other things in the long-headers draft need revising as a result of RFC 7045, since its whole topic is the handling of extension headers ("This document updates RFC 2460 to clarify how intermediate nodes should deal with such extension headers and with any that are defined in the future.")
Yes, we'll revise it, thanks for the comment.
Y'all also need to take account of RFC 7112, which forbids fragmented header chains.
We do mention 7112 but I agree, we should explicitly mention that header chain is limited by a packet size...Will update the doc.
Hi Silvia and list, Weekend:-) Anyway: Silvia Hagen <silvia.hagen@sunny.ch> writes:
I keep stumbling about that "recommendational wording" in RFC 2460 everytime I teach it.
So do I. The problem underneath however is anything but trivial.
Couldn't we update RFC2460 and make this list a strict order?
As I understand it the way extension headers work was effectively a response to the severe limitations of header options in IPv4 (like 60 octets max, not even enough to do a proper record route option). The reason for the rather relaxed wording here is in all likelyhood that people at that time didn't want to impose any rules cast in stone which would later on turn out to be another similar limitation; for example, if a subsequently defined extension header was specified, it would be rather difficult to retrofit. However, with the experience we have with extension headers so far I guess you're right, it is likely time to investigate the possibility to make the ordering and such mandatory. That will allow for much better hardware implementations, if nothing else.
I would want my firewall to notify me if the EHs in a packet do not follow the list. And limiting the number of possible EHs per packet might be a good idea.
Right, but there are a number of differences between a firewall and a high end router, so this should really be investigated from a wide range of perspectives, from microcontroller based embedded devices to high end routers, and from consumer-managed home environments to high security environments. This is quite likely a pretty big job. Cheers, Benedikt -- Benedikt Stockebrand, Stepladder IT Training+Consulting Dipl.-Inform. http://www.stepladder-it.com/ Business Grade IPv6 --- Consulting, Training, Projects BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
participants (16)
-
Benedikt Stockebrand
-
Brian E Carpenter
-
Enno Rey
-
Eric Vyncke (evyncke)
-
Fernando Gont
-
Fred Baker (fred)
-
Gert Doering
-
Jen Linkova
-
Joe Touch
-
Mark ZZZ Smith
-
Ole Troan
-
Ronald Bonica
-
Silvia Hagen
-
sthaug@nethelp.no
-
Tore Anderson
-
Warren Kumari