Router MAC-Addresses @ IXs
Dear Exchange Point members and operators, I have a small inquiry today: could you please have a look at the low-order bit distribution of (Cisco) router MAC addresses at your exchange point(s) and tell me the result ? Background: port-grouping algorithms (like Cisco FastEtherChannel on Cat5xxx) on switches tend to use low-order bits of MAC addresses (XOR of source/dest, or similar) for load-sharing purposes. Now it turned out, probably because "Vienna is different", that more than 90% of all (Cisco) router FastEthernet interfaces connected to the Vienna Internet eXchange have MAC addresses ending with Hex 0 !!! And nope, it's not Halloween today (for those who understand German ;-) This of course does break port-group load-balancing on any such switch with only routers connected ... :-( Any ideas, similar experiences, workarounds (other than to assign MAC addresses to exchange point routers) ? Kind regards CP --- ---------------------------------------------------------------------- --- --- Christian Panigl : Vienna University Computer Center - ACOnet --- --- VUCC - ACOnet - VIX : -------------------------------------------- --- --- Universitaetsstrasse 7 : Mail: Panigl@CC.UniVie.ac.at (CP8-RIPE) --- --- A-1010 Vienna / Austria : Tel: +43 1 4277-14032 (Fax: -9140) --- --- ---------------------------------------------------------------------- ---
On Thu, Oct 12, 2000 at 05:29:16PM +0200, Christian Panigl, ACOnet/VIX/UniVie wrote:
Dear Exchange Point members and operators,
I have a small inquiry today: could you please have a look at the low-order bit distribution of (Cisco) router MAC addresses at your exchange point(s) and tell me the result ?
Attached is the output of 'show arp' on a router connected to LINX, and one connected to AMS-IX /Jesper -- Jesper Skriver, jesper(at)skriver(dot)dk - CCIE #5456 Work: Network manager @ AS3292 (Tele Danmark DataNetworks) Private: Geek @ AS2109 (A much smaller network ;-) One Unix to rule them all, One Resolver to find them, One IP to bring them all and in the zone to bind them.
On Thu, 12 Oct 2000, Christian Panigl, ACOnet/VIX/UniVie wrote:
Now it turned out, probably because "Vienna is different", that more than 90% of all (Cisco) router FastEthernet interfaces connected to the Vienna Internet eXchange have MAC addresses ending with Hex 0 !!! And nope, it's not Halloween today (for those who understand German ;-)
I wonder if it's because of the position that the FE interfaces are in people's Cisco interfaces, coupled with the way the MAC addresses are handed to the port adapter by the chassis in 7xxx boxes, that you tend to end up with Hex 0? I've noticed that on all LINX's own 7200s, fa0/0 ends in 0, and fa1/0 ends in 8, for example. Looking at the LINX table (which Jesper has sent, so I won't waste bandwidth), I see a lot of 0's and 8's. Could it be that there are a lot of people with fa0/0 of 7200's connected to the VIX? We appear to have found a flaw in the Cisco load-balancing hash, which is expecting mostly end-stations (servers/workstations with NICs) with a wide spread of low bits. Of course, we then hit this with they way C7xxx boxes give out MAC addresses and it all goes wonky.
This of course does break port-group load-balancing on any such switch with only routers connected ... :-(
Any ideas, similar experiences, workarounds (other than to assign MAC addresses to exchange point routers) ?
Extreme implement round-robin per packet load sharing, which may help, as well as source port-based and destination MAC-based load-sharing. Foundry implement source MAC or destination MAC load sharing. LINX don't seem to suffer too badly regarding imbalances on our trunk groups. Not sure if it's just that Foundry and Extreme's algorithms are more clueful than Cisco's or the distribution of addresses is better. Mike -- Mike Hughes Network Architect London Internet Exchange mike@linx.net http://www.linx.net/ "Only one thing in life is certain: init is Process #1"
Hi, On Thu, Oct 12, 2000 at 07:51:58PM +0100, Mike Hughes wrote:
On Thu, 12 Oct 2000, Christian Panigl, ACOnet/VIX/UniVie wrote:
Now it turned out, probably because "Vienna is different", that more than 90% of all (Cisco) router FastEthernet interfaces connected to the Vienna Internet eXchange have MAC addresses ending with Hex 0 !!! And nope, it's not Halloween today (for those who understand German ;-)
I wonder if it's because of the position that the FE interfaces are in people's Cisco interfaces, coupled with the way the MAC addresses are handed to the port adapter by the chassis in 7xxx boxes, that you tend to end up with Hex 0?
That's my theory as well. The Chassis has a handful of MAC addresses that are then handed out to the PAs following some algorithm, like "8 MACs per Slot" (which would be needed for OIR on PA-8E's, for example).
I've noticed that on all LINX's own 7200s, fa0/0 ends in 0, and fa1/0 ends in 8, for example. Looking at the LINX table (which Jesper has sent, so I won't waste bandwidth), I see a lot of 0's and 8's. Could it be that there are a lot of people with fa0/0 of 7200's connected to the VIX?
Definitely so at the INXS and DECIX (7200s, some 71xx's, a few 75xx's and GSRs).
We appear to have found a flaw in the Cisco load-balancing hash, which is expecting mostly end-stations (servers/workstations with NICs) with a wide spread of low bits. Of course, we then hit this with they way C7xxx boxes give out MAC addresses and it all goes wonky.
Yep. Gert Doering -- NetMaster -- SpaceNet GmbH Mail: netmaster@Space.Net Joseph-Dollinger-Bogen 14 Tel : +49-89-32356-0 80807 Muenchen Fax : +49-89-32356-299
participants (4)
-
Christian Panigl, ACOnet/VIX/UniVie -
Gert Doering -
Jesper Skriver -
Mike Hughes