After the initial step of determining the set of commands needed to bring up a VIMAGE/jail network (described here), I started reorganizing it to more closely mimic the order in which these commands would be called by a Mininet script. The typical order of operations and their corresponding commands are roughly:
- Instantiate a topology (Topo) object: (no corresponding step)
- Add Switches and Hosts to Topo object: Start up some jails with bridges, if they are switches, and shells, if they’re hosts
- Interconnect the Switches and Hosts by adding Links to the Topo object: create epairs, and move the interfaces to the jails)
- Initialize the network with the Topo object as its topology: Bring the interfaces up, and if the jails represent switches, add the interface to the bridge
But, while trying to recreate the same –linear,2 network, I noticed the following messages in
epair2b: DAD detected duplicate IPv6 address fe80:2::ff:70ff:fe00:40b: NS in/out/loopback=4/1/0, NA in=0 epair2b: DAD complete for fe80:2::ff:70ff:fe00:40b - duplicate found epair2b: manual intervention required epair2b: possible hardware address duplication detected, disable IPv6 epair3b: DAD detected duplicate IPv6 address fe80:2::ff:70ff:fe00:40b: NS in/out/loopback=4/1/0, NA in=0 epair3b: DAD complete for fe80:2::ff:70ff:fe00:40b - duplicate found epair3b: manual intervention required epair3b: possible hardware address duplication detected, disable IPv6
And a bit above, the following:
epair2a: Ethernet address: 02:ff:20:00:03:0a epair2b: Ethernet address: 02:ff:70:00:04:0b epair2a: link state changed to UP epair2b: link state changed to UP epair3a: Ethernet address: 02:ff:20:00:03:0a epair3b: Ethernet address: 02:ff:70:00:04:0b epair3a: link state changed to UP epair3b: link state changed to UP
As the DAD (Duplicate Address Detection) warnings suggested, the same MAC (hardware) addresses were indeed being reused for the epair* interfaces being created.
Searching for the warnings eventually brought me to a thread describing the exact mechanics behind the issue – I had interleaved the steps for creating and moving the interfaces. The MAC addresses for epair* interfaces are generated from a globally tracked
if_index counter. This value increases by one for each interface created (so +2 for each
ifconfig epair create), and decreases by one for each destroyed or moved to a VIMAGE jail. The problem arises when epair creation is interleaved with moving them to jails; The
- increases by two for the first epairs created
- drops back down by two when they are moved, and
- take on the same values as for the first epairs when the next epairs are created
In fact, since the value of
if_index is used directly in the 4th and 5th byte of the MAC address (first and last are hard-coded and the rest, set by other means), we can see in the
dmesg output that the index values 3 and 4 are being reused repeatedly.
It also explains why I didn’t see this issue initially, since I was creating all of the epairs at once and then moving them later on, making the changes in
if_index monotonic. While I could reorganize the commands so that it both follows Mininet’s conventions and the
if_index doesn’t fluctuate, I manually assigned unique addresses to each epair for the time being:
# ifconfig epair1a ether 02:ff:00:00:01:14 #from s1 (jid 1) to h1 (jid 4) # ifconfig epair1b ether 02:ff:00:00:01:41 #from s1 (jid 4) to h1 (jid 1) ...
Of course, I’ll use a less manual approach for generating a unique MAC address in Mininet.
if_index is not guaranteed to monotonically increase, but the number in the interface name (the ‘n’ in epair[n]) does, so I decided to use that as a base for my unique MACs.