Creating networks with VIMAGE jails and epairs.

This is part of a series of notes on the experimental process of getting Mininet to run on FreeBSD.

The first step is to identify the components and commands that are required to implement the basic features. For an emulator like Mininet, this would be 1) the ability to build custom network topologies, and 2) the ability to interact with the topology by sending traffic across it, and monitoring the traffic flowing through the network.

Custom topologies

Mininet allows users to build custom network topologies by interconnecting node and link Mininet objects. Here, jails with VIMAGE replace the mount and network namespaces used to implement the nodes, and epairs replace the veth virtual Ethernet pairs implementing the links.

This link provides clear instructions for getting VIMAGE up and running for a simple topology, making it a good place to start. At the time that this post was written, the stable release (10.2) VIMAGE isn’t enabled by default, and required a custom kernel.

Since the initial (and primary) focus at this time is in building custom topologies, the jails aren’t given their own directory trees, and their paths are set to /.

Handling traffic

In addition to being able to build out topologies, Mininet also allows users to interact with their networks with tools such as ping, traceroute, and tcpdump, which require creating raw sockets from within the jails. This can be enabled by setting security.jail.allow_raw_sockets to 1, or by passing allow.raw_sockets as a command to the jail utility when creating the jails.

Finally, the jails that represent network nodes (e.g. switches and routers, as opposed to end hosts) need some mechanism to move traffic. In Mininet, this would typically be an OpenFlow-programmable software switch such as Open vSwitch or the CPqD software switch. Although the former is available in the ports collection, to reduce the number of moving parts, the if_bridge network device will be used for the time being to narrow down to the core set of commands needed to bring up a topology capable of carrying traffic.

Manual topology construction

The following steps identify the steps and commands required to manually construct what Mininet calls a linear,2 topology:

s1---s2
|    |
h1   h2

where h1 and h2 represent hosts on the network, and s1 and s2, the network nodes (switches).

  1. Prepare the host. After enabling VIMAGE in the kernel:
    # kldload if_bridge
    # sysctl security.jail.allow_raw_sockets=1
  2. Create jails. Since allow_raw_sockets was set in the host, there is no need to pass allow.raw_sockets to jail.
    # jail -c vnet name=s1 jid=1 path=/ persist
    # jail -c vnet name=s2 jid=2 path=/ persist
    # jail -c vnet name=h1 jid=3 path=/ persist
    # jail -c vnet name=h2 jid=4 path=/ persist
    

    jls should now show your jails (jls -v will show you more, including the assigned names):

    # jls
       JID  IP Address      Hostname                      Path
         1  -                                             /
         2  -                                             /
         3  -                                             /
         4  -                                             /
  3. Create bridges in the ‘network node’ jails (JIDs 1,2, and 3)
    # jexec s1 ifconfig bridge1 create up
    # jexec s2 ifconfig bridge2 create up
  4. Create virtual Ethernet links (epairs) and interconnect the jails
    # ifconfig epair1 create      # s1  h1
    # ifconfig epair2 create      # s2  h2
    # ifconfig epair3 create      # s1  s2
    # ifconfig epair1a vnet s1
    # ifconfig epair1b vnet h1
    # ifconfig epair2a vnet s2
    # ifconfig epair2b vnet h2
    # ifconfig epair3a vnet s1
    # ifconfig epair3b vnet s2
  5. Add epair interfaces to each bridge and bring them up
    jexec s1 ifconfig bridge1 addm epair1a addm epair3a
    jexec s1 ifconfig epair1a up
    jexec s1 ifconfig epair3a up
    jexec s2 ifconfig bridge2 addm epair2a addm epair3b
    jexec s2 ifconfig epair2a up
    jexec s2 ifconfig epair3b up
  6. Configure IP addresses for ‘host’ jail interfaces
    # jexec h1 ifconfig epair1b 10.0.0.1 up
    # jexec h2 ifconfig epair2b 10.0.0.2 up

Sanity-checking the topology

It should now be possible to ping from one host to another:

# jexec h1 ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=0.046 ms
...
^C
--- 10.0.0.2 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.046/0.052/0.055/0.004 ms

It should also be possible to monitor the traffic passing through a network node (e.g. by running tcpdump) while the hosts are pinging one another.

Teardown

Once a topology is no longer needed, it should be torn down and the virtual links and jails destroyed.

  1. Remove epairs from jails and destroy them (removing one end of an epair destroys both endpoints)
    # ifconfig epair1a -vnet s1
    # ifconfig epair2a -vnet s2
    # ifconfig epair1a destroy
    # ifconfig epair2a destroy
  2. Destroy the bridges
    jexec s1 ifconfig bridge1 destroy
    jexec s2 ifconfig bridge2 destroy
  3. Destroy jails
    jail -r s1
    jail -r s2
    jail -r h1
    jail -r h2

The idea is that the commands (and procedures) that have been identified here can be retrofitted into Mininet.

[to be continued]

Multiple controller domains in Mininet, and link.

Usually, a network emulated in Mininet assumes that all of the switches in a network are connected to the same controller(s). It is possible to point different switches in the same network to different controllers, as shown in some examples here and here, but I wanted make the procedure of creating such a network less manual, and handle a bit more like the existing Topo and Mininet classes.

What I ended up with was a Domain construct that does the work of pointing different switches in the same Mininet topology to different controllers in a way that fits better with the usual Mininet APIs. A Domain defines a group of switches in a network that are connected to the same (set of) controller(s). A Domain can be built up in a similar way as Topo objects and loaded into the emulated network, and requires no modifications to Mininet to be used.

(For those who come across this page and think this sounds useful, a link to the source is provided here.)

Example 1 – two-domain network.

To use this construct, multiple Domains are loaded into a single Mininet object and then interconnected, resulting in a single network where different parts are controlled by different controllers. For example, if the following topology is desired:

   domain 1    :   domain 2
               :  
     [c1]      :     [c2] 
     /  \      :     /  \
[sw11]--[sw12]---[sw21]--[sw22]
               :

The network can be seen as being built of two domains, each with two switches. Then, the following Domain defines a single one of those domains:

class TestDomain(Domain):
    """
    A tiny domain of two nodes connected together
    """
    def build(self):
    sw1 = 'sw%s1' % self.getId()
    sw2 = 'sw%s2' % self.getId()
    self.addSwitch(sw1)
    self.addSwitch(sw2)

    self.addLink(sw1, sw2)

And then, given that the TestDomain example is saved in a file called testdomain.py, the following script will construct the full network and start a CLI:

from domains import Domain
from testdomain import TestDomain
from mininet.net import Mininet
from mininet.cli import CLI
from mininet.log import setLogLevel, info

if __name__ == '__main__':
    setLogLevel('info')

    #instantiate and build two TestDomains with Domain IDs '1' and '2'
    domain1 = TestDomain(1)
    domain2 = TestDomain(2)
    domain1.addController('c1')
    domain2.addController('c2')
    domain1.build()
    domain2.build()

    # add prepared domains to the network, and initialise
    net = Mininet()
    domain1.injectInto(net)
    domain2.injectInto(net)
    net.build()

    # connect the domains together
    Domain.interConnect(domain1.getSwitches('sw12'), domain2.getSwitches('sw21'), net)

    # start()ing the Domains associates the switches in the domains
    # with the controllers that had been added to them
    domain1.start()
    domain2.start()

    # start up a CLI, as usual. net.stop() will stop the domains it knows about.
    CLI(net)
    net.stop()

For such a small multi-domain network, it might seem overkill – but domains simplify the script given more complex or larger topologies.

At the end of the day, domains are still a hack, so things like connecting two domains is still pretty clunky-looking, even though it’s functional.

Example 2 – building Domains sans child class.

Just like a Topo or Mininet object, you don’t necessarily need a child class of a Domain to build one:

# default ID for a Domain is 0
domain = Domain()
domain.addSwitch('sw1', cls=UserSwitch)
domain.addSwitch('sw2', cls=UserSwitch)
domain.addLink('sw1', 'sw2')

# use as above, sans the call to build()
domain.injectInto(net)
...

As shown above, normal Mininet API arguments can also be passed into a Domain’s methods. Some things differ slightly, like how the string names of nodes are passed to addLink(). Internally, all that these methods do is store the arguments until injectInto() is called, upon which they are passed to the Mininet object’s mid-level API and realized as Mininet network objects.