Getting Xen up and running:...

Networking: Background

Whenever a domU is started, it receives at least one virtual Ethernet card, eth0. That card is hard-wired to a virtual interface, vifU.0, in dom0. The final network architecture depends on what the vif-script does with that vifU.0 interface.

The network-script and vif-script options in xend-config.sxp simply name scripts in /etc/xen/scripts which set up or tear down networking for each virtual machine. Don't forget to do sudo /etc/init.d/xend restart after changing xend-config.sxp. For more information on the configurations which come with Xen (bridged/routed/NAT), see their respective scripts, or the XenNetworking article linked to at the top of this post.

In bridged mode (the network-bridge and vif-bridge scripts), the physical ethernet card of dom0, and all vifU.0 endpoints, are connected to a virtual bridge, xenbr0. The bridge works like a real bridge, learning which MAC addresses are at which ports, and sending traffic that arrives from any card to the appropriate destination, or to all ports if the destination is unknown. This is how any domU started can appear as peers on the same network that has dom0. This also means that domU should be given a sane IP address for the physical network, or use DHCP to get an address from the physical network's DHCP server. In other words, bridged mode does not really create a "virtual network" at all; it simply connects all Xen domains to the real network.

In routed mode (network-route and vif-route, naturally), each vifU.N interface must have a static IP address, and the vifU.N card is given a static route in dom0 for that single address. The bridge interface, xenbr0, is not used in this configuration, and will only be visible in dom0 if you've used bridge-mode since rebooting.

Routed mode is fundamentally incompatible with DHCP. If a domU doesn't have a static address, the vif-route script doesn't assign any route to the new domU. Since the DHCP packet is addressed to the new IP address, it follows the default route onto the physical LAN, instead of going to the domU which actually needs it. On the other hand, bridged mode operates below the IP level, which makes it difficult to block DHCP requests from traveling out to the physical branch of the LAN. netfilter actually does seem powerful enough to do it, but I couldn't figure out how.

DHCP for domU

To make DHCP work, add a line to the domU's configuration file (e.g. /etc/xen/xen1.cfg):

kubikal$ sudo -e /etc/xen/xen1.cfg
<add the following line in the networking section, above the 'vif' setting:>
dhcp = "dhcp"
<you should also set a MAC address:>
vif  = [ 'mac=aa:cc:00:00:00:01' ]
<save and quit>
kubikal$ sudo xm shutdown xen1
<wait for the VM to shut down....>
kubikal$ sudo xm create /etc/xen/xen1.cfg

Now xen1 will be able to get an address from any DHCP server that it can see, either the existing one for the physical LAN in bridged mode, or the one we set up in routed+DHCP mode.

Customized Routing

Unable to do what I wanted with the default Xen scripts, I decided to customize my own. It turns out that the network-route script was suitable for my purposes, but vif-route was not. I created a new script, called vif-private-subnet, which simply adds the vif to a bridge, and brings it up. Make sure this script is executable after placing it in /etc/xen/scripts. Here is the code for the script:

#!/bin/bash
#============================================================================
# /etc/xen/vif-private-subnet
#
# Script for configuring a vif on a private subnet, visible only to dom0.
# The hotplugging system will call this script if it is specified either in
# the device configuration given to Xend, or the default Xend configuration
# in /etc/xen/xend-config.sxp.  If the script is specified in neither of those
# places, then vif-bridge is the default.
#
# Usage:
# vif-route-bridge (add|remove|online|offline)
#
# Environment vars:
# vif         vif interface name (required).
# bridge      bridge interface name (default br0).
# XENBUS_PATH path to this device's details in the XenStore (required).
#============================================================================

dir=$(dirname "$0")
. "$dir/vif-common.sh"

bridge="${bridge:-br0}"

case "$command" in
    online)
        brcmd='addif'
        cmdprefix=''
        ;;
    offline)
        do_without_error ifdown ${vif}
        brcmd='delif'
        cmdprefix='do_without_error'
        ;;
esac

${cmdprefix} brctl ${brcmd} ${bridge} ${vif}
r2=$?
${cmdprefix} ifconfig ${vif} up
r1=$?

log debug "vif-route-bridge $command for $vif on $bridge: ifconfig=>$r1 brctl=>$r2"
if [ "$command" == "online" ]
then
  success
fi

To keep the Xen side as simple as possible, I configured the bridge outside of the Xen scripts. (Unlike the bridge scripts, which take care of creating the bridge themselves.) This is the relevant part of /etc/network/interfaces:

# Create the backend Xen bridge
auto br0
iface br0 inet static
	# bridge_ports tells ifupdown that this is a bridge
	bridge_ports none
	# assign address for dom0 to use on the virtual net
	address 192.168.0.254
	netmask 255.255.255.0
	# Fix checksum errors (DHCP in domU fails without this)
	post-up ethtool -K br0 tx off

# Statically configure eth0 so the DSL modem can have a static route
# through eth0 to the 192.168.0.x network on br0. This lets machines
# on the LAN talk to the domUs by IP. The gateway here is the DSL
# modem.
auto eth0
iface eth0 inet static
	address 192.168.11.100
	netmask 255.255.255.0
	gateway 192.168.11.254

I'm not clear on the reasons yet, but giving the bridge an IP automatically lets dom0 use it as a virtual network card. No extra interface needs to be created and attached to the bridge, and no route needs to be created manually. Do note, however, that the gateway given for eth0 needs to have a static route in order for the domU machines to be visible from the LAN. That route should direct traffic for 192.168.0.0/24 to 192.168.11.100, the public IP of kubikal.

Serving DHCP

I installed dnsmasq for serving DHCP to the domU machines. dnsmasq also serves DNS, and can link DHCP addresses to hostnames. I decided to have '.xen' as the domain for the virtual network, and the IP range was set in the bridge configuration above. For symmetry with the DSL modem, dom0 also sits at the last address on the virtual network, 192.168.0.254.

This is the minimal set of options needed in /etc/dnsmasq.conf:

domain-needed
resolv-file=/etc/resolv.upstream.conf
local=/xen/
user=dnsmasq
group=daemon
domain=xen
dhcp-range=192.168.0.1,192.168.0.253,3d
dhcp-leasefile=/var/lib/dnsmasq/leases

I moved the initial /etc/resolv.conf to the mentioned /etc/resolv.upstream.conf, and then set /etc/resolv.conf to direct all traffic to 127.0.0.1, where dnsmasq listens. All the DHCP clients, by default, are given the address of dom0 as a DNS server, and thus also resolve names using dnsmasq.

The final piece of the puzzle is to make the domU send a hostname to dnsmasq. Without that hostname, dnsmasq doesn't know what the domU should be called, and therefore can't add it to DNS. On Debian, the hostname line in /etc/network/interfaces only applies to the DHCP clients which are not installed by default. dhclient3 does its work using /etc/dhcp3/dhclient.conf instead. Once send host-name is uncommented and set to "xen1.xen", then the DNS query returns the correct IP address. (And I begin to think of a better name than xen1....)

Is that all?

It's been a long, complicated process, with a number of false starts. Hopefully, I didn't miss anything. Let me know if I did.