[Spice-devel] RFC: HowTo setup a spice live migration test setup
Hans de Goede
hdegoede at redhat.com
Tue Aug 31 00:47:13 PDT 2010
Hi All,
As you know my first week I've been working on setting up
a live migration test environment (not that I've run much tests
yet).
I thought it would be good to document what I did / how to
create a similar setup. This is intended to be put at:
http://spice-space.org/page/Howtos
But before I put it there I was wondering if anyone has any
comments / corrections ?
Here is my draft version of the doc:
HowTo setup a spice live migration test setup
---------------------------------------------
This document describes how to setup a realistic environment
for doing live migration of qemu virtualmachines (vm's)
which are using the qxl (spice) vga device and have a spice
client connected.
The described setup does this all without using RHEV-VM,
this is mostly meant for people doing spice development who
want to test live migration without using RHEV-VM.
1. Needed hardware
------------------
For a *realistic* setup you will need at least 4 machines:
1) Hypervisor A, this machine will be running vm's
2) Hypervisor B, this machine will be running vm's
3) Storage server, for live migration storage shared between
the hypervisors is needed in a realistic environment this
will live on a separate machine or a SAN
4) A client
These machines need to be connected to each other using a
network with good performance at a minimum a 100Mbit switched
network, if you want to go fancy you can create a separate
storage LAN and normal network LAN.
The 2 hypervisors need to be 64 bit machines with hardware
virtualisation capable cpu's. Preferably they both should
have AMD or both Intel cpu's. If one of them has an AMD cpu
and one an Intel cpu you can only use recent qemu-kvm versions.
2. Installing the machines
--------------------------
1) The hypervisors, you may want to make the hypervisors
multiboot between for example EL-5 x86_64 and EL-6 x86_64,
so that you can test live migration with EL-5 hypervisors
with a EL-6 spice client connected or visa versa. Note that
live migration between different version hypervisors is not
supported for all possible combinations.
Once you've the base os's installed you can either install
the spice client, server and qemu version from packages or
build them from source (see the building spice howto).
2) The storage server. The storage server can serve vm disk
images either over nfs or export them as iscsi disks. Since
iscsi disks perform better, this documents describes how to
do things using iscsi.
For the storage server you can install any linux as base os
you like. Before installing the base os you need to think
about where the images will be stored, will they be normal
files, or are you going to make them disk partitions / lvm
logical volumes? Files are the most flexible, performance
wise disk partitions or logical volumes are better. Logical
volumes also allow a decent amount of flexibility, so those
are advised.
Once the base os is installed, it is time to configure the
machine to serve a number of iscsi targets. First install
scsi-target-utils, for example:
yum install scsi-target-utils
If you've chosen to use files you need to create empty images
files using for example dd:
dd if=/dev/zero of=image.img bs=1024 count=20000000
Now edit /etc/tgt/targets.conf and define as many targets there
as you want to have vm disk images. The file includes examples
of howto define a target. Once the targets are defined restart
the tgtd service, for example on Fedora / EL:
service tgtd restart
3) The client, you likely want to make the client machine
multiboot. So that you can test with both the linux and Windows
client and that you can test different client versions.
3. Prepping the hypervisors
---------------------------
Besides installing spice server and qemu noth hypervisors require some
more special preperations. They need to connect to the iscsi targets
on the storage server and there network settings need to be configured
so that qemu can run in bridged mode, which is necessary to keep
the vm's network connection working after migrating it to a different
machine.
1) Preparing the network configuration to allow vm's to connect in
bridged mode. On Fedora / EL this can be done by modifying the
relevant /etc/sysconfig/network-scripts/ifcfg-eth# to look like this:
DEVICE=eth#
TYPE=Ethernet
HWADDR=xx:xx:xx:xx:xx:xx
ONBOOT=yes
NM_CONTROLLED=no
USERCTL=no
BRIDGE=br0
And then create a /etc/sysconfig/network-scripts/ifcfg-br0 file like
this:
DEVICE=br0
TYPE=Bridge
ONBOOT=yes
NM_CONTROLLED=no
USERCTL=no
DELAY=0
BOOTPROTO=none
IPADDR=192.168.1.101
PREFIX=24
GATEWAY=192.168.1.1
DNS1=192.168.1.1
IPV6INIT=no
Adjusting the IP settings to the settings the relevant eth# used to have.
Then restart the network boot service:
service network restart
On Fedora you also need to enable the network servce to run on reboot,
using for example "ntsysv".
Last you will need to create a /etc/qemu-ifup script, with the following
contents:
#!/bin/sh
echo "Executing /etc/qemu-ifup"
echo "Bringing up $1 for bridged mode..."
ifconfig $1 0.0.0.0 promisc up
echo "Adding $1 to br0..."
brctl addif br0 $1
sleep 2
Don't forget the chmod +x it!
2) Make the iscsi export disk images available as iscsi disks on *both* the
hypervisors. Connecting to an iscsi disk consists of 2 steps:
1) Discover available images, to discover available images do:
iscsiadm -m discovery -t sendtargets -p 192.168.1.100
Where 192.168.1.100 is the ip address (or the hostname) of the
storage server. This gives for example the following output:
192.168.1.100:3260,1 iqn.localdomain.shalem:iscsi1
192.168.1.100:3260,1 iqn.localdomain.shalem:iscsi2
2) Login to specific images, for example:
iscsiadm -m node -T iqn.localdomain.shalem:iscsi1 -l
Note that atleast on Fedora / EL you need to do this only once,
if the iscsi boot service is enabled (the default) the system
will automatically re-connect to these iscsi "disks" on reboot.
3. Starting a vm
----------------
Now it is time to start a vm with qxl vga. The basic form is like this:
sudo qemu-system-x86_64 -enable-kvm -m 2048 -smp 2 -name foobar \
-drive file=/dev/disk/by-path/ip-192.168.1.100:3260-iscsi-iqn.localdomain.shalem:iscsi1-lun-1,media=disk \
-net nic,macaddr=52:54:00:7a:b4:7c,vlan=0 -net tap,vlan=0 \
-vga qxl -spice port=5930,disable-ticketing \
-monitor stdio
You should change the -name, file=, macaddr= and port= parameters for
each additional vm you want to start. Note the /dev/disk/by-path/ method
of pointing at the iscsi target which holds the vm's disk image. The
drive must be specified this way so that the filename will work on both
hypervisors, this is a must for migration to work.
When you first start the machine you will likely want to add an option
for it to find installation media, usally using the -cdrom parameter, see
man qemu.
If your virtual machine is going to run windows xp, you likely want to
add ",model=rtl8139" to the -net nic,... parameter as the default is
to emulate an e1000 and xp does not have a driver for that.
If your virtual machine is going to run a recent linux, you may want to
",model=virtio" to both the -drive file=... and the -net nic,...
parameters. With older qemu's you also need to add ,boot=on to the
-drive paramter to boot from a virtio disk.
4. Connecting the client
------------------------
Connecting to the vm using the linux client is simple:
spicec -h 192.168.1.101 -p 5930
Where the -h parameter is the ip address or hostname for the hypervisor
and -p is the port you specified for spice when starting the vm
Note that with older spice versions spicec is installed under /usr/libexec
and thus needs to be started as:
/usr/libexec/spicec -h 192.168.1.101 -p 5930
5. Migrating the vm to the other hypervisor
-------------------------------------------
To migrate the vm to the other hypervisor, you first need to start
a vm on the other hypervisor ready to receive the vm state info from
the running vm. Use the exact same qemu cmdline as you used to start
the original (source) vm and add "-incoming tcp:0:4444" at the end
of the cmdline.
Then on the monitor if the source vm (the terminal / console from
which the source vm was started), type the following commands:
"migrate_set_speed 5m"
Limit the speed with which the migration data will be send from
one hypervisor to the other, otherwise they take all bandwidth
and the user experience in the client suffers.
spice_migrate_info 192.168.1.102 5930
Tell qemu to tell the spice client to get ready to stop
talking to the current vm (192.168.1.101:5930) and to
connect to the new vm (192.168.1.102:5930) as soon as the
migration is done.
migrate -d tcp:192.168.1.102:4444
And start the actual migration, when this command exits the
migration is finished and the source vm can be quit.
Regards,
Hans
More information about the Spice-devel
mailing list