Kernel Virtual Machines using libvirt

I use iSCSI from a giant ZFS storage pool in order to setup virtual machines. There's a couple advantages to this, as you get the power of ZFS snapshots and the ability to grow your virtual machine's disks fairly easily. To make administrating things a bit easier, I use one large iSCSI target and put LVM on top of this. So really this article will apply to anyone that wants to use logical volumes for their disks, which is much preferred over using files.

I run my servers all in a headless manner, so I will be using virsh and virt-install over virt-manager. Also, I will be using Fedora 12 for the virtual host as this is where KVM is actively developed and generally where you will find the best performance / stability.

Installing the required software

Fedora has a nice "Virtualization" group package which will get you setup to run KVM based virtual machines.

# yum groupinstall "Virtualization" -y
# chkconfig libvirt on

Since I want my virtual machines to have direct access to the network, I will put one of my NICs into a bridge. You will want to do the same unless you prefer the default NAT'd mode.

Network configuration

To configure your network you need to create /etc/sysconfig/network-scripts/ifcfg-br0 for a bridged interface. Use the following as an example:

# cat ifcfg-br0
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
IPADDR=10.10.0.3
NETMASK=255.255.0.0
ONBOOT=YES

Then same directory, you need to edit the file for the interface you want attached to the bridge. For my configuration I'm using eth3.

# cat ifcfg-eth3
DEVICE=eth3
HWADDR=00:15:17:D6:35:69
ONBOOT=YES
TYPE=Ethernet
IPV6INIT=no
USERCTL=no
BRIDGE=br0

That's it. I highly recommend removing NetworkManager. Don't just disable it, it's a waste of space and complicates things.

# yum remove -y NetworkManager
# chkconfig network on
# service network restart
# ifconfig br0
br0       Link encap:Ethernet  HWaddr 00:15:17:D6:35:69  
          inet addr:10.10.0.3  Bcast:10.10.255.255  Mask:255.255.0.0
          inet6 addr: fe80::215:17ff:fed6:3569/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:68994 errors:0 dropped:0 overruns:0 frame:0
          TX packets:25910 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:59178792 (56.4 MiB)  TX bytes:1746255 (1.6 MiB)

Now you have br0 to put virtual NICs (those created by KVM) on and access your network directly.

Disk configuration

Since LVM is useful in this situation, we're going to use it. I'm starting with a freshly partitioned disk so I will go over creating a proper LVM configuration. You need to know which partition and disk you will be creating this on, so if you don't know, do not just blindly run these commands. Consult with fdisk -l and the man pages.

# fdisk -l /dev/sdc

Disk /dev/sdc: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xae668fde

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       26108   209712478+  8e  Linux LVM

# pvcreate /dev/sdc1
# vgcreate zfs_lun /dev/sdc1
# lvcreate -L 20G zfs_lun -n guest1
# lvdisplay /dev/zfs_lun/guest1 
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  --- Logical volume ---
  LV Name                /dev/zfs_lun/guest1
  VG Name                zfs_target
  LV UUID                pVt1tv-OZSr-3joc-xaO1-iklH-z2eS-FEAbRf
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                20.00 GB
  Current LE             5120
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

You now have a logical volume of 20GB that you can use for installing virtual machines. To briefly go over the commands above, first we needed to create the physical volume on top of /dev/sdc1. From there you create a volume group, which is a collection of logical volumes. You can have more than one of these on a disk, but for my purposes I just let the volume group consume the entire physical volume. Finally, we setup a logical volume with 20GB of space. The last line is useful for double checking your work, which is always a good idea.

libvirt Configuration

Since I'm on an isolated trusted network, I like to have full access to the VNC serial connections made available. By default these only listen on localhost, and ssh port forwarding becomes tedious when you are trying to manage more than one at a time. So to open up VNC to the world, you need to uncomment this line in /etc/libvirt/qemu.conf:

# VNC is configured to listen on 127.0.0.1 by default.
# To make it listen on all public interfaces, uncomment
# this next option.
#
# NB, strong recommendation to enable TLS + x509 certificate
# verification when allowing public access
#
vnc_listen = "0.0.0.0"

Restart libvirt after, also we'll make sure it starts on boot at this time:

# service libvirtd restart
# chkconfig libvirtd on

virt-install, it's easy!

This command line utility is great. I strongly recommend reading the man page for this one as you will find useful information for installing guests the way you would like. I find it much more powerful than virt-manager, as last I checked virt-manager leaves out a few useful features from the libvirt library. (Disclaimer: I haven't used virt-manager in a long long time, at least in development years)

# virt-install --name guest1 \
    --ram 1024 \
    --location http://mirror.crazy.lan/Fedora/12/Fedora/x86_64/os \
    --file /dev/zfs_lun/guest1 \
    --vnc \
    --bridge=br0

The long names of each option should make this fairly easy to understand. Location refers to the location of your installation media. This can be a variety of options (check the man page!) but http is my personal choice. File of course specifies the disk that you will be using, vnc tells qemu-kvm to allow VNC access, and bridge specifies the network bridge to attach to.

An interesting option that I once played around with was --pxe. This allows your VM to boot over the network and install from there. This is a great little option if you've already got the infrastructure in place for netbooting, and possibly well worth setting up if you're tired of carrying around install CDs.

virsh

Your virtual shell. This is where you can modify your VMs and again, well worth looking at the man page for. If you're too lazy, just type virsh help and you'll be presented with a list of options. So earlier we started an install, how can we check if it's running and connect to the serial console?

# virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list
 Id Name                 State
----------------------------------
  1 guest1               running

virsh # vncdisplay guest1
:0

This tells you what VMs are running and the state they are in. The vncdisplay command lets you find out which display to connect to when using vnc. So from your management station you can run the following to get in:

vncviewer virt02.crazy.lan:0

Now you have a VNC window open and you will be taken through a normal installation process. That's it. There's a ton of things you can do with libvirt and really it just requires some reading. At some point I may consider looking more into virt-manager, but honestly it leaves some to be desired compared to the already powerful command line tools.

Tags: