That Chris Brown's Blog » kvm

Xen vs KVM – horses for courses?

Chris — Sat, 17 Oct 2009 12:04:14 +0000

As anyone who’s read any of my previous posts will know, I’ve been considering Red Hat’s adoption of KVM over Xen and wondering if a move to KVM is the right thing to do.

I think “The right thing” depends a fair bit on where you are, and what your requirements are. Obvious? Perhaps, but then sometimes it’s the obvious things we overlook.

I think there’s no pressing need to migrate my hosted server. It works pretty well on Xen and now I’ve sorted out my mess of repo priorities, and I’m a bit less cavalier with updates, I don’t expect it to present me any significant problems.

My home server is a different thing. It’s not currently commissioned so there’s no downtime to worry about, I’d like to be able to run PVM and HVM Windows and Linux guests, including possibly LinuxMCE one day, and carrying most weight for me at present, it’s going to also be my network gateway.

Now, purists might not think running VM services on my gateway is a smart idea, and I should put in a separate host for each, but it’s a model I’d like to try out as I have some work project ideas that might benefit.

My thinking is that if I’m doing security policy, NAT, bandwidth shaping and VPN on a machine that also runs guests, KVM might make sense – doing that work in a Xen dom0 would involve a bit more switching of network data in and out of the Hypervisor and guests. In theory, Linux being the hypervisor should mean it’s only doing that for network traffic to guests, which will only be anything I can’t sensibly run in the base Linux OS.

There may be some argument that my hosted server may benefit too, as in a bonded setup, the current state of Xen networking seems to suggest that the dom0 has to do all the work anyway – I have my dom0 using regular Linux networking (patched to fix an issue bridging a bonded interface) to get my guests on a bridge to a fully bonded interface.

KVM will be an interesting experience to compare with Xen. If the networking is faster, as long as performance of guests is good, it may make sense to choose it ultimately because Red Hat have, and out-of-the-box compatability is no bad thing.

For now, I’m happy to keep my hosted workhorse on Xen and try the new contender out on a home server first.

LVM saves the day, possibly…

Chris — Tue, 28 Jul 2009 18:53:42 +0000

OK, so I’m mulling the move from Xen to KVM. This initially looked like I’d have to make duplicate virtual disks in KVM for my Xen guests, then shut the guests down, block copy content across and patch up MBR and boot my physical box into a stock kernel and hope KVM comes up with the goods.

Well, a little bit of experimentation with a rarely-used VM and I’ve found much to my joy that because I used LVM logical volumes to create my Xen guests virtual disks, I can use kpartx to add device mapping for them and (when the guest is safely shut down!) mount them up in my dom0 and read/write to them. Thi sis good news because it means they have a full virtual disk setup, with a /boot partition, a root partition and a swap partition all living in a single logical volume. This means KVM should pick them straight up and work with them unmodified.

Which hopefully means all I have to do is set a stock kernel to be the boot default in my guests, shut them down, set my dom0 to boot from a stock kernel, load up the KVM kernel module and add some guests from the logical volumes.

If it’s that easy, I’ll be a happy person indeed!

In theory – migrating from Xen to KVM under CentOS 5.3

Chris — Tue, 28 Jul 2009 14:56:59 +0000

I have a CentOS 5.3 (actually a CentOS 5.2 build upgraded in yum, but that’s not terribly important) x86_64 machine that is in fact a Xen hypervisor kernel with said CentOS 5.3 running as the dom0 (service VM), and currently four guest VMs also running CentOS 5.2 x86_64, upgraded to CentOS 5.3 with yum. This was a kind of interesting project in itself, in that I used the Gitco repos in order to get Xen 3.3.1 on the CentOS build, and I had to do some hackery of the Xen and stock CentOS networking scripts to make things work. The box doesn’t use Xen network scripts and xenbr0, because they broke pretty badly with bonded dual NICs. It has a fairly standard Ethernet bond setup going, then a patched RH network script so bridges don’t init before the bond, a standard Linux bridge br0, which all my Xen vifs attach to, and a virbr0 which doesn’t have any physical NICs attached, but does have an IP address and dhcpd running against it, so I have a place to quickly fire up new VM guests without having to assign network addresses.

This complexity lead to a few problems, partly during installation, when I ended up temporarily having to go wiht a non-Xen kernel in order to update bits and ipeces and set up repos so I could go back to a Xen kernel, and when yum wanted to update to kernel-xen-2.6.18-128.1.10.el5 and like a trusting fool I let it, and promptly had no running VMs on the next reboot. Hacking grub back to 2.6.18-128.1.6.el5xen seemed to sort it, and it never got changed from there.

Actually, to tell the trust, I also have another machine built a similar way, sans CentOS 5.3 upgrade. This one was meant to potentially be a home to a windows machine, or most likely a Linux MCE installation running in a non-PV guest. That one didn’t get installed yet, and has been sitting under my desk for many a month, waiting for cabling. Oops.

My guest DomUs are living in Xen virtual disks which are inside LVM logical volumes as far as the dom0 is concerned. If I’d been smarter, I might have set up three LVs for each domU – one for /boot, one swap and one for /, so that I cold more easily manage them from not only the guest domU, but also the dom0 if the domU was offline for any reason.

The increased exposure KVM is getting in RHEL 5.4 is making me wonder if all this Xen tinkering is worth it. My machine only runs Linux guests, and then only really for a bit of process separation security. Given that the dom0 has to do a bit of service work for guest domUs anyway, and it seems that this involves a bit of context switching overhead, I don’t see that doing the hypervisor function in the kernel is going to be a problem for me, especially on that currently unused box, which is going to be a home all-in-one media, file/print, NATting, caching and bandwidth arbitrating machine. As I was going to just accept doing all the networking for that (including the firewall/NAT/bandwidth and some VPN stuff) in the dom0 instead of making the Xen networking even more complicated with dual virtual NICs and a virtual firewall box, I guess just making all that work in plain old Linux and then loading up a kernel module when I want a media centre guest VM (yes, I know I could just be less lazy and make all the bits of Linux MCE work under CentOS natively, but starting a VM and installing an OS-up image for LMCE seems like a good thing) installing makes a bit of sense.

The currently unused machine isn’t so bad – as it currently has no VMs and didn’t get the 5.2/5.3/Gitco hybrid horror treatment, I could just strip Gitco repo, group uninstall Virtualisation (sorry Virtualization), and boot it as a regular Linux kernel, do the networkingy bits (and make better use of my rather rubbish 1.5Mb/s of patchy ADSL) and then worry about that media server when the time is right. The dual-NIC networkign will be far easier – one outside one inside, attach a separate bridge to each and I could then attach VMs to the “public” or “private” bridges and the host OS will iptables them like any other hosts. Great!

It’s that first server that is going to be ‘interesting’. Fedora’s virt-v2v is a ways off yet, so I’ve got to think of smart ways of moving painlessly from Xen to KVM without losing services for long. The current outline plan goes something like:

Install a plain old Linux kernel into each VM guest
Install a plain old Linux kernel into the current dom0
Make new LVM logical volumes for guest VMs – possibly three each for /boot, swap and /
Modify guest VM grub config to default the plain old Linux kernel
Shutdown guest VMs
Reboot dom0 as a plain old Linux kernel, albeit with KVM support loaded
Make any networking changes required
Create new KVM virtual machine images in the logival volumes
Stop the new KVM guests
Mount guests Xen virtual ‘disks’ (actually just some partitions I think, not full bootable disk images) on loopback in the dom0
dd the contents of the Xen partitions into the new LVM logical volumes, overwriting the guest images created at step 7
Start the KVM guests
Panic when it goes wrong (had to be step 13, didn’t it?)

I’m hoping that the fact I already do the networking outside of Xen, and just use standard Linux bridging on and the libvirt virbr0, will mean a slightly easier migration – all I have to do is get my instances booting off a standard kernel instead of a Xenified one, and inside a different virtual disk format.

I’ll still need my standard-breaking networking patch to enable bonds before bridges though. Hopefully RH fix that one in RHEL 5.4 so CentOS picks it up in the next release.