OK, OK – I was stupid.

As part of my ongoing move from Xen to KVM for VM hosting, I fireed up a server that had been sitting about too long switched off under the desk, thinkig that it would make an ideal machine to try KVM out on.  A yum groupremove Virtualization seemed to produce no issues, so I was looking good to go with a bit fo repo cleanup and then an update before upgrading from CentOS 5.2 to 5.3 with yum.  Then I did something stupid.

I renamed the logical volume that / is mounted from, becuase I thought the name wasn’t descriptive enough.  Then I rebooted the box.  Oh yes.

Cue a kernel panic when grub couldn’t find the root partition.  Oh Chris, how SMRT!  The physical box doesn’t have a CD/DVD drive (it’s meant to be a headless box) but luckily it does have USB ports, and I had a spare screen and keyboard about, and I happeend to have a 4GB USB thumb drive in my laptop bag.  A quick Google search turned up Pendrivelinux.com and a handy guide to getting CentOS 5 working on USB.  I already had a CentOS 5.2 Live CD image on my hard drive, so a quick upload of the thumb drive contents, a quick download of the Fedora LiveUSB Creator and I was in business!

It was a pretty simple recovery.  I had to unmount and remount the LVs becuase CentOS Live CD mounted them read-only, but that was easy enough – the device mapper nodes were all there and sensibly named too, so it was pretty intuitive.  The I just had to edit the LV names in /boot/grub/grub.conf, /etc/fstab and /etc/mtab.  For good measure I removed the duplicate entry in /etc/blkid/blkid.tab too.

Reboot, pull out the USB drive at the BIOS screen and a few moment later, there was my host back in the land of the serving. Phew!

A lucky escape this time, but it shows the risk of unintended consequences.  The sooner I get KVM sorted out, and all the real work done in VM guests in LVs, the better, because at least then I’ll be able to make a snapshot before I try anything like a sysadmin task!