OK, OK – I was stupid.
As part of my ongoing move from Xen to KVM for VM hosting, I fireed up a server that had been sitting about too long switched off under the desk, thinkig that it would make an ideal machine to try KVM out on. A yum groupremove Virtualization seemed to produce no issues, so I was looking good to go with a bit fo repo cleanup and then an update before upgrading from CentOS 5.2 to 5.3 with yum. Then I did something stupid.
I renamed the logical volume that / is mounted from, becuase I thought the name wasn’t descriptive enough. Then I rebooted the box. Oh yes.
Cue a kernel panic when grub couldn’t find the root partition. Oh Chris, how SMRT! The physical box doesn’t have a CD/DVD drive (it’s meant to be a headless box) but luckily it does have USB ports, and I had a spare screen and keyboard about, and I happeend to have a 4GB USB thumb drive in my laptop bag. A quick Google search turned up Pendrivelinux.com and a handy guide to getting CentOS 5 working on USB. I already had a CentOS 5.2 Live CD image on my hard drive, so a quick upload of the thumb drive contents, a quick download of the Fedora LiveUSB Creator and I was in business!
It was a pretty simple recovery. I had to unmount and remount the LVs becuase CentOS Live CD mounted them read-only, but that was easy enough – the device mapper nodes were all there and sensibly named too, so it was pretty intuitive. The I just had to edit the LV names in /boot/grub/grub.conf, /etc/fstab and /etc/mtab. For good measure I removed the duplicate entry in /etc/blkid/blkid.tab too.
Reboot, pull out the USB drive at the BIOS screen and a few moment later, there was my host back in the land of the serving. Phew!
A lucky escape this time, but it shows the risk of unintended consequences. The sooner I get KVM sorted out, and all the real work done in VM guests in LVs, the better, because at least then I’ll be able to make a snapshot before I try anything like a sysadmin task!
Recent Comments