My server runs CentOs 5.x, and uses the Gitco repo to provide later versions of Xen. I’ve previously sorted out the issues (at least in earlier Xen and CentOS versions) around bridging guest vifs to a bonded interface (I replaced Xen entwork scripts with stock CentOS ones, which needed patching to bring bonds up before bridges). Everything’s been fine with one small caveat – any time yum updates a kernel, before rebooting, I need to go check grub.conf to make sure that it’s using the xen version from gitco not the one matching the centOS kernel version.

So, with the Xen kernel 3.4.1 from Gitco, I did a yum update which included kernel 2.6.18-164.11.1.el5xen and somehow forgot to check grub.conf before rebooting. I waited and waited, but got no response from my server.

Luckily, I have a remote console on my physical server, so I was able to investigate. I found that the dom0 startup was hanging at “Bringing up interface bond0″. I hit the virtual reset button and dropped into the grub menu. I edited the kernel line to use xen.gz-3.4.1 and thought that would be it.

No, same problem. Same thing booting kernel 2.6.18-164.10.1.el5xen as well, either on its matching Xen kernel version, or 3.4.1 from Gitco. However, I could boot dom0 into 2.6.18-164.9.1.el5xen against Xen kernel 3.4.1, albeit with terribly slow network performance and “xen_net: Memory squeeze in netback driver” logged regularly in /var/log/messages, and network access to my guests was so slow that nearly everything timed out.

A bit of research on Google suggested that adding “dom0_mem=” to the kernel line in grub.conf might help.

Sure enough, it did – it cured the “memory squeeze” messages, and it’s let me boot dom0 into kernel 2.6.18-164.11.1.el5xen on top of Xen kernel 3.4.1 as well.

My guess is that kernels after 2.6.18-164.9.1.el5xen have some change that means without a restriction on dom0 memory, something somewhere breaks, and one of the side effects is terribly slow networking, possibly only when bonded?

I have swapped out a Q8200 processor for a Q8400 processor in my home server, so I can play with KVM.  This means a BIOS update to give me the enable/disable option for Intel Virtualization [sic] Technology.  Now I’ve updated many a BIOS in my time, so it didn’t even occur to me that it might be anything other than trivial.

I hopped on Intel’s site and went to the downloads for the board.  Windows executable, raw BIOS file for a BIOS recovery function my board doesn’t even have or “OEM kit” with the BIOS file and a DOS flash utility.  Here’s where the fun starts.  The machine doesn’t have a floppy drive and didn’t (until today) have a CD/DVD drive either.  I have previously booted this machine from a ‘live’ CentOS 5 USB stick.

So, I installed an IDE CD/DVD writer while I was in the case swapping CPUs.  This shoudl make it really easy, right?  Wrong.

First, I used CD Burner XP and a downloaded FreeDOS image to make a bootable CD with an emulated 2.88MB A: floppy and the flash imaging tool on C:, which would be the CD-ROM.  This didn’t even boot FreeDOS, it just did nothing.

Next I tried my CentOS USB stick, and hacked the syslinux boot menu to add a new FreeDOS item, using memdisk as kernel and a 1.44MB FreeDOS image as the initrd.  This gave me a working FreeDOS environment with C: on the USB stick.  Great!  So, I added the Intel flash tools and booted it up.  The Intel flash tool loaded, let me pick the BIOS image and then promptly told me the update had failed.  Trying to run the flash utility a second time hung the box.

Back to CD, this time with a CD-RW so I’m not making coasters.  I used InfraRecorder to make a bootable CD with the FreeDOS image that booted fine fron syslinux.  I added the flash utility and burnt the CD.  This started to boot FreeDOS but then threw an “Invalid Opcode” error and hung.

I thought I’d try a ‘real’ MS-DOS 6.22 and found one on  bootdisk.com, which turned out to be a Windows executable that requires a floppy disk.

I found the flashrom project, which looks great, but no record of my board on the site, and this is just flashing a BIOS – surely it can be done simply?

Well, it can, sort of.  I went back to my trusty USB CentOS and downloaded the FreeDOS diskette image that had worked previously.  I mounted it up on a loopback and removed anything not needed to boot FreeDOS to make space for the Intel utility and BIOS file, which I then copied in.  I unmounted the floppy image, used mkisofs to make a bootable .iso image and used cdrecord to blank my CD-RW and burn the image to it.

Success!  A bootable FreeDOS CD, and a working flash utility!  All it took was buying a CD/DVD drive, having available a USB stick, a CentOS Live CD image, the Fedora Live USB creator, a FreeDOS image, the Intel FLASH files and several hours trying various combinations until one worked.

Intel used to also supply .iso file downloads once upon a time, so you could just download, burn and boot.  What happened to that?

Intel people – if you’re going to only provide a ‘bare’ DOS flasher and a Windows application, and make life more difficult for non-Windows or DOS users, can I ask that you please, please give a little bit of software engineering time to help the flashrom team get more Intel flash updates supported, to make your boards more attractive to Linux users.  Please?

OK, OK – I was stupid.

As part of my ongoing move from Xen to KVM for VM hosting, I fireed up a server that had been sitting about too long switched off under the desk, thinkig that it would make an ideal machine to try KVM out on.  A yum groupremove Virtualization seemed to produce no issues, so I was looking good to go with a bit fo repo cleanup and then an update before upgrading from CentOS 5.2 to 5.3 with yum.  Then I did something stupid.

I renamed the logical volume that / is mounted from, becuase I thought the name wasn’t descriptive enough.  Then I rebooted the box.  Oh yes.

Cue a kernel panic when grub couldn’t find the root partition.  Oh Chris, how SMRT!  The physical box doesn’t have a CD/DVD drive (it’s meant to be a headless box) but luckily it does have USB ports, and I had a spare screen and keyboard about, and I happeend to have a 4GB USB thumb drive in my laptop bag.  A quick Google search turned up Pendrivelinux.com and a handy guide to getting CentOS 5 working on USB.  I already had a CentOS 5.2 Live CD image on my hard drive, so a quick upload of the thumb drive contents, a quick download of the Fedora LiveUSB Creator and I was in business!

It was a pretty simple recovery.  I had to unmount and remount the LVs becuase CentOS Live CD mounted them read-only, but that was easy enough – the device mapper nodes were all there and sensibly named too, so it was pretty intuitive.  The I just had to edit the LV names in /boot/grub/grub.conf, /etc/fstab and /etc/mtab.  For good measure I removed the duplicate entry in /etc/blkid/blkid.tab too.

Reboot, pull out the USB drive at the BIOS screen and a few moment later, there was my host back in the land of the serving. Phew!

A lucky escape this time, but it shows the risk of unintended consequences.  The sooner I get KVM sorted out, and all the real work done in VM guests in LVs, the better, because at least then I’ll be able to make a snapshot before I try anything like a sysadmin task!

OK, so I’m mulling the move from Xen to KVM.  This initially looked like I’d have to make duplicate virtual disks in KVM for my Xen guests, then shut the guests down, block copy content across and patch up MBR and boot my physical box into a stock kernel and hope KVM comes up with the goods.

Well, a little bit of experimentation with a rarely-used VM and I’ve found much to my joy that because I used LVM logical volumes to create my Xen guests virtual disks, I can use kpartx to add device mapping for them and (when the guest is safely shut down!) mount them up in my dom0 and read/write to them.  Thi sis good news because it means they have a full virtual disk setup, with a /boot partition, a root partition and a swap partition all living in a single logical volume.  This means KVM should pick them straight up and work with them unmodified.

Which hopefully means all I have to do is set a stock kernel to be the boot default in my guests, shut them down, set my dom0 to boot from a stock kernel, load up the KVM kernel module and add some guests from the logical volumes.

If it’s that easy, I’ll be a happy person indeed!

I have a CentOS 5.3 (actually a CentOS 5.2 build upgraded in yum, but that’s not terribly important) x86_64 machine that is in fact a Xen hypervisor kernel with said CentOS 5.3 running as the dom0 (service VM), and currently four guest VMs also running CentOS 5.2 x86_64, upgraded to CentOS 5.3 with yum.  This was a kind of interesting project in itself, in that I used the Gitco repos in order to get Xen 3.3.1 on the CentOS build, and I had to do some hackery of the Xen and stock CentOS networking scripts to make things work.  The box doesn’t use Xen network scripts and xenbr0, because they broke pretty badly with bonded dual NICs.  It has a fairly standard Ethernet bond setup going, then a patched RH network script so bridges don’t init before the bond, a standard Linux bridge br0, which all my Xen vifs attach to, and a virbr0 which doesn’t have any physical NICs attached, but does have an IP address and dhcpd running against it, so I have a place to quickly fire up new VM guests without having to assign network addresses.

This complexity lead to a few problems, partly during installation, when I ended up temporarily having to go wiht a non-Xen kernel in order to update bits and ipeces and set up repos so I could go back to a Xen kernel, and when yum wanted to update to kernel-xen-2.6.18-128.1.10.el5 and like a trusting fool I let it, and promptly had no running VMs on the next reboot.  Hacking grub back to 2.6.18-128.1.6.el5xen seemed to sort it, and it never got changed from there.

Actually, to tell the trust, I also have another machine built a similar way, sans CentOS 5.3 upgrade.  This one was meant to potentially be a home to a windows machine, or most likely a Linux MCE installation running in  a non-PV guest.  That one didn’t get installed yet, and has been sitting under my desk for many a month, waiting for cabling.  Oops.

My guest DomUs are living in Xen virtual disks which are inside LVM logical volumes as far as the dom0 is concerned.  If I’d been smarter, I might have set up three LVs for each domU – one for /boot, one swap and one for /, so that I cold more easily manage them from not only the guest domU, but also the dom0 if the domU was offline for any reason.

The increased exposure KVM is getting in RHEL 5.4 is making me wonder if all this Xen tinkering is worth it.  My machine only runs Linux guests, and then only really for a bit of process separation security.  Given that the dom0 has to do a bit of service work for guest domUs anyway, and it seems that this involves a bit of context switching overhead, I don’t see that doing the hypervisor function in the kernel is going to be a problem for me, especially on that currently unused box, which is going to be a home all-in-one media, file/print, NATting, caching and bandwidth arbitrating machine.  As I was going to just accept doing all the networking for that (including the firewall/NAT/bandwidth and some VPN stuff) in the dom0 instead of making the Xen networking even more complicated with dual virtual NICs and a virtual firewall box, I guess just making all that work in plain old Linux and then loading up a kernel module when I want a media centre guest VM (yes, I know I could just be less lazy and make all the bits of Linux MCE work under CentOS natively, but starting a VM and installing an OS-up image for LMCE seems like a good thing) installing makes a bit of sense.

The currently unused machine isn’t so bad – as it currently has no VMs and didn’t get the 5.2/5.3/Gitco hybrid horror treatment, I could just strip Gitco repo, group uninstall Virtualisation (sorry Virtualization), and boot it as a regular Linux kernel, do the networkingy bits (and make better use of my rather rubbish 1.5Mb/s of patchy ADSL) and then worry about that media server when the time is right.  The dual-NIC networkign will be far easier – one outside one inside, attach a separate bridge to each and I could then attach VMs to the “public” or “private” bridges and the host OS will iptables them like any other hosts.  Great!

It’s that first server that is going to be ‘interesting’.  Fedora’s virt-v2v is a ways off yet, so I’ve got to think of smart ways of moving painlessly from Xen to KVM without losing services for long.  The current outline plan goes something like:

1. Install a plain old Linux kernel into each VM guest
2. Install a plain old Linux kernel into the current dom0
3. Make new LVM logical volumes for guest VMs – possibly three each for /boot, swap and /
4. Modify guest VM grub config to default the plain old Linux kernel
5. Shutdown guest VMs
6. Reboot dom0 as a plain old Linux kernel, albeit with KVM support loaded
7. Make any networking changes required
8. Create new KVM virtual machine images in the logival volumes
9. Stop the new KVM guests
10. Mount guests Xen virtual ‘disks’ (actually just some partitions I think, not full bootable disk images) on loopback in the dom0
11. dd the contents of the Xen partitions into the new LVM logical volumes, overwriting the guest images created at step 7
12. Start the KVM guests
13. Panic when it goes wrong (had to be step 13, didn’t it?)

I’m hoping that the fact I already do the networking outside of Xen, and just use standard Linux bridging on and the libvirt virbr0, will mean a slightly easier migration – all I have to do is get my instances booting off a standard kernel instead of a Xenified one, and inside a different virtual disk format.

I’ll still need my standard-breaking networking patch to enable bonds before bridges though.  Hopefully RH fix that one in RHEL 5.4 so CentOS picks it up in the next release.

In memory of WordPress Backup Week four years ago, I thought I’d knock up a little script to backup my WordPress database and file content.

You might be wondering why I’d want to script it, seeing as there’s plenty of Wordpress plugins to backup your Wordpress content from the safe confines of WordPress administration. 

Well, my thinking was that anything that is activated by a plugin in WordPress is fine if WordPress is working.  If it is completely out-of-action, you’re going to have to recover your backup from the command line.  I’m comfortable with the command line and a bit of (low quality) shell scripting, so why not roll my own simple backup that grabs all the file and database content and makes a backup file out of it?  Besides, while the Wordpress Mobile Pack gives you the administration essentials, it’s not great at handling plugins, and I quite like the idea of being able to perform an emergency restore with nothing but a Smartphone or PDA with a copy of Pocket PuTTY installed on it.

There’s advice on WordPress backups on the Codex, which makes a good starting point.  I also found some old script examples here.  Time to do a bit of modifying!  I liked the original idea of having a 7-day rolling backup set to prevent disk space being overrun, but I also liked the idea of backups stamped with the date they were taken, to ease any restore.  So, I decided that it would be simple to adjust the script to still create 7 day-of-week folders and just store the backed-up content in a compressed tar archive using the date taken as the filename.

There were a few more changes I made to the code.  While I appreciate the ‘clean’ approach of putting usernames and passwords in a separate file, anyone finding the script would soon see where to look to find those details, so it’s not adding much in the way of security.  I also made a few tweaks to the MySQL commands used to get the table listing.

Here’s the code I got to:

#!/bin/bash # MYSQL Backup Script
# Contains portions of code and ideas from http://blogs.linux.ie/xeer/2005/06/28/simple-mysql-backup/
# and http://ocaoimh.ie/simple-mysql-backup/
# Get day of week to keep 7 rolling backups sequence, and date to name tgz files for ease of identification export d=$(date +%Y-%m-%d) export s=$(date +%u)
# Set up the paths that we're going to use for source and backup
export wppath=/var/www/html
export savepath=/var/www/backup
export tmppath=/tmp/wpback-$s-$d
# The wp database to backup
export db=wp_myblog
# Username and password for the wp database - SUGGEST A READ-ONLY USER IS USED!
export wpuser=wp_backupuser
export wppass=MyP4sswordG0esH3re
# Tell syslog and any interative user that we're running
logger "Wordpress backup script backing up $wppath to $savepath/$s/$d.tgz"
echo "Wordpress backup script backing up $wppath to $savepath/$s/$d.tgz"
# Make the temporary backup path
mkdir -p $tmppath
echo "Dumping database: $db..."
for tab in `mysql --user=$wpuser --password=$wppass -s -e "show tables;" $db`;
do
echo "Dumping table: $tab"        
mysqldump --user=$wpuser --password=$wppass --add-drop-table --allow-keywords -q -a -c $db $tab > $tmppath/$tab.sql
done
echo "Archiving Wordpress file content..."
tar -cpf $tmppath/wp-files.tar $wppath/
# Clear and remake the backup path
rm -rf $savepath/$s
mkdir -p $savepath/$s
echo "Compressing database and files into $savepath/$s/$d.tgz"
tar -czf $savepath/$s/$d.tgz $tmppath/*
echo "Deleting temporary files"
rm -rf $tmppath
echo "Wordpress backup complete"

 So, what does the code do?  Well, first the date tool is used to obtain the day of week and date string, then a few user-modifiable strings define where to back up Wordpress content from, where to make temporary files and where to save the backup archive.  The script quickly informs any interative user, and the system log, that it is running then we get to the important bit.

The MySQL command-line client is called, specifying the username and password, using the -s parameter to tell the client to output a simple listing with one result per line, and the -e parameter with the command to execute, in this case “show tables” and finally the database to execute the command against.  This fills the tab script variable with a list of tables to back up.

A temporary location is created to store the database tables and files until the final backup file is created.

Next, the MySQL dump client is called once for each table, again supplying username and password, and with parameters indicating that the dump should create a file that deletes any existing table before inserting data, and ignores any reserved words it sees, and stores the results for each table in a text file in the temporary location.

Then, the Wordpress content directory is archived using the tar utility, specifying to preserve premissions attributes.  This backup is not compressed, as the script then goes on to clear any existing day-of-week directory, and creates the final backup archive in there as a compressed tar archive containing all the SQL table dumps and the single WordPress file archive just created.

Finally, the script clears out the temporary location it created.

To use the script, it needs to be downloaded to a place outside of your WordPress content area (so it can’t be viewed by visitors to your web site), and given suitable permissions to run, with a command such as such as “chmod 744 wp_backup.sh”.

The lines at the top of the script that define wppath (WordPress files location), savepath (place to put the day-of-week directories), tmppath (place to put temporary files), db (MySQL database to backup), wpuser (MySQL user account for backing up) and finally wppass (MySQL user password) will need to be edited to suit the local WordPress and MySQL installations.

Next the MySQL user account needs to be created.  For a little bit of extra security, I recommend using a dedicated user account with read-only access to the database.  This is easy to create.  Simply start the MySQL client with the -p parameter (type mysql -p from a shell prompt) and enter you MySQL root password when prompted.  Next create a user account and grant read rights with the command GRANT SELECT, LOCK TABLES on wp_myblog.* TO ‘wp_backupuser’@'localhost’ IDENTIFIED BY ”MyP4sswordG0esH3re”;

The script will execute from the shell, and will display some progress as it goes.

To automate the process with cron, simply type crontab -e to edit the cron table, and add a line like this:

1 0 * * * /var/www/domains/thatchrisbrown.com/www/backup/wp_backup.sh

This tells cron to call the script at one minute past midnight every day.  You can verify the script has run my looking in the /var/log/messages file for the script announcing itself when run.

Disclaimer: This was written on a CentOS 5.3 box using fairly standard installs of Apache, PHP, MySQL and Wordpress.  Your environment may vary, so some elements may need adjusting to suit.  This script and its author makes no claims to fitness, usability, suitability for any specific purpose, safety or neatness, ingenuity, tidiness or prettiness of code.  You use it at your own risk.

You know when Word does stupid things, like changing the formatting of a whole paragraph when you simply edit a tiny piece of content, like a single word or sentence?  Well, it seems the Wordpress editor is just as bad.  I’ve just had to flip dozens of revisions and manually edit the HTML content to fix an issue that started when embedding an image, then editing its properties afterwards.  Wordpress started sticking post body text into the picture caption, almost randomly.

I think I’ve got it right now.  If the previous post looks odd (missing pictures, blocks of text in captions etc), do let me know and I’ll try to sort it out.

Sometimes you just can’t capitalise on an opportunity.

I’ve been working in the garden, and had a skip outside the front of the house, which had dug up turf thrown in it. I was going to photograph it under the caption “I don’t like football”.

However, I was muddy, sweaty and had dirty great boots on – not suitable for nipping in and fetching the camera! Besides, my digging assistants might have thrown me in the skip if I’d stopped to “Prat about with cameras” while they were working.

By the time we’d finished for the day, the turf was buried and the shot was lost.

The moral of the story? I’m not entirely sure – perhaps it is that you should choose helpers that understand the sudden urge to stop everything and pick up a camera!

A quick post prompted by an article in Layers magazine online.  While I agree with the spirit of the article, and applaud the use of a high-profile site to remind artists and photographers to take steps to protect their work from copyright theft, I think some of the suggested protections are counterproductive, as even the article sort of admits in places.

Take  image sizes displayed online as an example.  Average screen sizes and display resolutions are getting bigger, and an 800 x 600 picture is no longer going to fill many viewer’s displays.  While sites like Flickr will upscale a small image in the slideshow function, that upsizing is going to introduce some loss of quality.  Do you want to show your work off in a way that will likely decrease the perceived quality?

Speaking of which, over-compressing the JPEG file as an anti-theft mesaure?  Call me an old cynic if you like, but have you been to retailers like Next in the UK and had a close look at the images they sell ready-framed up as ready-to-hang artwork?  OK, they may only be £30 or so, but all the same, there’s some pretty nasty JPEG and/or upscaling artifacting going on in all of the ones I’ve seen.

So, keeping a small image size and adding JPEG compression artifacts isn’t going to put people off, not if they’ye already paying £30 for a badly-printed, poor-quality image in a wooden frame made in some cheap third-world factory.  Sad though it is, is you want some protection from abuse of your images that you put online, I think branding is the only way to add a modicum of security.

Years ago, I started branding with a small, discreet footer added to my pictures.  This started off as plain white text, in a rather nasty font, and later got slightly neater with a sans-serif font in white on a trasparent black bar, so it blended in the bottom of the image slightly more.  However, seeing examples where footers had just been cropped out by the abusers, I decided it was time to do something more difficult to erase.

I now have a Photoshop action that adds a white text footer on a 50% opaque black bar, and another that adds large 10% opaque white text right across the middle of an image.  It’s not often that an image is useful without the middle, so it’s pretty crop-proof.  This lets me still save at up to 1024 pixels, and fairly good quality levels; I tend to use Photoshop quality level 10 or the Lightroom equivalent for the web.  I’ve automated these using a droplet so I can invoke as a post-export action from Lightroom, which hopefully I’ll expand on a bit a future post. Ideally I’d like to set it up using ActionScript, so it can look at the metadata and stamp the right year on the file itself, and accomodate wildly differing image sizes automatically; another future post topic, should I ever get that working!

See some examples of this branding approach on my Flickr Photostream.  I don’t think it’s terribly intrusive, nor do I think it spoils the picture too badly.  I’d be interested to see any comments from other photographers and digital artists.