I’ve been wrestling with squid today, and although I have inky stains on my body and soul, I’ve made some headway. I’ve been trying to be a good sysadmin and not just disable selinux, or set it to permissive and forget it, at the first sign of trouble. Thing is, I wanted a non-default squid configuration, and selinux got in the way. I had two problems that I had to solve – non standard listen ports, and non-standard cache location.

I installed squid with the usual yum install squid command on my CentOS 5.4 server. I want to be able to use a single squid instance to support both configured browsers (on the basis that it’s better if browsers know they are being proxied) and transparent (sometimes called “intercept”) proxying for applications with their own http access libraries, and guest machines on my network that may not want to configure my proxy settings, or don’t respect autoconfigure via DHCP. So, after using webmin to set up a few basics, I jumped into the /etc/squid/squid.conf file and set up the following:
# Listen on internal address for VLAN 2
http_port 10.254.225.65:3128
http_port 10.254.225.65:3129 transparent

# Listen on internal address for VLAN 3
http_port 10.254.225.129:3128
http_port 10.254.225.129:3129 transparent

I then used Webmin to create a new logical volume on my RAID array, a nice healthy 64GB. Big for a cache, but I want to try and cache installers and downloaded updates, so I want plenty of space. I created the logical volume, mounted it on /webcache and jumped back into squid config in Webmin and added a cache location /webcache of 5000MB, 16 1st-level folders and 256 2nd-level folders.

Squid failed to start. OK, I thought, I’ll initialise the cache with squid -z. That ran fine as root. Squid still wouldn’t start. A check of the log files showed that squid was complaining “cache_dir /webcache: (13) Permission denied“. Of course, I hadn’t set permissions on /webcache, so a quick chown -R squid:squid /webcache and I thought I was done. I was not – squid still wouldn’t start.

I realised selinux might be the culprit and checked this by running setenforce Permissive and lo and behold, squid started. A review of /var/log/audit/audit.log suggested that there were two problems – the non-standard port 3129 and the non-standard cache location.

The port issue was relatively easy to fix. semanage port -l | grep "cache" to list any selinux information about ports that cache processes are allowed to use suggested the following:
http_cache_port_t tcp 3128, 8080, 8118, 11211, 10001-10010
http_cache_port_t udp 3130, 11211

So, I needed to add port 3129 to the list. A quick Google found a note about adding non-standard Apache ports, and from that is was easy to work out that the magic line would be semanage port -a -t http_cache_port_t -p tcp 3129 which adds port 3129 to the list of allowed ports for http caches. I didn’t expect this to get squid running, but at least the next failure gave me a smaller set of audit log entries to work on!

Next, I needed to fix the file contexts for /webcache. This was a little more complex, being a novice to selinux. I discovered that I could view existing context with ls -Z so at least I could assess my efforts. I also found that it’s possible to change the file context with chcon, but this may not survive reboots, a restorecon (restore contexts) command or a “relabel” of the filesystem. I needed a way to make sure that selinux knew that /webcache was meant to be used for squid, so I disn’t have to keep telling it.

After some poking around, and trying a few things out to no avail, I found that I needed to use a command grep -hi "squid" /etc/selinux/targeted/contexts/files/file_contexts* which showed me where the selinux policy knew about allowable locations for squid to keep files, as follows:
/etc/squid(/.*)? system_u:object_r:squid_conf_t:s0
/var/log/squid(/.*)? system_u:object_r:squid_log_t:s0
/var/squidGuard(/.*)? system_u:object_r:squid_cache_t:s0
/var/spool/squid(/.*)? system_u:object_r:squid_cache_t:s0
/usr/share/squid(/.*)? system_u:object_r:squid_conf_t:s0
/var/cache/squid(/.*)? system_u:object_r:squid_cache_t:s0
/var/log/squidGuard(/.*)? system_u:object_r:squid_log_t:s0
/usr/sbin/squid -- system_u:object_r:squid_exec_t:s0
/var/run/squid\.pid -- system_u:object_r:squid_var_run_t:s0
/usr/lib/squid/cachemgr\.cgi -- system_u:object_r:httpd_squid_script_exec_t:s0
/usr/lib64/squid/cachemgr\.cgi -- system_u:object_r:httpd_squid_script_exec_t:s0

So, it seems all I needed to do was to add to that list, and all would be good. There was a small wrinkle to this too though. semanage fcontext -a -t squid_cache_t /webcache would seem to do the necessary magic, but in fact semanage fcontext doesn’t work like a regular filesystem tool, so there’s no recursion. It didn’t seem “clean” to then iterate through each sub-folder (all 4096 of them!) and set the context.

The answer was actually simple. Delete the cache folders with rm -rf /webcache/0* and rm -rf /webcache/swap* and then set the required context on /webcache in the policy with semanage fcontext -a -t squid_cache_t /webcache and finally make this the current context with restorecon -R /webcache. Then, with context set in the parent folder, running squid -z meant that the 4096 cache directories were also created with the right context.

I re-enabled Enforcing mode, squid started and my selinux policy is aware of my non-standard configuration, so hopefully future updates won’t break it. My small concern is that future updates to the selinux policy may overwrite my changes – I don’t know how selinux works well enough to know if my policy changes will survive or be overwritten. At least I’ve got my own handy guide to how to make it work again if I need it…

OK, OK – I was stupid.

As part of my ongoing move from Xen to KVM for VM hosting, I fireed up a server that had been sitting about too long switched off under the desk, thinkig that it would make an ideal machine to try KVM out on.  A yum groupremove Virtualization seemed to produce no issues, so I was looking good to go with a bit fo repo cleanup and then an update before upgrading from CentOS 5.2 to 5.3 with yum.  Then I did something stupid.

I renamed the logical volume that / is mounted from, becuase I thought the name wasn’t descriptive enough.  Then I rebooted the box.  Oh yes.

Cue a kernel panic when grub couldn’t find the root partition.  Oh Chris, how SMRT!  The physical box doesn’t have a CD/DVD drive (it’s meant to be a headless box) but luckily it does have USB ports, and I had a spare screen and keyboard about, and I happeend to have a 4GB USB thumb drive in my laptop bag.  A quick Google search turned up Pendrivelinux.com and a handy guide to getting CentOS 5 working on USB.  I already had a CentOS 5.2 Live CD image on my hard drive, so a quick upload of the thumb drive contents, a quick download of the Fedora LiveUSB Creator and I was in business!

It was a pretty simple recovery.  I had to unmount and remount the LVs becuase CentOS Live CD mounted them read-only, but that was easy enough – the device mapper nodes were all there and sensibly named too, so it was pretty intuitive.  The I just had to edit the LV names in /boot/grub/grub.conf, /etc/fstab and /etc/mtab.  For good measure I removed the duplicate entry in /etc/blkid/blkid.tab too.

Reboot, pull out the USB drive at the BIOS screen and a few moment later, there was my host back in the land of the serving. Phew!

A lucky escape this time, but it shows the risk of unintended consequences.  The sooner I get KVM sorted out, and all the real work done in VM guests in LVs, the better, because at least then I’ll be able to make a snapshot before I try anything like a sysadmin task!

I have a CentOS 5.3 (actually a CentOS 5.2 build upgraded in yum, but that’s not terribly important) x86_64 machine that is in fact a Xen hypervisor kernel with said CentOS 5.3 running as the dom0 (service VM), and currently four guest VMs also running CentOS 5.2 x86_64, upgraded to CentOS 5.3 with yum.  This was a kind of interesting project in itself, in that I used the Gitco repos in order to get Xen 3.3.1 on the CentOS build, and I had to do some hackery of the Xen and stock CentOS networking scripts to make things work.  The box doesn’t use Xen network scripts and xenbr0, because they broke pretty badly with bonded dual NICs.  It has a fairly standard Ethernet bond setup going, then a patched RH network script so bridges don’t init before the bond, a standard Linux bridge br0, which all my Xen vifs attach to, and a virbr0 which doesn’t have any physical NICs attached, but does have an IP address and dhcpd running against it, so I have a place to quickly fire up new VM guests without having to assign network addresses.

This complexity lead to a few problems, partly during installation, when I ended up temporarily having to go wiht a non-Xen kernel in order to update bits and ipeces and set up repos so I could go back to a Xen kernel, and when yum wanted to update to kernel-xen-2.6.18-128.1.10.el5 and like a trusting fool I let it, and promptly had no running VMs on the next reboot.  Hacking grub back to 2.6.18-128.1.6.el5xen seemed to sort it, and it never got changed from there.

Actually, to tell the trust, I also have another machine built a similar way, sans CentOS 5.3 upgrade.  This one was meant to potentially be a home to a windows machine, or most likely a Linux MCE installation running in  a non-PV guest.  That one didn’t get installed yet, and has been sitting under my desk for many a month, waiting for cabling.  Oops.

My guest DomUs are living in Xen virtual disks which are inside LVM logical volumes as far as the dom0 is concerned.  If I’d been smarter, I might have set up three LVs for each domU – one for /boot, one swap and one for /, so that I cold more easily manage them from not only the guest domU, but also the dom0 if the domU was offline for any reason.

The increased exposure KVM is getting in RHEL 5.4 is making me wonder if all this Xen tinkering is worth it.  My machine only runs Linux guests, and then only really for a bit of process separation security.  Given that the dom0 has to do a bit of service work for guest domUs anyway, and it seems that this involves a bit of context switching overhead, I don’t see that doing the hypervisor function in the kernel is going to be a problem for me, especially on that currently unused box, which is going to be a home all-in-one media, file/print, NATting, caching and bandwidth arbitrating machine.  As I was going to just accept doing all the networking for that (including the firewall/NAT/bandwidth and some VPN stuff) in the dom0 instead of making the Xen networking even more complicated with dual virtual NICs and a virtual firewall box, I guess just making all that work in plain old Linux and then loading up a kernel module when I want a media centre guest VM (yes, I know I could just be less lazy and make all the bits of Linux MCE work under CentOS natively, but starting a VM and installing an OS-up image for LMCE seems like a good thing) installing makes a bit of sense.

The currently unused machine isn’t so bad – as it currently has no VMs and didn’t get the 5.2/5.3/Gitco hybrid horror treatment, I could just strip Gitco repo, group uninstall Virtualisation (sorry Virtualization), and boot it as a regular Linux kernel, do the networkingy bits (and make better use of my rather rubbish 1.5Mb/s of patchy ADSL) and then worry about that media server when the time is right.  The dual-NIC networkign will be far easier – one outside one inside, attach a separate bridge to each and I could then attach VMs to the “public” or “private” bridges and the host OS will iptables them like any other hosts.  Great!

It’s that first server that is going to be ‘interesting’.  Fedora’s virt-v2v is a ways off yet, so I’ve got to think of smart ways of moving painlessly from Xen to KVM without losing services for long.  The current outline plan goes something like:

1. Install a plain old Linux kernel into each VM guest
2. Install a plain old Linux kernel into the current dom0
3. Make new LVM logical volumes for guest VMs – possibly three each for /boot, swap and /
4. Modify guest VM grub config to default the plain old Linux kernel
5. Shutdown guest VMs
6. Reboot dom0 as a plain old Linux kernel, albeit with KVM support loaded
7. Make any networking changes required
8. Create new KVM virtual machine images in the logival volumes
9. Stop the new KVM guests
10. Mount guests Xen virtual ‘disks’ (actually just some partitions I think, not full bootable disk images) on loopback in the dom0
11. dd the contents of the Xen partitions into the new LVM logical volumes, overwriting the guest images created at step 7
12. Start the KVM guests
13. Panic when it goes wrong (had to be step 13, didn’t it?)

I’m hoping that the fact I already do the networking outside of Xen, and just use standard Linux bridging on and the libvirt virbr0, will mean a slightly easier migration – all I have to do is get my instances booting off a standard kernel instead of a Xenified one, and inside a different virtual disk format.

I’ll still need my standard-breaking networking patch to enable bonds before bridges though.  Hopefully RH fix that one in RHEL 5.4 so CentOS picks it up in the next release.