Skip to content

Category: Linux

Live P2V migration, on the cheap

Last week I performed a live physical to virtual migration for a customer. Ghetto style.
I was actually surprised by its simplicity and how well it turned out in the end.

Basically what I did was boot the target server to a live environment in order to be able to have raw access to the disk. In this case I’ve used FAI’s Sysinfo mode, but booting any Linux Live CD would have worked equally well. After it’s booted I’ve created the the necessary partitions, created filesytems and mounted them. When those steps were done it was time to sync data and make the virtual system bootable.

Usually I would advise against this kind of migration but in this case there was only static data involved.
It worked out well for me, but your mileage may vary.

Alright here’s what I did step by step…

1. Backup partition layout and MBR

source:~ # sfdisk -d /dev/sda > /tmp/source-partitions
source:~ # dd if=/dev/sda of=/tmp/source-mbr.img bs=512 count=1
source:~ # scp /tmp/source-* target:/tmp/

2. Restore partition layout and MBR

target:~ # dd if=/tmp/source-mbr.img of=/dev/sda bs=446 count=1
target:~ # sfdisk /dev/xvda < /tmp/source-partitions

3. Create a filesystem on each partition

target:~ # for partition in xvda1 xvda6 xvda7 xvda8 xvda9 xvda10; do mkfs.ext4 /dev/$partition; done

4. Create a swap space

mkswap /dev/xvda2
swapon /dev/xvda2

5. Mount the newly created partitions

target:~ # mount /dev/xvda1 /mnt/tmp/
target:~ # mount /dev/xvda6 /mnt/tmp/var/
target:~ # mount /dev/xvda7 /mnt/tmp/tmp/
target:~ # mount /dev/xvda8 /mnt/tmp/usr
target:~ # mount /dev/xvda9 /mnt/tmp/home/
target:~ # mount /dev/xvda10 /mnt/tmp/data/

6. Synchronise data from source to target

target:~ # for i in $(seq 1 10);
rsync -av --delete --progress \
--exclude=/dev \
--exclude=/sys \
--exclude=/proc \
--exclude=/mnt/ \
--exclude=/media/ \
source:/* /mnt/tmp/;
target:~ #

Repeat this step until you’re confident nothing has changed anymore.

7. Enter the synchronized environment

target:~ # mount -t proc none /mnt/tmp/proc
target:~ # mount --rbind /dev /mnt/tmp/dev

target:~ # chroot /mnt/tmp /bin/bash
chroot:~ # source /etc/profile

8. Fix fstab entries

(only for Xen vm’s)

chroot:~ # sed -i 's+/dev/sda+/dev/xvda+g' /etc/fstab
chroot:~ # vim /etc/fstab

9. Fix network settings

chroot:~ # vim /etc/network/interfaces

10. Make the system bootable

chroot:~ # grep -v rootfs /proc/mounts > /etc/mtab
chroot:~ # grub-install --no-floppy /dev/xvda
chroot:~ # update-grub

Once you’ve run the above commands it’s time to exit the chroot, unmount all partitions and reboot
into your migrated environment. Enjoy!

“Dynamic lookup is deprecated. Support will be removed in Puppet 2.8.” when using facts in templates

Recently ran into some of these when running Puppet 2.7.x:

warning: Dynamic lookup of $variable is deprecated. Support 
will be removed in Puppet 2.8.  Use a fully-qualified variable
name (e.g., $classname::variable) or parameterized classes.

The solution was pretty obvious and it’s easy to fix because the solution is mentioned in the warning. However the warning fails to mention what to do with Facter facts inside ERB templates. Figuring out how to fix this took me quite a while although it’s easy too.

Let’s start by taking a look at this snippet:

NameVirtualHost <%= 'ipaddress' %>:80
< VirtualHost <%= 'ipaddress' %>:80 >
	DocumentRoot /var/www/vhosts/

If we’d apply this template in a manifest using Puppet 2.7.x we would run into the warning mentioned above because the IP address fact is out of scope. In order to avoid this we use lookupvar for Facter facts:

NameVirtualHost <%= scope.lookupvar('ipaddress') %>:80
< VirtualHost <%= scope.lookupvar('ipaddress') %>:80 >
	DocumentRoot /var/www/vhosts/

Fixing strange device names using Ubuntu templates on VMware ESX or vSphere

If you regularly deploy Ubuntu VM templates on your VMware ESX(i) or VMware vSphere boxes you will probably run into strange network device numbers. This is caused by a udev rule. This problem has been confirmed to exist in Ubuntu 9.04, 9.10 and 10.04. I haven’t had the time to check out other versions of Ubuntu. It’s also still existing in RHEL 6 and Scientific Linux 6.

As you can see below we have two ethernet devices: eth4 and eth5 instead of the usual eth0
and eth1.

root@box:~# ifconfig | grep Link
lo        Link encap:Local Loopback  
eth4  Link encap:UNSPEC  HWaddr  
eth5  Link encap:UNSPEC  HWaddr  

Lucky for us it’s very simple to persistenly assign the correct device names to the corresponding mac address.

root@box:~# grep eth4 /etc/udev/rules.d/70-persistent-net.rules 
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", 
ATTR{address}=="00:11:22:33:44:55", ATTR{dev_id}=="0x0", 
ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

Use your favorite editor to edit /etc/udev/rules.d/70-persistent-net.rules.
Replace NAME=”eth4″ with NAME=”eth0″ and do the same with eth5.
Save the config file, reboot and you’re done!

After writing this guide I’ve also found this issue to exist on other udev-based distro’s (e.g. OpenSUSE) and other VMware products too (e.g. Fusion and Workstation). The same fix applies, so no worries there.

Shared hosting attack mitigation, part 1: Apache MPM-ITK

When it comes to shared hosting we often get to see the same patterns and CMS installs over and over again. We have the big guns: Joomla, WordPress, Drupal, Typo3. And then there are smaller guys like e107, eZ Publish, XOOPS. Occasionally we run into TinyCMS, MiaCMS or even a YupiCMS install. They all have several things in common: they all try to be user friendly and easy to set-up. They’ll make you have a website online in a matter of hours or even minutes. So far, so good.

But what about patching? People tend to neglect to keep their installs up-to-date. We often get to hear people are afraid to break their corporate website or they rely on the web agency who built their website. However most of these agencies don’t want to take the risk either, unless there’s a hefty monthly support fee involved. And even when there is one they don’t always get around to do so.

This might become a serious problem for anyone running (on) a shared hosting server. XSS attacks, SQL injection and other kinds of intrusion might lead to collateral damage. Other people’s sites just might get attacked too. By default any vHost is readable and more importantly writeable by the user that is used to run PHP scripts. In most cases this is the Apache user.

If we can run our hosted PHP scripts using a separate user for each vHost we should be able to mitigate a large amount of common attacks towards other vHosts! However it isn’t feasible to run each hosting inside a separate chrooted environment. It would take too much time to set-up and it’s rather complex to patch it.

This is where Apache MPM-ITK comes into play. It’s an MPM (Multi-Processing Module) that is able to use a separate uid/guid per vHost without the need for a separate chroot for each hosting.

Enough chit-chat. Let’s get to it.

Install a default LAMP stack:

# apt-get install apache2-mpm-itk apache2-utils apache2.2-common 
defoma fontconfig-config libapache2-mod-php5 libapr1 libaprutil1 
libexpat1 libfontconfig1 libfreetype6 libgd2-xpm libjpeg62 
libltdl3 libmcrypt4 libpq5 libt1-5 libxpm4 openssl ssl-cert 
openssl-blacklist php5-common php5-cli php5-gd php5-mcrypt 
php5-mysql ttf-dejavu ttf-dejavu-core ttf-dejavu-extra

If you’re already running a LAMP stack it’s easy to replace the default MPM:

# apt-get remove apache2-mpm-prefork
# apt-get install apache2-mpm-itk

Configuring MPM ITK isn’t exactly brain surgery. Simply add an IfModule statement to your existing vHost config or use the following example to create a new one:

# cat /etc/apache2/sites-available/vhost-example.conf
<VirtualHost *:80>
  <IfModule mpm_itk_module>
    AssignUserId example example

  DocumentRoot /var/www/vhosts/example/public_html/

  CustomLog /var/www/vhosts/example/logs/access_log combined
  ErrorLog /var/www/vhosts/example/logs/error_log

  ScriptAlias /cgi-bin/ /var/www/vhosts/example/cgi-bin/
  DirectoryIndex index.php index.html index.htm

  <Directory "/var/www/vhosts/example/cgi-bin/">
    Options +ExecCGI
    AllowOverride none

Don’t forget to create the default user and appropriate folders. Change the permissions afterwards:

# useradd example
# mkdir -p /var/www/vhosts/example/
# mkdir -p /var/www/vhosts/example/public_html/
# mkdir -p /var/www/vhosts/example/cgi-bin/
# mkdir -p /var/www/vhosts/example/logs/
# chown -R example:example /var/www/vhosts/example/

Once this has been done, enable your website and test the configuration before reloading the Apache daemon:

# a2ensite vhost-example.conf
Enabling site vhost-example.conf.
Run '/etc/init.d/apache2 reload' to activate new configuration!

# apache2ctl configtest
Syntax OK

# /etc/init.d/apache2 reload
Reloading web server config: apache2.

And that’s it! Use phpinfo(); if you want to check you’re actually running on MPM ITK. It should be listed in Loaded Modules. Another way to verify is using ps or top. The PHP scripts for your newly created vHost should be running on it’s own separate user:

15059 example 20  0 24596 5624 2468 R  0.7  3.9  0:00.02 apache2
15060 example 20  0 24596 5496 2340 R  0.7  3.8  0:00.02 apache2
15061 example 20  0 24596 5412 2256 R  0.7  3.7  0:00.02 apache2
15062 example 20  0 24596 5360 2204 R  0.7  3.7  0:00.02 apache2

There is obviously a performance penalty involved when using MPM ITK.
I’ve compared both the default MPM Prefork and MPM ITK using Apache Bench and phpinfo() serving the content:

# ab -n 2000 -c 4

mpm prefork vs mpm itk

Please do bear in mind that this was tested on low power hardware (Atom 1.6GHz cpu).

Authenticated SMTP with Postfix on CentOS, the easy way

This will be more of a future reference than an actual howto. It’s far from feature complete but it will get you started on authenticated SMTP sessions using Postfix. Quick & dirty.

Installing Postfix and SASL on CentOS:

# yum install postfix
# yum install cyrus-sasl cyrus-sasl-plain cyrus-sasl-md5

Let’s move on to the configuration now. Below you will find my default template
for /etc/postfix/

# cat /etc/postfix/
smtpd_banner = $myhostname ESMTP $mail_name

myhostname = example.tld
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = example.tld
mydestination = example.tld, localhost.localdomain, localhost
transport_maps =
relayhost =
mynetworks =
#mynetworks = hash:/etc/postfix/networks

smtpd_sasl_path = sasl2/smtpd.conf
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = example.tld
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes

recipient_delimiter = +
inet_interfaces = all

smtpd_recipient_restrictions = permit_mynetworks,

Use smtpd_recipient_restrictions to make sure you’re not running an Open Relay server accepting spam from anyone. It will accept unauthenticated sessions originating from localhost only. However authenticated sessions are generally allowed.

Make sure your Postfix daemon is actually able to communicate with the sasl daemon.
To accomplish this append this to /etc/postfix/

# cat /etc/postfix/
submission inet n       -       n       -       -       smtpd
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_security_options=noanonymous
  -o smtpd_sasl_local_domain=example.tld
  -o header_checks=
  -o body_checks=
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject_unauth_destination
  -o smtpd_sasl_security_options=noanonymous,noplaintext
  -o smtpd_sasl_tls_security_options=noanonymous

Next up we’re going to add our users to the sasl database:

# saslpasswd2 -c -u $hostname $user

While we’re at it, it might be a good idea to fix permissions on the sasl database. Otherwise Postfix will be unable to read from it or write to it.

# chown postfix:postfix /etc/sasldb2
# chmod 660 /etc/sasldb2

We’re almost there. To configure the sasl daemon itself:

# cat /usr/lib/sasl2/smtpd.conf
pwcheck_method: auxprop
auxprop_plugin: sasldb
# cat /usr/lib/sasl/smtpd.conf
pwcheck_method: auxprop
auxprop_plugin: sasldb

And finally restart the daemons.

# /etc/init.d/sasld restart
# /etc/init.d/postfix restart

Be sure to confirm it’s working using both your default mail client and /var/log/maillog.


Monitoring MySQL with Munin on a DirectAdmin platform

Today I’ll be showing you how to monitor MySQL with Munin on a DirectAdmin platform. I’ve tested this setup for a customer on a CentOS box. It should be fairly easy to adapt this to Debian. You probably won’t even need to change credentials at all on a Debian box given the fact that it has an /etc/mysql/debian.cnf file by default. Although I’m not sure DirectAdmin puts it to good use. Any Debian/DirectAdmin users out there? Feel free to comment.

Let’s start off by checking the proper MySQL login credentials on our CentOS/RHEL box:

# cat /usr/local/directadmin/conf/mysql.conf

Easy enough. Let’s move on to installing munin and applying the credentials to the MySQL monitoring plugin. Munin isn’t available in the default repository. Not to worry, it’s in the Fedora Project’s EPEL repository for CentOS/RHEL. If you don’t have EPEL enabled yet be sure to check the excellent FAQ on the subject.

Or you could just move on to installing the repository.
For i386/i686:

# rpm -Uvh

For x86_64:

For x86_64: 
# rpm -Uvh

Now we can go on installing munin:

# yum install munin munin-node

Sadly this doesn’t install pull all the necessary dependencies. Not sure why the package maintainer missed out on this but it’s rather easy to fix it:

# yum install perl-Cache perl-Cache-Cache 
# yum install perl-IPC-ShareLite perl-DBD-MySQL

On to the credentials part. Edit the mysql plugin on line 132:

# vim +132 /usr/share/munin/plugins/mysql_

You should see something like this with the credentials left blank:

my %config = (
    'dsn'        => $ENV{'mysqlconnection'} || 'DBI:mysql:mysql',
    'user'       => $ENV{'mysqluser'}       || 'da_admin',
    'password'   => $ENV{'mysqlpassword'}   || 'removed',

As you can see I’ve already filled in the blanks.

Once the plugin has been configured we’re able to apply it. Before applying I’d suggest you take a look at what graphs are available:

# /usr/share/munin/plugins/mysql_ suggest

To apply all of them simply run the following:

# cd /etc/munin/plugins
# ln -sf /usr/share/munin/plugins/mysql_ mysql_
# for i in `./mysql_ suggest`; \
do ln -sf /usr/share/munin/plugins/mysql_ $i; done

If you only need a few of them you can apply them this way:

# cd /etc/munin/plugins
# ln -sf /usr/share/munin/plugins/mysql_ mysql_
# ln -sf /usr/share/munin/plugins/mysql_ mysql_bin_relay_log
# ln -sf /usr/share/munin/plugins/mysql_ mysql_commands
# ln -sf /usr/share/munin/plugins/mysql_ mysql_connections
# ln -sf /usr/share/munin/plugins/mysql_ $any_other_graph

Be sure to reload munin-node:

# /etc/init.d/munin-node restart

And that’s it. Enjoy your graphs at 🙂

Munin MySQL InnoDB graph

Using expect scripts to backup your Cisco configuration

In this short howto I’ll explain how to use expect scripts with Cisco devices. In this example I’m going to use it to backup the current running configuration.


  • A working tftp server
  • Expect
  • Lucky for us both requirements are available in all major distro’s.

    The Debian/Ubuntu way:

    sudo apt-get install tftp tftpd expect

    Next on our todo list is configuring the tftp server. This should also be fairly easy:

    # cat /etc/xinetd.d/tftp
    service tftp
        protocol        = udp
        port            = 69
        socket_type     = dgram
        wait            = yes
        user            = nobody
        server          = /usr/sbin/in.tftpd
        server_args     = /tftpboot
        disable         = no

    Restart your xinetd server when done.

    # /etc/init.d/xinetd restart

    Make sure the /tftpboot folder exists and is owned by user and group nobody:

    # chown -R nobody:nobody /tftpboot

    You should also create an empty file where you’d like to save your configuration and rerun the above command to adjust permissions.

    # touch /tftpboot/config
    # chown -R nobody:nobody /tftpboot

    You should also create an empty file where you’d like to save your configuration and rerun the above command to adjust permissions.

    # touch /tftpboot/config
    # chown -R nobody:nobody /tftpboot

    We can now test our newly configured tftpd server:
    Create a new file in your home dir called config and put some random text in it.

    # cat /home/user/config
    test 12
    # tftp
    tftp> open localhost
    tftp> put config
    Sent 146 bytes in 0.0 seconds
    # cat /tftpboot/config
    test 12

    Excellent! We’re ready to receive config files from the Cisco device.

    Below you will find an example script:

    ## TomDV
    # ---------------- configuration ---------------- #
    set device    # cisco device
    set tftp      # tftp server
    set user someuser           # username
    set pass ultrasecret        # password
    set config                  # config destination
    set timeout 60
    # -------------- do not edit below -------------- #
    spawn telnet $device
    expect "Password:"
    send "$pass\n"
    expect ">"
    send "en\n"
    expect "Password:"
    send "$pass\n"
    send "copy running-config tftp://$tftp/$config\n\n"
    expect "$tftp"
    send "\n"
    expect "$config"
    send "\n"
    send "exit\n"

    Save it anywhere you like and run it from the shell. You’ll see something like this in your logs:

    user in.tftpd[22304]: connect from (
    user tftpd[22305]: tftpd: trying to get file: config
    user tftpd[22305]: tftpd: serving file from /tftpboot

    That’s it. Your current Cisco config has been saved to /tftpboot/config.

    I wouldn’t recommend using this into production without proper firewalling. You can get the same results by using snmp. But that’s however a subject for another howto.

    Monitor DNS blacklist entries with Zabbix

    One of the smaller projects I’ve been working on lately is monitoring Realtime DNS Blacklists (RBL’s) status with Zabbix. I’m confident most of you are already familiar with RBL’s. For those who are not, here’s a small introduction shamelessly stolen from Wikipedia:

    A DNSBL (DNS-based Blackhole  List, Block List, or Blacklist; see below) is a list of IP addresses published through the Internet Domain Name Service in a particular format. DNSBLs are most often used to publish the addresses of computers or networks linked to spamming; most mail server  software can be configured to reject or flag messages which have been sent from a site listed on one or more such lists.

    And that’s exactly what we’re going to monitor. If we are listed on one of those RBL’s we’d like to know about it, don’t we? 

    Let’s get to it then. First of all we need an up to date list of RBL’s which we can use to check whether we’re listed or not. You could try the list I’m maintaining and using for my own monitoring purposes. The most recent version can be found here. It contains a whopping 92 RBL’s to get you started with.

    Now that we have an up-to-date list of common used RBL’s it’s time for some shell scripting:

    ## TomDV
    ## 2010-01-25
    cd /usr/share/zabbix/
    RBL="`cat rbl_list.txt`"
    W=$( echo ${1} | cut -d. -f1 )
    X=$( echo ${1} | cut -d. -f2 )
    Y=$( echo ${1} | cut -d. -f3 )
    Z=$( echo ${1} | cut -d. -f4 )
    for i in $RBL
        RESULT=$( host -t a $Z.$Y.$X.$W.$i 2>&1 )
        if [ $? -eq 0 ]
            #echo “The IP ADDRESS ${1} is listed at $i:\n$RESULT” ## DEBUG
            let "STATUS += 1"
        #echo $RESULT ## DEBUG
    if [ $STATUS -lt 1 ]
        echo 0
        echo $STATUS

    This script takes the IP address of your server as input.

    I’ve intentionally left the debug code inside the script. This way the output can be used right away within Zabbix. However if you’re listed on one of the blacklists you can run the script with the debug code uncommented and you get a list of all the RBL’s you’re listed in.

    I’ve put this script in /usr/share/zabbix, along with the rbl_list.txt file you can find above.

    # cat /etc/zabbix/zabbix_agent.d/rbl.conf

    I also have the following line in /etc/zabbix/zabbix_agentd.conf and /etc/zabbix/zabbix_agent.conf to load custom config files:


    And that’s about it. Let’s see if we’re listed in any of the RBL’s:

    # zabbix_agent -t rbl.mx1; zabbix_agent -t rbl.mx2;
    rbl.mx1                                    [t|0]
    rbl.mx2                                    [t|0]

    Any value above zero means you’re listed. I guess we’re safe.
    If you’re listed just uncomment the debug code. It will show you which RBL’s you’re in.

    Happy monitoring! 🙂

    Install Xen and libvirt on Debian Lenny

    This should be an easy to follow guide about how to install Xen on Debian 5. You should be able to copy/paste most parts of on your shell. Please run this only on a clean and up-to-date Debian system.

    Alright let’s get to it quick ‘n dirty.

    First of all make sure your Debian install actually is up-to-date:

    # apt-get update; apt-get upgrade

    Let’s see which Xen kernel images are available and pick the most recent one to install:

    # apt-cache search xen | grep image | awk '{print $1}'
    # apt-get install `apt-cache search xen-linux-system \
    | sort | tail -1 | awk '{print $1}'`

    Once this is done reboot your system, login again and run:

    # uname -a
    Linux elysium 2.6.26-2-xen-amd64 #1 SMP Thu Feb 11 02:57:18 UTC 2010 x86_64 GNU/Linux

    As you can see, the system is running kernel 2.6.26-2 with the xen-amd64 patch set.
    As of now we should have Dom-0 available:

    # xm list
    Name                   ID   Mem VCPUs      State   Time(s)
    Domain-0               0  3885     2     r-----      8.5


    Let’s move on to the network. By default there is no bridge available from the virtual machines towards the external network. It’s fairly easy to accomplish though:

    # vim /etc/xen/xend-config.sxp

    Look for the following line and uncomment it:

    (network-script network-bridge)

    And while we’re editing the xend-config.sxp file, change the following line:

    (xend-unix-server no)


    (xend-unix-server yes)

    Be sure to reload the new settings:

    # /etc/init.d/xend restart

    If you don’t edit this line or if you don’t reload, you obviously won’t be able to install or manage your virtual machines. You’d get to see errors like this:

    ERROR    internal error failed to connect to xend
    Traceback (most recent call last):
      File "/usr/bin/virt-install", line 693, in
      File "/usr/bin/virt-install", line 508, in main
        conn = cli.getConnection(options.connect)
      File "/var/lib/python-support/python2.5/virtinst/", line 123, in getConnection
      File "/usr/lib/python2.5/site-packages/", line 140, in open
        if ret is None:raise libvirtError('virConnectOpen() failed')
    libvirtError: internal error failed to connect to xend

    There are more options available to connect to xend.: xend-http-server, xend-tcp-xmlrpc-server, xend-unix-xmlrpc-server, xend-relocation-server.

    The reason why I chose xend-unix-server over anything else is pretty straight forward. It only listens on the Unix socket layer which doesn’t need any other networking protocol to operate. You could compare it to connecting to localhost, but without the need for a networking device (e.g. lo0).

    If you want to be able to manage this Xen server from a central node, be sure to change the appropriate management protocol. It’s beyond the scope of this howto, but I might post an howto on this subject later on this blog.

    On to installing a virtual machine!

    I for one am a big fan of abstraction layers. It makes life for a SysAdmin or DevOp so much easier if you’re able to run the same commands on different operating systems, platforms or architectures in general. That’s why I use libvirt. It’s a collection of libraries and tools that can be used to deploy virtual machines on different types of virtualization systems. Including but not limited to Xen, KVM and Qemu.

    It’s also available as a Debian package, so installing it is very straight forward:

    # apt-get install libvirt-bin virtinst

    Once this is done we can install our first virtual machine:

    # virt-install \
    --name=test-debian-install \
    --ram=1024 \
    --file-size=10 \
    --nographics \
    --paravirt \
    --file=/var/lib/xen/images/test-debian-install.img \

    It’s probably a good idea to store this as a shell script on your Xen host for future reference.
    You should see a familiar installer within seconds after invoking the command.

    Once the install has completed you should be greeted with your new virtual machine’s login prompt:

    Debian GNU/Linux 5.0 test-debian-install hvc0
    test-debian-install login:

    To exit your virtual machine’s console, simply press Ctrl-]