Skip to content

Penumbra Posts

Creating a bunch of Gitlab group projects using a Ruby wrapper

Just recently I needed a way to create +50 projects. Being the lazy ops guy I am, I quickly whipped this up using NARKOZ’s Ruby wrapper.

The tricky part was switching the context to a different namespace which was only possible using two undocumented parameters.
Patch sent and awaiting approval merged.

#!/usr/bin/env ruby
require 'gitlab'

group_id = 5
token    = 'sekret'
endpoint = 'http://gitlab.tld/api/v3'

projects = %w(

Gitlab.configure do |config|
  config.endpoint      = endpoint
  config.private_token = token

projects.each do |proj|
    :description    => "Project #{proj}",
    :issues_enabled => true,
    :wiki_enabled   => false,
    :group_id       => group_id,
    :namespace_id   => group_id

Live P2V migration, on the cheap

Last week I performed a live physical to virtual migration for a customer. Ghetto style.
I was actually surprised by its simplicity and how well it turned out in the end.

Basically what I did was boot the target server to a live environment in order to be able to have raw access to the disk. In this case I’ve used FAI’s Sysinfo mode, but booting any Linux Live CD would have worked equally well. After it’s booted I’ve created the the necessary partitions, created filesytems and mounted them. When those steps were done it was time to sync data and make the virtual system bootable.

Usually I would advise against this kind of migration but in this case there was only static data involved.
It worked out well for me, but your mileage may vary.

Alright here’s what I did step by step…

1. Backup partition layout and MBR

source:~ # sfdisk -d /dev/sda > /tmp/source-partitions
source:~ # dd if=/dev/sda of=/tmp/source-mbr.img bs=512 count=1
source:~ # scp /tmp/source-* target:/tmp/

2. Restore partition layout and MBR

target:~ # dd if=/tmp/source-mbr.img of=/dev/sda bs=446 count=1
target:~ # sfdisk /dev/xvda < /tmp/source-partitions

3. Create a filesystem on each partition

target:~ # for partition in xvda1 xvda6 xvda7 xvda8 xvda9 xvda10; do mkfs.ext4 /dev/$partition; done

4. Create a swap space

mkswap /dev/xvda2
swapon /dev/xvda2

5. Mount the newly created partitions

target:~ # mount /dev/xvda1 /mnt/tmp/
target:~ # mount /dev/xvda6 /mnt/tmp/var/
target:~ # mount /dev/xvda7 /mnt/tmp/tmp/
target:~ # mount /dev/xvda8 /mnt/tmp/usr
target:~ # mount /dev/xvda9 /mnt/tmp/home/
target:~ # mount /dev/xvda10 /mnt/tmp/data/

6. Synchronise data from source to target

target:~ # for i in $(seq 1 10);
rsync -av --delete --progress \
--exclude=/dev \
--exclude=/sys \
--exclude=/proc \
--exclude=/mnt/ \
--exclude=/media/ \
source:/* /mnt/tmp/;
target:~ #

Repeat this step until you’re confident nothing has changed anymore.

7. Enter the synchronized environment

target:~ # mount -t proc none /mnt/tmp/proc
target:~ # mount --rbind /dev /mnt/tmp/dev

target:~ # chroot /mnt/tmp /bin/bash
chroot:~ # source /etc/profile

8. Fix fstab entries

(only for Xen vm’s)

chroot:~ # sed -i 's+/dev/sda+/dev/xvda+g' /etc/fstab
chroot:~ # vim /etc/fstab

9. Fix network settings

chroot:~ # vim /etc/network/interfaces

10. Make the system bootable

chroot:~ # grep -v rootfs /proc/mounts > /etc/mtab
chroot:~ # grub-install --no-floppy /dev/xvda
chroot:~ # update-grub

Once you’ve run the above commands it’s time to exit the chroot, unmount all partitions and reboot
into your migrated environment. Enjoy!

“Dynamic lookup is deprecated. Support will be removed in Puppet 2.8.” when using facts in templates

Recently ran into some of these when running Puppet 2.7.x:

warning: Dynamic lookup of $variable is deprecated. Support 
will be removed in Puppet 2.8.  Use a fully-qualified variable
name (e.g., $classname::variable) or parameterized classes.

The solution was pretty obvious and it’s easy to fix because the solution is mentioned in the warning. However the warning fails to mention what to do with Facter facts inside ERB templates. Figuring out how to fix this took me quite a while although it’s easy too.

Let’s start by taking a look at this snippet:

NameVirtualHost <%= 'ipaddress' %>:80
< VirtualHost <%= 'ipaddress' %>:80 >
	DocumentRoot /var/www/vhosts/

If we’d apply this template in a manifest using Puppet 2.7.x we would run into the warning mentioned above because the IP address fact is out of scope. In order to avoid this we use lookupvar for Facter facts:

NameVirtualHost <%= scope.lookupvar('ipaddress') %>:80
< VirtualHost <%= scope.lookupvar('ipaddress') %>:80 >
	DocumentRoot /var/www/vhosts/

puppetmaster-passenger session ticket A: tlsv1 alert decrypt error

There is a bug in the default puppetmaster vhost that’s included in Ubuntu-10.10’s puppetmaster-passenger package.

# puppetd --server puppet.fqdn --waitforcert 60 --no-usecacheonfailure
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert decrypt error
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

Lucky for us this is easily fixed using the patch below.
If you have trouble copy/pasting it, here’s a direct link: apache_passenger_tlsv1.patch.

*** puppetmaster	Mon Feb  21 15:25:28 2011
---	Mon Feb  21 15:25:13 2011
*** 13,19 ****
          SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
          # If Apache complains about invalid signatures on the CRL, you can try disabling
          # CRL checking by commenting the next line, but this is not recommended.
!         SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
          # Set to require if this puppetmaster doesn't issue certificates
          # to puppet clients.
          # NB: this requires SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem
--- 13,20 ----
          SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
          # If Apache complains about invalid signatures on the CRL, you can try disabling
          # CRL checking by commenting the next line, but this is not recommended.
!         # default: SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
!         SSLCARevocationPath     /var/lib/puppet/ssl/ca/crl
          # Set to require if this puppetmaster doesn't issue certificates
          # to puppet clients.
          # NB: this requires SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem

You can apply it using:

# patch -i apache_passenger_tlsv1.patch \

I’ve already filed a bug and supplied the solution a while ago. It has been confirmed but it’s still not in the default repositories yet, which is beyond my reach.

Veewee 0.1.16: ‘we tried to create a box or a box was here before but now it’s gone’

A few days ago I ran into a strange error using Veewee 0.1.16 on Mac OSX 10.6.7. Due to this error I couldn’t create a single basebox. No matter what I did. However thanks to @patrickdebois’ speedy support it was easily fixed.

For those who can’t wait for the fix to enter upstream I’ve included the patch below.
And also a direct link veewee_shell_osx10.6.7.patch, if you have trouble copy/pasting it.

# cat veewee_shell_osx10.6.7.patch 
*** shell.rb	Wed Apr  6 15:09:59 2011
---	Wed Apr  6 15:10:03 2011
*** 3,16 ****
  module Veewee
    class Shell
!     def self.execute(command,options = {})
        IO.popen("#{command}") { |f| print f }
      #pty allows you to gradually see the output of a local command
!       def self.execute2(command, options = {} )
          require "pty"
                PTY.spawn( command ) do |r, w, pid|
--- 3,16 ----
  module Veewee
    class Shell
!     def self.execute2(command,options = {})
        IO.popen("#{command}") { |f| print f }
      #pty allows you to gradually see the output of a local command
!       def self.execute(command, options = {} )
          require "pty"
                PTY.spawn( command ) do |r, w, pid|

You can apply it using:

# sudo patch -i shell_osx10.6.7.patch \

Fixing strange device names using Ubuntu templates on VMware ESX or vSphere

If you regularly deploy Ubuntu VM templates on your VMware ESX(i) or VMware vSphere boxes you will probably run into strange network device numbers. This is caused by a udev rule. This problem has been confirmed to exist in Ubuntu 9.04, 9.10 and 10.04. I haven’t had the time to check out other versions of Ubuntu. It’s also still existing in RHEL 6 and Scientific Linux 6.

As you can see below we have two ethernet devices: eth4 and eth5 instead of the usual eth0
and eth1.

root@box:~# ifconfig | grep Link
lo        Link encap:Local Loopback  
eth4  Link encap:UNSPEC  HWaddr  
eth5  Link encap:UNSPEC  HWaddr  

Lucky for us it’s very simple to persistenly assign the correct device names to the corresponding mac address.

root@box:~# grep eth4 /etc/udev/rules.d/70-persistent-net.rules 
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", 
ATTR{address}=="00:11:22:33:44:55", ATTR{dev_id}=="0x0", 
ATTR{type}=="1", KERNEL=="eth*", NAME="eth4"

Use your favorite editor to edit /etc/udev/rules.d/70-persistent-net.rules.
Replace NAME=”eth4″ with NAME=”eth0″ and do the same with eth5.
Save the config file, reboot and you’re done!

After writing this guide I’ve also found this issue to exist on other udev-based distro’s (e.g. OpenSUSE) and other VMware products too (e.g. Fusion and Workstation). The same fix applies, so no worries there.

Enabling Multitasking and Homescreen wallpapers on the iPhone 3G iOS4

For those who are still using the trusty old iPhone 3G the recent iOS4 release didn’t really bring much interesting features. Among the list of long-awaited features one in particular wasn’t available on the 3G: multitasking.

Another feature that wasn’t available is the Homescreen wallpaper option. Although it was possible to overcome this on the old firmwares using tools like Winterboard. Unfortunately Winterboard hasn’t been released for the new iOS4 just yet.

Luckily for us poor saps Apple has only decided to disable these features and not to remove them completely from the iOS4 firmware. Both features can be enabled easily by editing just one .plist file.

First you’ll need to Jailbreak your device using PwnageTool4. It’s a self explanatory tool, so there’s no use in guiding you through these steps. Be sure to install OpenSSH using Cydia once the Jailbreak has been executed.

Now SCP into your device and transfer the file called N82AP.plist.

# scp -P22 root@ /tmp/

Now use the XCode Property List Editor to edit the plist file. XCode can be obtained freely from the Apple Dev site. Add both “multitasking” and “homescreen-wallpaper” entries as booleans to Root – capabilities.
plist editor

Save, exit and transfer your edited N82AP.plist file back to your device like this:

# scp -P22 N82AP.plist root@

Reboot your iPhone. Both Multitasking and Homescreen wallpaper are now enabled!


Shared hosting attack mitigation, part 1: Apache MPM-ITK

When it comes to shared hosting we often get to see the same patterns and CMS installs over and over again. We have the big guns: Joomla, WordPress, Drupal, Typo3. And then there are smaller guys like e107, eZ Publish, XOOPS. Occasionally we run into TinyCMS, MiaCMS or even a YupiCMS install. They all have several things in common: they all try to be user friendly and easy to set-up. They’ll make you have a website online in a matter of hours or even minutes. So far, so good.

But what about patching? People tend to neglect to keep their installs up-to-date. We often get to hear people are afraid to break their corporate website or they rely on the web agency who built their website. However most of these agencies don’t want to take the risk either, unless there’s a hefty monthly support fee involved. And even when there is one they don’t always get around to do so.

This might become a serious problem for anyone running (on) a shared hosting server. XSS attacks, SQL injection and other kinds of intrusion might lead to collateral damage. Other people’s sites just might get attacked too. By default any vHost is readable and more importantly writeable by the user that is used to run PHP scripts. In most cases this is the Apache user.

If we can run our hosted PHP scripts using a separate user for each vHost we should be able to mitigate a large amount of common attacks towards other vHosts! However it isn’t feasible to run each hosting inside a separate chrooted environment. It would take too much time to set-up and it’s rather complex to patch it.

This is where Apache MPM-ITK comes into play. It’s an MPM (Multi-Processing Module) that is able to use a separate uid/guid per vHost without the need for a separate chroot for each hosting.

Enough chit-chat. Let’s get to it.

Install a default LAMP stack:

# apt-get install apache2-mpm-itk apache2-utils apache2.2-common 
defoma fontconfig-config libapache2-mod-php5 libapr1 libaprutil1 
libexpat1 libfontconfig1 libfreetype6 libgd2-xpm libjpeg62 
libltdl3 libmcrypt4 libpq5 libt1-5 libxpm4 openssl ssl-cert 
openssl-blacklist php5-common php5-cli php5-gd php5-mcrypt 
php5-mysql ttf-dejavu ttf-dejavu-core ttf-dejavu-extra

If you’re already running a LAMP stack it’s easy to replace the default MPM:

# apt-get remove apache2-mpm-prefork
# apt-get install apache2-mpm-itk

Configuring MPM ITK isn’t exactly brain surgery. Simply add an IfModule statement to your existing vHost config or use the following example to create a new one:

# cat /etc/apache2/sites-available/vhost-example.conf
<VirtualHost *:80>
  <IfModule mpm_itk_module>
    AssignUserId example example

  DocumentRoot /var/www/vhosts/example/public_html/

  CustomLog /var/www/vhosts/example/logs/access_log combined
  ErrorLog /var/www/vhosts/example/logs/error_log

  ScriptAlias /cgi-bin/ /var/www/vhosts/example/cgi-bin/
  DirectoryIndex index.php index.html index.htm

  <Directory "/var/www/vhosts/example/cgi-bin/">
    Options +ExecCGI
    AllowOverride none

Don’t forget to create the default user and appropriate folders. Change the permissions afterwards:

# useradd example
# mkdir -p /var/www/vhosts/example/
# mkdir -p /var/www/vhosts/example/public_html/
# mkdir -p /var/www/vhosts/example/cgi-bin/
# mkdir -p /var/www/vhosts/example/logs/
# chown -R example:example /var/www/vhosts/example/

Once this has been done, enable your website and test the configuration before reloading the Apache daemon:

# a2ensite vhost-example.conf
Enabling site vhost-example.conf.
Run '/etc/init.d/apache2 reload' to activate new configuration!

# apache2ctl configtest
Syntax OK

# /etc/init.d/apache2 reload
Reloading web server config: apache2.

And that’s it! Use phpinfo(); if you want to check you’re actually running on MPM ITK. It should be listed in Loaded Modules. Another way to verify is using ps or top. The PHP scripts for your newly created vHost should be running on it’s own separate user:

15059 example 20  0 24596 5624 2468 R  0.7  3.9  0:00.02 apache2
15060 example 20  0 24596 5496 2340 R  0.7  3.8  0:00.02 apache2
15061 example 20  0 24596 5412 2256 R  0.7  3.7  0:00.02 apache2
15062 example 20  0 24596 5360 2204 R  0.7  3.7  0:00.02 apache2

There is obviously a performance penalty involved when using MPM ITK.
I’ve compared both the default MPM Prefork and MPM ITK using Apache Bench and phpinfo() serving the content:

# ab -n 2000 -c 4

mpm prefork vs mpm itk

Please do bear in mind that this was tested on low power hardware (Atom 1.6GHz cpu).

Authenticated SMTP with Postfix on CentOS, the easy way

This will be more of a future reference than an actual howto. It’s far from feature complete but it will get you started on authenticated SMTP sessions using Postfix. Quick & dirty.

Installing Postfix and SASL on CentOS:

# yum install postfix
# yum install cyrus-sasl cyrus-sasl-plain cyrus-sasl-md5

Let’s move on to the configuration now. Below you will find my default template
for /etc/postfix/

# cat /etc/postfix/
smtpd_banner = $myhostname ESMTP $mail_name

myhostname = example.tld
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = example.tld
mydestination = example.tld, localhost.localdomain, localhost
transport_maps =
relayhost =
mynetworks =
#mynetworks = hash:/etc/postfix/networks

smtpd_sasl_path = sasl2/smtpd.conf
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain = example.tld
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes

recipient_delimiter = +
inet_interfaces = all

smtpd_recipient_restrictions = permit_mynetworks,

Use smtpd_recipient_restrictions to make sure you’re not running an Open Relay server accepting spam from anyone. It will accept unauthenticated sessions originating from localhost only. However authenticated sessions are generally allowed.

Make sure your Postfix daemon is actually able to communicate with the sasl daemon.
To accomplish this append this to /etc/postfix/

# cat /etc/postfix/
submission inet n       -       n       -       -       smtpd
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_security_options=noanonymous
  -o smtpd_sasl_local_domain=example.tld
  -o header_checks=
  -o body_checks=
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject_unauth_destination
  -o smtpd_sasl_security_options=noanonymous,noplaintext
  -o smtpd_sasl_tls_security_options=noanonymous

Next up we’re going to add our users to the sasl database:

# saslpasswd2 -c -u $hostname $user

While we’re at it, it might be a good idea to fix permissions on the sasl database. Otherwise Postfix will be unable to read from it or write to it.

# chown postfix:postfix /etc/sasldb2
# chmod 660 /etc/sasldb2

We’re almost there. To configure the sasl daemon itself:

# cat /usr/lib/sasl2/smtpd.conf
pwcheck_method: auxprop
auxprop_plugin: sasldb
# cat /usr/lib/sasl/smtpd.conf
pwcheck_method: auxprop
auxprop_plugin: sasldb

And finally restart the daemons.

# /etc/init.d/sasld restart
# /etc/init.d/postfix restart

Be sure to confirm it’s working using both your default mail client and /var/log/maillog.