Feb 172010
 

One of the smaller projects I’ve been working on lately is monitoring Realtime DNS Blacklists (RBL’s) status with Zabbix. I’m confident most of you are already familiar with RBL’s. For those who are not, here’s a small introduction shamelessly stolen from Wikipedia:

A DNSBL (DNS-based Blackhole  List, Block List, or Blacklist; see below) is a list of IP addresses published through the Internet Domain Name Service in a particular format. DNSBLs are most often used to publish the addresses of computers or networks linked to spamming; most mail server  software can be configured to reject or flag messages which have been sent from a site listed on one or more such lists.

And that’s exactly what we’re going to monitor. If we are listed on one of those RBL’s we’d like to know about it, don’t we? 

Let’s get to it then. First of all we need an up to date list of RBL’s which we can use to check whether we’re listed or not. You could try the list I’m maintaining and using for my own monitoring purposes. The most recent version can be found here. It contains a whopping 92 RBL’s to get you started with.

Now that we have an up-to-date list of common used RBL’s it’s time for some shell scripting:

#!/bin/bash

## TomDV
## 2010-01-25
## http://blog.penumbra.be/2010/02/zabbix-monitor-dns-blacklists/

cd /usr/share/zabbix/
RBL="`cat rbl_list.txt`"

W=$( echo ${1} | cut -d. -f1 )
X=$( echo ${1} | cut -d. -f2 )
Y=$( echo ${1} | cut -d. -f3 )
Z=$( echo ${1} | cut -d. -f4 )

STATUS=0

for i in $RBL
do
    RESULT=$( host -t a $Z.$Y.$X.$W.$i 2>&1 )
    if [ $? -eq 0 ]
    then
        #echo “The IP ADDRESS ${1} is listed at $i:\n$RESULT” ## DEBUG
        let "STATUS += 1"
    fi
    #echo $RESULT ## DEBUG
done

if [ $STATUS -lt 1 ]
then
    echo 0
else
    echo $STATUS
fi

This script takes the IP address of your server as input.

I’ve intentionally left the debug code inside the script. This way the output can be used right away within Zabbix. However if you’re listed on one of the blacklists you can run the script with the debug code uncommented and you get a list of all the RBL’s you’re listed in.

I’ve put this script in /usr/share/zabbix, along with the rbl_list.txt file you can find above.

# cat /etc/zabbix/zabbix_agent.d/rbl.conf
UserParameter=rbl.mx1,/usr/share/zabbix/zabbix-rbl.sh 1.2.3.4
UserParameter=rbl.mx2,/usr/share/zabbix/zabbix-rbl.sh 5.6.7.8

I also have the following line in /etc/zabbix/zabbix_agentd.conf and /etc/zabbix/zabbix_agent.conf to load custom config files:

Include=/etc/zabbix/zabbix_agent.d/

And that’s about it. Let’s see if we’re listed in any of the RBL’s:

# zabbix_agent -t rbl.mx1; zabbix_agent -t rbl.mx2;
rbl.mx1                                    [t|0]
rbl.mx2                                    [t|0]

Any value above zero means you’re listed. I guess we’re safe.
If you’re listed just uncomment the debug code. It will show you which RBL’s you’re in.

Happy monitoring! :)

Feb 142010
 

This should be an easy to follow guide about how to install Xen on Debian 5. You should be able to copy/paste most parts of on your shell. Please run this only on a clean and up-to-date Debian system.

Alright let’s get to it quick ‘n dirty.

First of all make sure your Debian install actually is up-to-date:

# apt-get update; apt-get upgrade

Let’s see which Xen kernel images are available and pick the most recent one to install:

# apt-cache search xen | grep image | awk '{print $1}'
linux-image-2.6-xen-amd64
linux-image-xen-amd64
linux-image-2.6.26-1-xen-amd64
linux-image-2.6.26-2-xen-amd64
xen-linux-system-2.6.26-1-xen-amd64
xen-linux-system-2.6.26-2-xen-amd64
# apt-get install `apt-cache search xen-linux-system \
| sort | tail -1 | awk '{print $1}'`

Once this is done reboot your system, login again and run:

# uname -a
Linux elysium 2.6.26-2-xen-amd64 #1 SMP Thu Feb 11 02:57:18 UTC 2010 x86_64 GNU/Linux

As you can see, the system is running kernel 2.6.26-2 with the xen-amd64 patch set.
As of now we should have Dom-0 available:

# xm list
Name                   ID   Mem VCPUs      State   Time(s)
Domain-0               0  3885     2     r-----      8.5

Perfection!

Let’s move on to the network. By default there is no bridge available from the virtual machines towards the external network. It’s fairly easy to accomplish though:

# vim /etc/xen/xend-config.sxp

Look for the following line and uncomment it:

(network-script network-bridge)

And while we’re editing the xend-config.sxp file, change the following line:

(xend-unix-server no)

Into:

(xend-unix-server yes)

Be sure to reload the new settings:

# /etc/init.d/xend restart

If you don’t edit this line or if you don’t reload, you obviously won’t be able to install or manage your virtual machines. You’d get to see errors like this:

ERROR    internal error failed to connect to xend
Traceback (most recent call last):
  File "/usr/bin/virt-install", line 693, in
    main()
  File "/usr/bin/virt-install", line 508, in main
    conn = cli.getConnection(options.connect)
  File "/var/lib/python-support/python2.5/virtinst/cli.py", line 123, in getConnection
    return libvirt.open(connect)
  File "/usr/lib/python2.5/site-packages/libvirt.py", line 140, in open
    if ret is None:raise libvirtError('virConnectOpen() failed')
libvirtError: internal error failed to connect to xend

There are more options available to connect to xend.: xend-http-server, xend-tcp-xmlrpc-server, xend-unix-xmlrpc-server, xend-relocation-server.

The reason why I chose xend-unix-server over anything else is pretty straight forward. It only listens on the Unix socket layer which doesn’t need any other networking protocol to operate. You could compare it to connecting to localhost, but without the need for a networking device (e.g. lo0).

If you want to be able to manage this Xen server from a central node, be sure to change the appropriate management protocol. It’s beyond the scope of this howto, but I might post an howto on this subject later on this blog.

On to installing a virtual machine!

I for one am a big fan of abstraction layers. It makes life for a SysAdmin or DevOp so much easier if you’re able to run the same commands on different operating systems, platforms or architectures in general. That’s why I use libvirt. It’s a collection of libraries and tools that can be used to deploy virtual machines on different types of virtualization systems. Including but not limited to Xen, KVM and Qemu.

It’s also available as a Debian package, so installing it is very straight forward:

# apt-get install libvirt-bin virtinst

Once this is done we can install our first virtual machine:

# virt-install \
--name=test-debian-install \
--ram=1024 \
--file-size=10 \
--nographics \
--paravirt \
--file=/var/lib/xen/images/test-debian-install.img \
--location=http://ftp.belnet.be/debian/dists/lenny/main/\
installer-i386

It’s probably a good idea to store this as a shell script on your Xen host for future reference.
You should see a familiar installer within seconds after invoking the command.

Once the install has completed you should be greeted with your new virtual machine’s login prompt:

Debian GNU/Linux 5.0 test-debian-install hvc0

test-debian-install login:

To exit your virtual machine’s console, simply press Ctrl-]

Enjoy!

Jan 102010
 

For those who want or need to build Zabbix 1.8 on CentOS 5: there is an excellent RPM Spec file available at Andrew Farley’s blog. He’s also been kind enough to host a series of precompiled RPM packages.

If you look at the changelog you might find yours truly. I’ve contributed a patch to add a couple of dependencies and to fix a couple of bugs. So I thought I should share.

First of all the RPM Spec file:

http://repo.andrewfarley.com/centos/specs/zabbix.spec

If you’d like to compile your own packages you might follow this guide:

# yum groupinstall "Development Tools"
# yum install rpmdevtools
# rpmdev-setuptree

To compile for your running architecture:

# rpmbuild -bb --clean zabbix.spec

Or if you’d like to build for a specific architecture:

# rpmbuild -bb --clean --target i686 zabbix.spec

After the compile process you’ll find the RPM files in the following directory:

~/rpmbuild/RPMS/zabbix*.rpm
Dec 282009
 

Lately I’ve seen some customers struggle with their external USB drives formatted in NTFS on CentOS servers. Because it’s such a common problem I decided to make a very quick howto on the subject.

The Fuse packages found in the default CentOS repository haven’t been compiled with NTFS-3G support. Even though there are fuse-ntfs3g packages available! To get around this I prefer to enable the RPM Forge repository. It’s very easy to do so.

On 32bit platforms:

# wget http://apt.sw.be/redhat/el5/en/i386/RPMS.dag/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
# rpm -i rpmforge*i386.rpm

On 64bit platforms:

# wget http://apt.sw.be/redhat/el5/en/x86_64/RPMS.dag/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
# rpm -i rpmforge*x86_64.rpm

Moving on:

# rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt
# yum update

Installing RPM Forge’s Fuse packages and dependencies:

# yum install -y kernel-devel kernel-headers
# yum install -y --enablerepo=rpmforge dkms dkms-fuse fuse
# yum install -y --enablerepo=rpmforge fuse-ntfs-3g-devel fuse-devel

For some strange reason, the Fuse module in the package isn’t compiled yet. Lucky for us the package maintainer has provided the source code for us:

# cd /usr/src/fuse-2.7.4-1.nodist.rf/
# ./configure
# make
# make install

And now the part most people seem to forget, insert the Fuse module into the kernel:

# insmod /lib/modules/2.6.18-164.9.1.el5/kernel/fs/fuse/fuse.ko

Mount the external disk:

# mount -t ntfs-3g /dev/sdc1 /mnt/usb/
# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/sdc5             466G   79M  466G   1% /mnt/usb

That’s all there’s to it!

[root@srv01 src]# yum install -y –enablerepo=rpmforge dkms dkms-fuse fuse
Thats
[root@srv01 src]# yum install -y –enablerepo=rpmforge fuse-ntfs-3g-devel.i386 fuse-devel.i386
Dec 152009
 
A couple weeks ago I was asked to implement a DDoS prevention system for a customer who had been suffering some DDoS events.
Shouldn’t be too hard. The only catch was that the box was running a Plesk 9 LAMP stack.
I chose to go with mod_evasive, a GPL2 licensed module for Apache[1-2].
It can be downloaded at http://www.zdziarski.com/blog/?page_id=442 either using the CVS repository or as a tarball.
I ended up using the latter. The current stable version at this point is 1.10.1.
This guide has been assembled using Centos 5.4 with a Plesk 9 LAMP stack. I haven’t tested it on anything else, but it should work just the same way it does in this guide.
===== Install =====
First things first. Let’s check out which versions of apxs are installed:
<code>
# updatedb; locate apxs | grep bin
/usr/local/psa/admin/bin/apxs
</code>
This version is Parallell’s default version, wich comes with Plesk.
If this is the only version you have available you will need to install the generic httpd-devel package. Parallell’s version of apxs is a bit limited and won’t compile the module.
<code>
# yum install httpd-devel
</code>
Give it another go and you should end up with something like this:
<code>
# updatedb; locate apxs | grep bin
/usr/local/psa/admin/bin/apxs
/usr/sbin/apxs
</code>
Onto downloading and extracting the mod_evasive module:
<code>
# cd /usr/src
# wget http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
# tar xvzf mod_evasive_1.10.1.tar.gz
mod_evasive/
mod_evasive/.cvsignore
mod_evasive/LICENSE
mod_evasive/Makefile.tmpl
mod_evasive/README
mod_evasive/mod_evasive.c

A couple weeks ago I was asked to implement a DDoS prevention system in Apache for a customer who had obviously been suffering some gnarly DDoS events. Shouldn’t be too hard. The only catch was that the box was running a Plesk 9 LAMP stack.

I chose to go with mod_evasive, a GPL2 licensed module for Apache[1-2]. It can be downloaded at http://www.zdziarski.com/projects/mod_evasive/ either using the CVS repository or as a tarball. I ended up using the latter. The current stable version at this point is 1.10.1.

This guide has been assembled using Centos 5.4 with a Plesk 9 LAMP stack. I haven’t tested it on anything else, but it should work just the same way it does in this guide.

Install

First things first. Let’s check out which versions of apxs are installed:

# updatedb; locate apxs | grep bin
/usr/local/psa/admin/bin/apxs

This version is Parallell’s default version, wich comes with Plesk.

If this is the only version you have available you will need to install the generic httpd-devel package. Parallell’s version of apxs is a bit limited and won’t compile the module.

# yum install httpd-devel

Give it another go and you should end up with something like this:

# updatedb; locate apxs | grep bin
/usr/local/psa/admin/bin/apxs
/usr/sbin/apxs

Onto downloading and extracting the mod_evasive module:

# wget http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
# tar xvzf mod_evasive_1.10.1.tar.gz mod_evasive/
mod_evasive/.cvsignore
mod_evasive/LICENSE
mod_evasive/Makefile.tmpl
mod_evasive/README
mod_evasive/mod_evasive.c
mod_evasive/mod_evasive20.c
mod_evasive/mod_evasiveNSAPI.c
mod_evasive/test.pl
mod_evasive/CHANGELOG

Be sure to check out the CHANGELOG and README files!

Even though people tend to forget this step… those files are included for a reason.
Let’s move on to compiling and actually installing the module inside the Plesk chroot:

# /usr/sbin/apxs -cia /usr/src/mod_evasive/mod_evasive20.c
/usr/lib/apr-1/build/libtool --silent --mode=compile gcc -prefer-pic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions \
-fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -fno-strict-aliasing \
-DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -pthread -I/usr/include/httpd  -I/usr/include/apr-1   -I/usr/include/apr-1 \
-c -o mod_evasive20.lo mod_evasive20.c && touch mod_evasive20.slo
mod_evasive20.c: In function 'access_checker':
mod_evasive20.c:212: warning: implicit declaration of function 'getpid'
mod_evasive20.c:212: warning: format '%ld' expects type 'long int', but argument 4 has type 'int'
mod_evasive20.c:229: warning: ignoring return value of 'system', declared with attribute warn_unused_result
mod_evasive20.c: In function 'destroy_hit_list':
mod_evasive20.c:301: warning: control reaches end of non-void function
mod_evasive20.c: In function 'create_hit_list':
mod_evasive20.c:118: warning: control reaches end of non-void function
/usr/lib/apr-1/build/libtool --silent --mode=link gcc -o mod_evasive20.la  -rpath /usr/lib/httpd/modules -module -avoid-version    mod_evasive20.lo
/usr/lib/httpd/build/instdso.sh SH_LIBTOOL='/usr/lib/apr-1/build/libtool' mod_evasive20.la /usr/lib/httpd/modules
/usr/lib/apr-1/build/libtool --mode=install cp mod_evasive20.la /usr/lib/httpd/modules/
cp .libs/mod_evasive20.so /usr/lib/httpd/modules/mod_evasive20.so
cp .libs/mod_evasive20.lai /usr/lib/httpd/modules/mod_evasive20.la
cp .libs/mod_evasive20.a /usr/lib/httpd/modules/mod_evasive20.a
chmod 644 /usr/lib/httpd/modules/mod_evasive20.a
ranlib /usr/lib/httpd/modules/mod_evasive20.a
PATH="$PATH:/sbin" ldconfig -n /usr/lib/httpd/modules
----------------------------------------------------------------------
Libraries have been installed in:
/usr/lib/httpd/modules
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH' environment variable
during linking
- use the `-Wl,--rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to `/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 755 /usr/lib/httpd/modules/mod_evasive20.so
[activating module `evasive20' in /etc/httpd/conf/httpd.conf]

Next up we need to restart apache to load the module:

# /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

Verify

Verify if the module is in the apache config:

# grep -i evasive /etc/httpd/conf/httpd.conf
LoadModule evasive20_module   /usr/lib/httpd/modules/mod_evasive20.so

Check wether the modtule is actually loaded:

# php -r 'phpinfo();' | grep -i evasive
^ Loaded Modules | core prefork http_core mod_so mod_auth_basic
mod_auth_digest mod_authn_file mod_authn_alias mod_authn_anon
mod_authn_dbm mod_authn_default mod_authz_host mod_authz_user
mod_authz_owner mod_authz_groupfile mod_authz_dbm mod_authz_default
util_ldap mod_authnz_ldap mod_include mod_log_config mod_logio
mod_env mod_ext_filter mod_mime_magic mod_expires mod_deflate
mod_headers mod_usertrack mod_setenvif mod_mime mod_dav mod_status
mod_autoindex mod_info mod_dav_fs mod_vhost_alias mod_negotiation
mod_dir mod_actions mod_speling mod_userdir mod_alias mod_rewrite
mod_proxy mod_proxy_balancer mod_proxy_ftp mod_proxy_http mod_proxy_connect
mod_cache mod_suexec mod_disk_cache mod_file_cache mod_mem_cache mod_cgi
mod_version **mod_evasive20** mod_perl mod_php5 mod_proxy_ajp mod_python mod_ssl |

Seems it’s loaded just fine.

Configure

Now let’s get started with the configuration. I couldn’t find any default config, but this one seems to run just fine. Even on a heavily visited shared hosting server.

Add the following rules at the end of /etc/httpd/conf/httpd.conf:

<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 6
DOSSiteCount 100
DOSPageInterval 2
DOSSiteInterval 2
DOSBlockingPeriod 600
</IfModule>

And let’s kick apache one last time:

# /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

NOTE

Be sure to keep an eye on your webstats!

There might be a sudden drop in the amount of unique visitors. This might be a result of an attack that’s been evaded. However if you’ve used different configuration parameters you might have restricted it too much and you’ll end up restricting valid customers too. I haven’t recieved any negative comments about this setup (yet?)

Use with caution!