Skip to content

Tag: apache

puppetmaster-passenger session ticket A: tlsv1 alert decrypt error

There is a bug in the default puppetmaster vhost that’s included in Ubuntu-10.10’s puppetmaster-passenger package.

# puppetd --server puppet.fqdn --waitforcert 60 --no-usecacheonfailure
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: tlsv1 alert decrypt error
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

Lucky for us this is easily fixed using the patch below.
If you have trouble copy/pasting it, here’s a direct link: apache_passenger_tlsv1.patch.

*** puppetmaster	Mon Feb  21 15:25:28 2011
--- puppetmaster.new	Mon Feb  21 15:25:13 2011
***************
*** 13,19 ****
          SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
          # If Apache complains about invalid signatures on the CRL, you can try disabling
          # CRL checking by commenting the next line, but this is not recommended.
!         SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
          # Set to require if this puppetmaster doesn't issue certificates
          # to puppet clients.
          # NB: this requires SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem
--- 13,20 ----
          SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
          # If Apache complains about invalid signatures on the CRL, you can try disabling
          # CRL checking by commenting the next line, but this is not recommended.
!         # default: SSLCARevocationFile     /var/lib/puppet/ssl/ca/ca_crl.pem
!         SSLCARevocationPath     /var/lib/puppet/ssl/ca/crl
          # Set to require if this puppetmaster doesn't issue certificates
          # to puppet clients.
          # NB: this requires SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem

You can apply it using:

# patch -i apache_passenger_tlsv1.patch \
/etc/apache2/sites-available/puppetmaster

I’ve already filed a bug and supplied the solution a while ago. It has been confirmed but it’s still not in the default repositories yet, which is beyond my reach.

Shared hosting attack mitigation, part 1: Apache MPM-ITK

When it comes to shared hosting we often get to see the same patterns and CMS installs over and over again. We have the big guns: Joomla, WordPress, Drupal, Typo3. And then there are smaller guys like e107, eZ Publish, XOOPS. Occasionally we run into TinyCMS, MiaCMS or even a YupiCMS install. They all have several things in common: they all try to be user friendly and easy to set-up. They’ll make you have a website online in a matter of hours or even minutes. So far, so good.

But what about patching? People tend to neglect to keep their installs up-to-date. We often get to hear people are afraid to break their corporate website or they rely on the web agency who built their website. However most of these agencies don’t want to take the risk either, unless there’s a hefty monthly support fee involved. And even when there is one they don’t always get around to do so.

This might become a serious problem for anyone running (on) a shared hosting server. XSS attacks, SQL injection and other kinds of intrusion might lead to collateral damage. Other people’s sites just might get attacked too. By default any vHost is readable and more importantly writeable by the user that is used to run PHP scripts. In most cases this is the Apache user.

If we can run our hosted PHP scripts using a separate user for each vHost we should be able to mitigate a large amount of common attacks towards other vHosts! However it isn’t feasible to run each hosting inside a separate chrooted environment. It would take too much time to set-up and it’s rather complex to patch it.

This is where Apache MPM-ITK comes into play. It’s an MPM (Multi-Processing Module) that is able to use a separate uid/guid per vHost without the need for a separate chroot for each hosting.

Enough chit-chat. Let’s get to it.

Install a default LAMP stack:

# apt-get install apache2-mpm-itk apache2-utils apache2.2-common 
defoma fontconfig-config libapache2-mod-php5 libapr1 libaprutil1 
libexpat1 libfontconfig1 libfreetype6 libgd2-xpm libjpeg62 
libltdl3 libmcrypt4 libpq5 libt1-5 libxpm4 openssl ssl-cert 
openssl-blacklist php5-common php5-cli php5-gd php5-mcrypt 
php5-mysql ttf-dejavu ttf-dejavu-core ttf-dejavu-extra

If you’re already running a LAMP stack it’s easy to replace the default MPM:

# apt-get remove apache2-mpm-prefork
# apt-get install apache2-mpm-itk

Configuring MPM ITK isn’t exactly brain surgery. Simply add an IfModule statement to your existing vHost config or use the following example to create a new one:

# cat /etc/apache2/sites-available/vhost-example.conf
<VirtualHost *:80>
  <IfModule mpm_itk_module>
    AssignUserId example example
  </IfModule>

  DocumentRoot /var/www/vhosts/example/public_html/
  ServerName example.com
  ServerAlias www.example.com

  CustomLog /var/www/vhosts/example/logs/access_log combined
  ErrorLog /var/www/vhosts/example/logs/error_log

  ScriptAlias /cgi-bin/ /var/www/vhosts/example/cgi-bin/
  DirectoryIndex index.php index.html index.htm

  <Directory "/var/www/vhosts/example/cgi-bin/">
    Options +ExecCGI
    AllowOverride none
  </Directory>
</VirtualHost>

Don’t forget to create the default user and appropriate folders. Change the permissions afterwards:

# useradd example
# mkdir -p /var/www/vhosts/example/
# mkdir -p /var/www/vhosts/example/public_html/
# mkdir -p /var/www/vhosts/example/cgi-bin/
# mkdir -p /var/www/vhosts/example/logs/
# chown -R example:example /var/www/vhosts/example/

Once this has been done, enable your website and test the configuration before reloading the Apache daemon:

# a2ensite vhost-example.conf
Enabling site vhost-example.conf.
Run '/etc/init.d/apache2 reload' to activate new configuration!

# apache2ctl configtest
Syntax OK

# /etc/init.d/apache2 reload
Reloading web server config: apache2.

And that’s it! Use phpinfo(); if you want to check you’re actually running on MPM ITK. It should be listed in Loaded Modules. Another way to verify is using ps or top. The PHP scripts for your newly created vHost should be running on it’s own separate user:

15059 example 20  0 24596 5624 2468 R  0.7  3.9  0:00.02 apache2
15060 example 20  0 24596 5496 2340 R  0.7  3.8  0:00.02 apache2
15061 example 20  0 24596 5412 2256 R  0.7  3.7  0:00.02 apache2
15062 example 20  0 24596 5360 2204 R  0.7  3.7  0:00.02 apache2

There is obviously a performance penalty involved when using MPM ITK.
I’ve compared both the default MPM Prefork and MPM ITK using Apache Bench and phpinfo() serving the content:

# ab -n 2000 -c 4 http://example.com/index.php

mpm prefork vs mpm itk

Please do bear in mind that this was tested on low power hardware (Atom 1.6GHz cpu).

Apache mod_evasive DDoS prevention on a CentOS 5.x Plesk environment

A couple weeks ago I was asked to implement a DDoS prevention system for a customer who had been suffering some DDoS events.
Shouldn’t be too hard. The only catch was that the box was running a Plesk 9 LAMP stack.
I chose to go with mod_evasive, a GPL2 licensed module for Apache[1-2].
It can be downloaded at http://www.zdziarski.com/blog/?page_id=442 either using the CVS repository or as a tarball.
I ended up using the latter. The current stable version at this point is 1.10.1.
This guide has been assembled using Centos 5.4 with a Plesk 9 LAMP stack. I haven’t tested it on anything else, but it should work just the same way it does in this guide.
===== Install =====
First things first. Let’s check out which versions of apxs are installed:
<code>
# updatedb; locate apxs | grep bin
/usr/local/psa/admin/bin/apxs
</code>
This version is Parallell’s default version, wich comes with Plesk.
If this is the only version you have available you will need to install the generic httpd-devel package. Parallell’s version of apxs is a bit limited and won’t compile the module.
<code>
# yum install httpd-devel
</code>
Give it another go and you should end up with something like this:
<code>
# updatedb; locate apxs | grep bin
/usr/local/psa/admin/bin/apxs
/usr/sbin/apxs
</code>
Onto downloading and extracting the mod_evasive module:
<code>
# cd /usr/src
# wget http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
# tar xvzf mod_evasive_1.10.1.tar.gz
mod_evasive/
mod_evasive/.cvsignore
mod_evasive/LICENSE
mod_evasive/Makefile.tmpl
mod_evasive/README
mod_evasive/mod_evasive.c

A couple weeks ago I was asked to implement a DDoS prevention system in Apache for a customer who had obviously been suffering some gnarly DDoS events. Shouldn’t be too hard. The only catch was that the box was running a Plesk 9 LAMP stack.

I chose to go with mod_evasive, a GPL2 licensed module for Apache[1-2]. It can be downloaded at http://www.zdziarski.com/projects/mod_evasive/ either using the CVS repository or as a tarball. I ended up using the latter. The current stable version at this point is 1.10.1.

This guide has been assembled using Centos 5.4 with a Plesk 9 LAMP stack. I haven’t tested it on anything else, but it should work just the same way it does in this guide.

Install

First things first. Let’s check out which versions of apxs are installed:

# updatedb; locate apxs | grep bin
/usr/local/psa/admin/bin/apxs

This version is Parallell’s default version, wich comes with Plesk.

If this is the only version you have available you will need to install the generic httpd-devel package. Parallell’s version of apxs is a bit limited and won’t compile the module.

# yum install httpd-devel

Give it another go and you should end up with something like this:

# updatedb; locate apxs | grep bin
/usr/local/psa/admin/bin/apxs
/usr/sbin/apxs

Onto downloading and extracting the mod_evasive module:

# wget http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
# tar xvzf mod_evasive_1.10.1.tar.gz mod_evasive/
mod_evasive/.cvsignore
mod_evasive/LICENSE
mod_evasive/Makefile.tmpl
mod_evasive/README
mod_evasive/mod_evasive.c
mod_evasive/mod_evasive20.c
mod_evasive/mod_evasiveNSAPI.c
mod_evasive/test.pl
mod_evasive/CHANGELOG

Be sure to check out the CHANGELOG and README files!

Even though people tend to forget this step… those files are included for a reason.
Let’s move on to compiling and actually installing the module inside the Plesk chroot:

# /usr/sbin/apxs -cia /usr/src/mod_evasive/mod_evasive20.c
/usr/lib/apr-1/build/libtool --silent --mode=compile gcc -prefer-pic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions \
-fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -fno-strict-aliasing \
-DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -pthread -I/usr/include/httpd  -I/usr/include/apr-1   -I/usr/include/apr-1 \
-c -o mod_evasive20.lo mod_evasive20.c && touch mod_evasive20.slo
mod_evasive20.c: In function 'access_checker':
mod_evasive20.c:212: warning: implicit declaration of function 'getpid'
mod_evasive20.c:212: warning: format '%ld' expects type 'long int', but argument 4 has type 'int'
mod_evasive20.c:229: warning: ignoring return value of 'system', declared with attribute warn_unused_result
mod_evasive20.c: In function 'destroy_hit_list':
mod_evasive20.c:301: warning: control reaches end of non-void function
mod_evasive20.c: In function 'create_hit_list':
mod_evasive20.c:118: warning: control reaches end of non-void function
/usr/lib/apr-1/build/libtool --silent --mode=link gcc -o mod_evasive20.la  -rpath /usr/lib/httpd/modules -module -avoid-version    mod_evasive20.lo
/usr/lib/httpd/build/instdso.sh SH_LIBTOOL='/usr/lib/apr-1/build/libtool' mod_evasive20.la /usr/lib/httpd/modules
/usr/lib/apr-1/build/libtool --mode=install cp mod_evasive20.la /usr/lib/httpd/modules/
cp .libs/mod_evasive20.so /usr/lib/httpd/modules/mod_evasive20.so
cp .libs/mod_evasive20.lai /usr/lib/httpd/modules/mod_evasive20.la
cp .libs/mod_evasive20.a /usr/lib/httpd/modules/mod_evasive20.a
chmod 644 /usr/lib/httpd/modules/mod_evasive20.a
ranlib /usr/lib/httpd/modules/mod_evasive20.a
PATH="$PATH:/sbin" ldconfig -n /usr/lib/httpd/modules
----------------------------------------------------------------------
Libraries have been installed in:
/usr/lib/httpd/modules
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH' environment variable
during linking
- use the `-Wl,--rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to `/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 755 /usr/lib/httpd/modules/mod_evasive20.so
[activating module `evasive20' in /etc/httpd/conf/httpd.conf]

Next up we need to restart apache to load the module:

# /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

Verify

Verify if the module is in the apache config:

# grep -i evasive /etc/httpd/conf/httpd.conf
LoadModule evasive20_module   /usr/lib/httpd/modules/mod_evasive20.so

Check wether the modtule is actually loaded:

# php -r 'phpinfo();' | grep -i evasive
^ Loaded Modules | core prefork http_core mod_so mod_auth_basic
mod_auth_digest mod_authn_file mod_authn_alias mod_authn_anon
mod_authn_dbm mod_authn_default mod_authz_host mod_authz_user
mod_authz_owner mod_authz_groupfile mod_authz_dbm mod_authz_default
util_ldap mod_authnz_ldap mod_include mod_log_config mod_logio
mod_env mod_ext_filter mod_mime_magic mod_expires mod_deflate
mod_headers mod_usertrack mod_setenvif mod_mime mod_dav mod_status
mod_autoindex mod_info mod_dav_fs mod_vhost_alias mod_negotiation
mod_dir mod_actions mod_speling mod_userdir mod_alias mod_rewrite
mod_proxy mod_proxy_balancer mod_proxy_ftp mod_proxy_http mod_proxy_connect
mod_cache mod_suexec mod_disk_cache mod_file_cache mod_mem_cache mod_cgi
mod_version **mod_evasive20** mod_perl mod_php5 mod_proxy_ajp mod_python mod_ssl |

Seems it’s loaded just fine.

Configure

Now let’s get started with the configuration. I couldn’t find any default config, but this one seems to run just fine. Even on a heavily visited shared hosting server.

Add the following rules at the end of /etc/httpd/conf/httpd.conf:

<IfModule mod_evasive20.c>
DOSHashTableSize 3097
DOSPageCount 6
DOSSiteCount 100
DOSPageInterval 2
DOSSiteInterval 2
DOSBlockingPeriod 600
</IfModule>

And let’s kick apache one last time:

# /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

NOTE

Be sure to keep an eye on your webstats!

There might be a sudden drop in the amount of unique visitors. This might be a result of an attack that’s been evaded. However if you’ve used different configuration parameters you might have restricted it too much and you’ll end up restricting valid customers too. I haven’t recieved any negative comments about this setup (yet?)

Use with caution!