Using expect scripts to backup your Cisco configuration

In this short howto I’ll explain how to use expect scripts with Cisco devices. In this example I’m going to use it to backup the current running configuration.

Requirements

  • A working tftp server
  • Expect
  • Lucky for us both requirements are available in all major distro’s.

    The Debian/Ubuntu way:

    sudo apt-get install tftp tftpd expect

    Next on our todo list is configuring the tftp server. This should also be fairly easy:

    # cat /etc/xinetd.d/tftp
    service tftp
    {
        protocol        = udp
        port            = 69
        socket_type     = dgram
        wait            = yes
        user            = nobody
        server          = /usr/sbin/in.tftpd
        server_args     = /tftpboot
        disable         = no
    }
    

    Restart your xinetd server when done.

    # /etc/init.d/xinetd restart

    Make sure the /tftpboot folder exists and is owned by user and group nobody:

    # chown -R nobody:nobody /tftpboot

    You should also create an empty file where you’d like to save your configuration and rerun the above command to adjust permissions.

    # touch /tftpboot/config
    # chown -R nobody:nobody /tftpboot

    You should also create an empty file where you’d like to save your configuration and rerun the above command to adjust permissions.

    # touch /tftpboot/config
    # chown -R nobody:nobody /tftpboot

    We can now test our newly configured tftpd server:
    Create a new file in your home dir called config and put some random text in it.

    # cat /home/user/config
    test 12
    
    # tftp
    tftp> open localhost
    tftp> put config
    Sent 146 bytes in 0.0 seconds
    
    # cat /tftpboot/config
    test 12

    Excellent! We’re ready to receive config files from the Cisco device.

    Below you will find an example script:

    #!/usr/bin/expect
    
    ## TomDV
    ## http://blog.penumbra.be/2010/02/expect-scripts-backup-cisco-config/
    
    # ---------------- configuration ---------------- #
    set device 192.168.0.100    # cisco device
    set tftp 192.168.0.200      # tftp server
    set user someuser           # username
    set pass ultrasecret        # password
    set config                  # config destination
    set timeout 60
    
    # -------------- do not edit below -------------- #
    spawn telnet $device
    expect "Password:"
    send "$pass\n"
    expect ">"
    send "en\n"
    expect "Password:"
    send "$pass\n"
    
    send "copy running-config tftp://$tftp/$config\n\n"
    expect "$tftp"
    send "\n"
    expect "$config"
    send "\n"
    send "exit\n"

    Save it anywhere you like and run it from the shell. You’ll see something like this in your logs:

    user in.tftpd[22304]: connect from 192.168.0.200 (192.168.0.200)
    user tftpd[22305]: tftpd: trying to get file: config
    user tftpd[22305]: tftpd: serving file from /tftpboot

    That’s it. Your current Cisco config has been saved to /tftpboot/config.

    I wouldn’t recommend using this into production without proper firewalling. You can get the same results by using snmp. But that’s however a subject for another howto.

    Monitor DNS blacklist entries with Zabbix

    One of the smaller projects I’ve been working on lately is monitoring Realtime DNS Blacklists (RBL’s) status with Zabbix. I’m confident most of you are already familiar with RBL’s. For those who are not, here’s a small introduction shamelessly stolen from Wikipedia:

    A DNSBL (DNS-based Blackhole  List, Block List, or Blacklist; see below) is a list of IP addresses published through the Internet Domain Name Service in a particular format. DNSBLs are most often used to publish the addresses of computers or networks linked to spamming; most mail server  software can be configured to reject or flag messages which have been sent from a site listed on one or more such lists.

    And that’s exactly what we’re going to monitor. If we are listed on one of those RBL’s we’d like to know about it, don’t we? 

    Let’s get to it then. First of all we need an up to date list of RBL’s which we can use to check whether we’re listed or not. You could try the list I’m maintaining and using for my own monitoring purposes. The most recent version can be found here. It contains a whopping 92 RBL’s to get you started with.

    Now that we have an up-to-date list of common used RBL’s it’s time for some shell scripting:

    #!/bin/bash
    
    ## TomDV
    ## 2010-01-25
    ## http://blog.penumbra.be/2010/02/zabbix-monitor-dns-blacklists/
    
    cd /usr/share/zabbix/
    RBL="`cat rbl_list.txt`"
    
    W=$( echo ${1} | cut -d. -f1 )
    X=$( echo ${1} | cut -d. -f2 )
    Y=$( echo ${1} | cut -d. -f3 )
    Z=$( echo ${1} | cut -d. -f4 )
    
    STATUS=0
    
    for i in $RBL
    do
        RESULT=$( host -t a $Z.$Y.$X.$W.$i 2>&1 )
        if [ $? -eq 0 ]
        then
            #echo “The IP ADDRESS ${1} is listed at $i:\n$RESULT” ## DEBUG
            let "STATUS += 1"
        fi
        #echo $RESULT ## DEBUG
    done
    
    if [ $STATUS -lt 1 ]
    then
        echo 0
    else
        echo $STATUS
    fi

    This script takes the IP address of your server as input.

    I’ve intentionally left the debug code inside the script. This way the output can be used right away within Zabbix. However if you’re listed on one of the blacklists you can run the script with the debug code uncommented and you get a list of all the RBL’s you’re listed in.

    I’ve put this script in /usr/share/zabbix, along with the rbl_list.txt file you can find above.

    # cat /etc/zabbix/zabbix_agent.d/rbl.conf
    UserParameter=rbl.mx1,/usr/share/zabbix/zabbix-rbl.sh 1.2.3.4
    UserParameter=rbl.mx2,/usr/share/zabbix/zabbix-rbl.sh 5.6.7.8

    I also have the following line in /etc/zabbix/zabbix_agentd.conf and /etc/zabbix/zabbix_agent.conf to load custom config files:

    Include=/etc/zabbix/zabbix_agent.d/

    And that’s about it. Let’s see if we’re listed in any of the RBL’s:

    # zabbix_agent -t rbl.mx1; zabbix_agent -t rbl.mx2;
    rbl.mx1                                    [t|0]
    rbl.mx2                                    [t|0]

    Any value above zero means you’re listed. I guess we’re safe.
    If you’re listed just uncomment the debug code. It will show you which RBL’s you’re in.

    Happy monitoring! :)

    Install Xen and libvirt on Debian Lenny

    This should be an easy to follow guide about how to install Xen on Debian 5. You should be able to copy/paste most parts of on your shell. Please run this only on a clean and up-to-date Debian system.

    Alright let’s get to it quick ‘n dirty.

    First of all make sure your Debian install actually is up-to-date:

    # apt-get update; apt-get upgrade

    Let’s see which Xen kernel images are available and pick the most recent one to install:

    # apt-cache search xen | grep image | awk '{print $1}'
    linux-image-2.6-xen-amd64
    linux-image-xen-amd64
    linux-image-2.6.26-1-xen-amd64
    linux-image-2.6.26-2-xen-amd64
    xen-linux-system-2.6.26-1-xen-amd64
    xen-linux-system-2.6.26-2-xen-amd64
    
    # apt-get install `apt-cache search xen-linux-system \
    | sort | tail -1 | awk '{print $1}'`

    Once this is done reboot your system, login again and run:

    # uname -a
    Linux elysium 2.6.26-2-xen-amd64 #1 SMP Thu Feb 11 02:57:18 UTC 2010 x86_64 GNU/Linux

    As you can see, the system is running kernel 2.6.26-2 with the xen-amd64 patch set.
    As of now we should have Dom-0 available:

    # xm list
    Name                   ID   Mem VCPUs      State   Time(s)
    Domain-0               0  3885     2     r-----      8.5

    Perfection!

    Let’s move on to the network. By default there is no bridge available from the virtual machines towards the external network. It’s fairly easy to accomplish though:

    # vim /etc/xen/xend-config.sxp

    Look for the following line and uncomment it:

    (network-script network-bridge)

    And while we’re editing the xend-config.sxp file, change the following line:

    (xend-unix-server no)

    Into:

    (xend-unix-server yes)

    Be sure to reload the new settings:

    # /etc/init.d/xend restart

    If you don’t edit this line or if you don’t reload, you obviously won’t be able to install or manage your virtual machines. You’d get to see errors like this:

    ERROR    internal error failed to connect to xend
    Traceback (most recent call last):
      File "/usr/bin/virt-install", line 693, in
        main()
      File "/usr/bin/virt-install", line 508, in main
        conn = cli.getConnection(options.connect)
      File "/var/lib/python-support/python2.5/virtinst/cli.py", line 123, in getConnection
        return libvirt.open(connect)
      File "/usr/lib/python2.5/site-packages/libvirt.py", line 140, in open
        if ret is None:raise libvirtError('virConnectOpen() failed')
    libvirtError: internal error failed to connect to xend

    There are more options available to connect to xend.: xend-http-server, xend-tcp-xmlrpc-server, xend-unix-xmlrpc-server, xend-relocation-server.

    The reason why I chose xend-unix-server over anything else is pretty straight forward. It only listens on the Unix socket layer which doesn’t need any other networking protocol to operate. You could compare it to connecting to localhost, but without the need for a networking device (e.g. lo0).

    If you want to be able to manage this Xen server from a central node, be sure to change the appropriate management protocol. It’s beyond the scope of this howto, but I might post an howto on this subject later on this blog.

    On to installing a virtual machine!

    I for one am a big fan of abstraction layers. It makes life for a SysAdmin or DevOp so much easier if you’re able to run the same commands on different operating systems, platforms or architectures in general. That’s why I use libvirt. It’s a collection of libraries and tools that can be used to deploy virtual machines on different types of virtualization systems. Including but not limited to Xen, KVM and Qemu.

    It’s also available as a Debian package, so installing it is very straight forward:

    # apt-get install libvirt-bin virtinst

    Once this is done we can install our first virtual machine:

    # virt-install \
    --name=test-debian-install \
    --ram=1024 \
    --file-size=10 \
    --nographics \
    --paravirt \
    --file=/var/lib/xen/images/test-debian-install.img \
    --location=http://ftp.belnet.be/debian/dists/lenny/main/\
    installer-i386

    It’s probably a good idea to store this as a shell script on your Xen host for future reference.
    You should see a familiar installer within seconds after invoking the command.

    Once the install has completed you should be greeted with your new virtual machine’s login prompt:

    Debian GNU/Linux 5.0 test-debian-install hvc0
    
    test-debian-install login:

    To exit your virtual machine’s console, simply press Ctrl-]

    Enjoy!

    Looking for open source projects that need help with packaging

    In follow up to a friend’s recent blogpost “Bored Java Dev looking for Open Source project” I’m also looking for an open source project to contribute to. I’m not that much of a developer but I’d like to get more familiar with Linux distribution packaging. I have basic experience creating
    Gentoo ebuilds, Debian DEB and CentOS RPM packages, but I want to learn and to get more involved.

    Anyone with a promising new open source project feel free to send me a request at
    tom [at] penumbra.be. I do however have some prerequisites:

    • Free and Open Source Software only, no exceptions
    • Non-commercial projects only
    • Preferably not limited to one (Linux) distribution
    • No Qt (KDE) applications due to personal preferences

    What I can offer:

    • Spare time
    • Dedication
    • Build farm on x86, x86-64 and UltraSparc64

    What I can’t offer:

    Zabbix 1.8 on CentOS 5

    For those who want or need to build Zabbix 1.8 on CentOS 5: there is an excellent RPM Spec file available at Andrew Farley’s blog. He’s also been kind enough to host a series of precompiled RPM packages.

    If you look at the changelog you might find yours truly. I’ve contributed a patch to add a couple of dependencies and to fix a couple of bugs. So I thought I should share.

    First of all the RPM Spec file:

    http://repo.andrewfarley.com/centos/specs/zabbix.spec

    If you’d like to compile your own packages you might follow this guide:

    # yum groupinstall "Development Tools"
    # yum install rpmdevtools
    # rpmdev-setuptree

    To compile for your running architecture:

    # rpmbuild -bb --clean zabbix.spec

    Or if you’d like to build for a specific architecture:

    # rpmbuild -bb --clean --target i686 zabbix.spec

    After the compile process you’ll find the RPM files in the following directory:

    ~/rpmbuild/RPMS/zabbix*.rpm

    NTFS-3G on CentOS 5

    Lately I’ve seen some customers struggle with their external USB drives formatted in NTFS on CentOS servers. Because it’s such a common problem I decided to make a very quick howto on the subject.

    The Fuse packages found in the default CentOS repository haven’t been compiled with NTFS-3G support. Even though there are fuse-ntfs3g packages available! To get around this I prefer to enable the RPM Forge repository. It’s very easy to do so.

    On 32bit platforms:

    # wget http://apt.sw.be/redhat/el5/en/i386/RPMS.dag/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
    # rpm -i rpmforge*i386.rpm

    On 64bit platforms:

    # wget http://apt.sw.be/redhat/el5/en/x86_64/RPMS.dag/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
    # rpm -i rpmforge*x86_64.rpm

    Moving on:

    # rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt
    # yum update

    Installing RPM Forge’s Fuse packages and dependencies:

    # yum install -y kernel-devel kernel-headers
    # yum install -y --enablerepo=rpmforge dkms dkms-fuse fuse
    # yum install -y --enablerepo=rpmforge fuse-ntfs-3g-devel fuse-devel

    For some strange reason, the Fuse module in the package isn’t compiled yet. Lucky for us the package maintainer has provided the source code for us:

    # cd /usr/src/fuse-2.7.4-1.nodist.rf/
    # ./configure
    # make
    # make install

    And now the part most people seem to forget, insert the Fuse module into the kernel:

    # insmod /lib/modules/2.6.18-164.9.1.el5/kernel/fs/fuse/fuse.ko

    Mount the external disk:

    # mount -t ntfs-3g /dev/sdc1 /mnt/usb/
    # df -h
    Filesystem            Size  Used Avail Use% Mounted on
    ...
    /dev/sdc5             466G   79M  466G   1% /mnt/usb

    That’s all there’s to it!

    [root@srv01 src]# yum install -y –enablerepo=rpmforge dkms dkms-fuse fuse
    Thats
    [root@srv01 src]# yum install -y –enablerepo=rpmforge fuse-ntfs-3g-devel.i386 fuse-devel.i386

    Apache mod_evasive DDoS prevention on a CentOS 5.x Plesk environment

    A couple weeks ago I was asked to implement a DDoS prevention system for a customer who had been suffering some DDoS events.
    Shouldn’t be too hard. The only catch was that the box was running a Plesk 9 LAMP stack.
    I chose to go with mod_evasive, a GPL2 licensed module for Apache[1-2].
    It can be downloaded at http://www.zdziarski.com/blog/?page_id=442 either using the CVS repository or as a tarball.
    I ended up using the latter. The current stable version at this point is 1.10.1.
    This guide has been assembled using Centos 5.4 with a Plesk 9 LAMP stack. I haven’t tested it on anything else, but it should work just the same way it does in this guide.
    ===== Install =====
    First things first. Let’s check out which versions of apxs are installed:
    <code>
    # updatedb; locate apxs | grep bin
    /usr/local/psa/admin/bin/apxs
    </code>
    This version is Parallell’s default version, wich comes with Plesk.
    If this is the only version you have available you will need to install the generic httpd-devel package. Parallell’s version of apxs is a bit limited and won’t compile the module.
    <code>
    # yum install httpd-devel
    </code>
    Give it another go and you should end up with something like this:
    <code>
    # updatedb; locate apxs | grep bin
    /usr/local/psa/admin/bin/apxs
    /usr/sbin/apxs
    </code>
    Onto downloading and extracting the mod_evasive module:
    <code>
    # cd /usr/src
    # wget http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
    # tar xvzf mod_evasive_1.10.1.tar.gz
    mod_evasive/
    mod_evasive/.cvsignore
    mod_evasive/LICENSE
    mod_evasive/Makefile.tmpl
    mod_evasive/README
    mod_evasive/mod_evasive.c

    A couple weeks ago I was asked to implement a DDoS prevention system in Apache for a customer who had obviously been suffering some gnarly DDoS events. Shouldn’t be too hard. The only catch was that the box was running a Plesk 9 LAMP stack.

    I chose to go with mod_evasive, a GPL2 licensed module for Apache[1-2]. It can be downloaded at http://www.zdziarski.com/projects/mod_evasive/ either using the CVS repository or as a tarball. I ended up using the latter. The current stable version at this point is 1.10.1.

    This guide has been assembled using Centos 5.4 with a Plesk 9 LAMP stack. I haven’t tested it on anything else, but it should work just the same way it does in this guide.

    Install

    First things first. Let’s check out which versions of apxs are installed:

    # updatedb; locate apxs | grep bin
    /usr/local/psa/admin/bin/apxs

    This version is Parallell’s default version, wich comes with Plesk.

    If this is the only version you have available you will need to install the generic httpd-devel package. Parallell’s version of apxs is a bit limited and won’t compile the module.

    # yum install httpd-devel

    Give it another go and you should end up with something like this:

    # updatedb; locate apxs | grep bin
    /usr/local/psa/admin/bin/apxs
    /usr/sbin/apxs
    
    

    Onto downloading and extracting the mod_evasive module:

    # wget http://www.zdziarski.com/projects/mod_evasive/mod_evasive_1.10.1.tar.gz
    # tar xvzf mod_evasive_1.10.1.tar.gz mod_evasive/
    mod_evasive/.cvsignore
    mod_evasive/LICENSE
    mod_evasive/Makefile.tmpl
    mod_evasive/README
    mod_evasive/mod_evasive.c
    mod_evasive/mod_evasive20.c
    mod_evasive/mod_evasiveNSAPI.c
    mod_evasive/test.pl
    mod_evasive/CHANGELOG

    Be sure to check out the CHANGELOG and README files!

    Even though people tend to forget this step… those files are included for a reason.
    Let’s move on to compiling and actually installing the module inside the Plesk chroot:

    # /usr/sbin/apxs -cia /usr/src/mod_evasive/mod_evasive20.c
    /usr/lib/apr-1/build/libtool --silent --mode=compile gcc -prefer-pic -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions \
    -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -fno-strict-aliasing \
    -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -pthread -I/usr/include/httpd  -I/usr/include/apr-1   -I/usr/include/apr-1 \
    -c -o mod_evasive20.lo mod_evasive20.c && touch mod_evasive20.slo
    mod_evasive20.c: In function 'access_checker':
    mod_evasive20.c:212: warning: implicit declaration of function 'getpid'
    mod_evasive20.c:212: warning: format '%ld' expects type 'long int', but argument 4 has type 'int'
    mod_evasive20.c:229: warning: ignoring return value of 'system', declared with attribute warn_unused_result
    mod_evasive20.c: In function 'destroy_hit_list':
    mod_evasive20.c:301: warning: control reaches end of non-void function
    mod_evasive20.c: In function 'create_hit_list':
    mod_evasive20.c:118: warning: control reaches end of non-void function
    /usr/lib/apr-1/build/libtool --silent --mode=link gcc -o mod_evasive20.la  -rpath /usr/lib/httpd/modules -module -avoid-version    mod_evasive20.lo
    /usr/lib/httpd/build/instdso.sh SH_LIBTOOL='/usr/lib/apr-1/build/libtool' mod_evasive20.la /usr/lib/httpd/modules
    /usr/lib/apr-1/build/libtool --mode=install cp mod_evasive20.la /usr/lib/httpd/modules/
    cp .libs/mod_evasive20.so /usr/lib/httpd/modules/mod_evasive20.so
    cp .libs/mod_evasive20.lai /usr/lib/httpd/modules/mod_evasive20.la
    cp .libs/mod_evasive20.a /usr/lib/httpd/modules/mod_evasive20.a
    chmod 644 /usr/lib/httpd/modules/mod_evasive20.a
    ranlib /usr/lib/httpd/modules/mod_evasive20.a
    PATH="$PATH:/sbin" ldconfig -n /usr/lib/httpd/modules
    ----------------------------------------------------------------------
    Libraries have been installed in:
    /usr/lib/httpd/modules
    If you ever happen to want to link against installed libraries
    in a given directory, LIBDIR, you must either use libtool, and
    specify the full pathname of the library, or use the `-LLIBDIR'
    flag during linking and do at least one of the following:
    - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
    during execution
    - add LIBDIR to the `LD_RUN_PATH' environment variable
    during linking
    - use the `-Wl,--rpath -Wl,LIBDIR' linker flag
    - have your system administrator add LIBDIR to `/etc/ld.so.conf'
    See any operating system documentation about shared libraries for
    more information, such as the ld(1) and ld.so(8) manual pages.
    ----------------------------------------------------------------------
    chmod 755 /usr/lib/httpd/modules/mod_evasive20.so
    [activating module `evasive20' in /etc/httpd/conf/httpd.conf]

    Next up we need to restart apache to load the module:

    # /etc/init.d/httpd restart
    Stopping httpd:                                            [  OK  ]
    Starting httpd:                                            [  OK  ]

    Verify

    Verify if the module is in the apache config:

    # grep -i evasive /etc/httpd/conf/httpd.conf
    LoadModule evasive20_module   /usr/lib/httpd/modules/mod_evasive20.so

    Check wether the modtule is actually loaded:

    # php -r 'phpinfo();' | grep -i evasive
    ^ Loaded Modules | core prefork http_core mod_so mod_auth_basic
    mod_auth_digest mod_authn_file mod_authn_alias mod_authn_anon
    mod_authn_dbm mod_authn_default mod_authz_host mod_authz_user
    mod_authz_owner mod_authz_groupfile mod_authz_dbm mod_authz_default
    util_ldap mod_authnz_ldap mod_include mod_log_config mod_logio
    mod_env mod_ext_filter mod_mime_magic mod_expires mod_deflate
    mod_headers mod_usertrack mod_setenvif mod_mime mod_dav mod_status
    mod_autoindex mod_info mod_dav_fs mod_vhost_alias mod_negotiation
    mod_dir mod_actions mod_speling mod_userdir mod_alias mod_rewrite
    mod_proxy mod_proxy_balancer mod_proxy_ftp mod_proxy_http mod_proxy_connect
    mod_cache mod_suexec mod_disk_cache mod_file_cache mod_mem_cache mod_cgi
    mod_version **mod_evasive20** mod_perl mod_php5 mod_proxy_ajp mod_python mod_ssl |

    Seems it’s loaded just fine.

    Configure

    Now let’s get started with the configuration. I couldn’t find any default config, but this one seems to run just fine. Even on a heavily visited shared hosting server.

    Add the following rules at the end of /etc/httpd/conf/httpd.conf:

    <IfModule mod_evasive20.c>
    DOSHashTableSize 3097
    DOSPageCount 6
    DOSSiteCount 100
    DOSPageInterval 2
    DOSSiteInterval 2
    DOSBlockingPeriod 600
    </IfModule>

    And let’s kick apache one last time:

    # /etc/init.d/httpd restart
    Stopping httpd:                                            [  OK  ]
    Starting httpd:                                            [  OK  ]

    NOTE

    Be sure to keep an eye on your webstats!

    There might be a sudden drop in the amount of unique visitors. This might be a result of an attack that’s been evaded. However if you’ve used different configuration parameters you might have restricted it too much and you’ll end up restricting valid customers too. I haven’t recieved any negative comments about this setup (yet?)

    Use with caution!

    pe·num·bra (pĭ-nŭm′brə) n. pl. pe·num·brae (-brē) or pe·num·bras 1. A partial shadow, as in an eclipse, between regions of complete shadow and complete illumination. See Synonyms at shade. 2. The grayish outer part of a sunspot.