Hazard's stuff

OSPF: a protocol from hell, or Type 5 vs Type 7 LSAs

I've always thought that OSPF was one of the rotten tomatoes in the generally nicely-looking suite of core IP protocols. Its architecture is simply not suited well for real-life carrier networks. OSPF tries to impose its own rules on how the network should be built, instead of providing flexibility to adjust the protocol to the network. However, instead of throwing OSPF out and concentrating on a better IGP (e.g. something EIGRP-like), Internet/IETF community kept adding more and more band-aids to it. As a result, we've got a number of monstrous and complex specifications, which even vendors don't fully understand, resulting in bugs and incompatibilities between implementations, as well as lots of confusion to engineers.

A few days ago I got hit by another case which re-enforced my beliefs. I had to implement an OSPF network which had both Type 5 LSA E2 and Type 7 N2 routes for the same prefix. Moreover, there was a mix of IOS and JUNOS speakers in the same network. So, what is the route selection algorithm in that scenario? Google and you'll find at least three different answers. An incorrect answer from one of very reputable sources will say that E2 route will win over N2 no matter what the cost is. Another answer is that E2 route will be preferred if it has the same or lower metric than N2 route. And the third answer is the opposite: N2 route will win over E2 if it has the same or lower metric.

Well, to ease the pain of future generations, I'll say that the correct answer is that N2 route will win unless E2 has lower cost. That is, unless that future generation lives in a world where they have released an RFC to supersede RFC 3101, since things might change the same way as they did since RFC 1587, which apparently specified opposite behavior. If you're using Cisco IOS, it depends on which IOS you use, many IOSes will prefer E2, as they follow the behavior from RFC 1587, even though they were released much later than RFC 3101. Other IOSes will prefer N2.

I rest my case.

Long live OSPF, the reason for late-night maintenance headaches! As well as an additional source of revenue for network engineer certifications. :)

Forcing Fedora's preupgrade to use servers in Europe instead of Asia

I've decided to upgrade my FC14 to FC16. Along the way, I decided to do it using a method that I've never used before - preupgrade. Supposedly it's one of the easiest and less time consuming methods. Not in Cyprus ...

How the heck is Cyprus related to preupgrade, you ask yourself? Well, it turns out that Fedora's mirror system assigns Cyprus to Asia. That might be correct in the strictest geographical sense (politically Cyprus is in European Union), but it is completely incorrect in terms of server proximity. Entire mirror list consists of servers in China and Japan (so fastestmirror doesn't help), while Cyprus' upstream Internet connectivity goes to West Europe. All of this results in very unreliable downloads at 8 kilobytes per second, which would require 10 days of waiting at best ...

So, I embarked on a quest to figure out how to force preupgrade to use European servers. After a few hours, extensive strace and several damaged rpm databases, I figured out that you need to do the following:

cd /usr/share/preupgrade
cp releases.list releases.txt
Edit releases.txt, add &country=de to all URLs (de stands for Germany, replace according to your preference)
Delete all mirrorlist.txt files under /var/cache/yum subdirectories
Run "preupgrade" from the same directory where releases.txt is located. E.g. /usr/share/preupgrade in context of my example.

preupgrade failed anyway because it needs 120 free space on the boot partition. But it's another story. :-)

In the meantime, I submitted bug #3086 to Fedora Infrastructure team. Let's hope they take a look at it.

nginx as protection against DDoS to Apache

A few days ago I was asked to help with a DDoS attack against a website. The DDoS itself was pretty generic, a small zombie network hammering particular URLs from the websites with GET requests. The websites were running on Apache, and despite that the target page was static, the DDoS was bringing Apache to its knees. System administrators tried to utilize various Apache modules and configuration tricks to protect against DDoS, but to no avail.

There was only one solution to this on my mind - install nginx. And that really helped. nginx is asynchronous by design and therefore handles load much much better. Whilst Apache was failing with several hundred simultaneous connections, nginx easily scaled to 10k caused by DDoS, whilst using only 20% CPU.

The first website was completely moved to nginx, with PHP being served through PHP/FastCGI. For the second website, the nginx was configured in proxy mode, so that it would forward all requests to the Apache, whilst enforcing limits against DDoS - 1 unique page request per IP per second, as well as blocking certain user agents. Below is an example configuration I created, relevant for CentOS/RHEL.


# For better scalability - set this to the number of CPU cores
worker_processes  1;

events {
    # Max number of connections per workers
    worker_connections  16384;
}

http {
    # This is a classifier which will enforce one request per IP per sec. 
    # It is applied to individual locations later.
    limit_req_zone  $binary_remote_addr  zone=ddos:30m   rate=1r/s;

    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen   80;
        # 1st website. This is a default webserver (_).
        # Runs completely from nginx.
        # PHP is served via PHP/FastCGI.
        server_name _;

        root /var/www/default;
        index  index.html;

        # Request-range protection fix using a regexp.
        if ($http_range ~ "(?:d*s*-s*d*s*,s*){5,}") {
                return 416;
        }

        location = / {   
            limit_req zone=ddos;
            expires 20m;
        }


        location ~ .php$ {
            # Forward all .php pages to PHP/FastCGI on port 9000. 
            root           html;
            fastcgi_pass   127.0.0.1:9000;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  /var/www/default/$fastcgi_script_name;
            include        fastcgi_params;
        }

        location / {
            expires 20m;
        }
    }

    server {
        listen   80;
        # Second website, responding to particular domains only.
        server_name [domain1] [domain2];
        root /var/www/secondweb;

        # Request-range protection fix.
        if ($http_range ~ "(?:d*s*-s*d*s*,s*){5,}") {
                return 416;
        }

        location = / {
            # Block particular user agent which was used for DDoS
            if ($http_user_agent ~* Opera) {
                return 403;
            }
            
            limit_req zone=ddos;
            expires 20m;

            index  index.html;
        }

        # Serve index.html statically, and limit it with anti-DDoS.
        # Note that = means full match only. 
        location = /index.html {    
            limit_req zone=ddos;
            expires 20m;

            index  index.html;
        }


        # Location without = means that any child URLs are also matched.
        location / {
            # Forward everything to upstream Apache server on port 8080.

            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_connect_timeout 10;
            proxy_read_timeout 10;
            proxy_set_header Range "";
            proxy_pass http://localhost:8080/;
            # if you want, limit it with anti-DDoS as well:
            # limit_req zone=ddos;
        }
    }
}

Now, you might want to know to how to start PHP/FastCGI on CentOS. You'll need a recent 5.x PHP for that. Below is an init script for CentOS, obviously you need to put it into /etc/init.d and configure using chkconfig to start on boot.


#!/bin/sh
#
# php-cgi - php-fastcgi swaping via  spawn-fcgi
#
# chkconfig:   - 85 15
# description:  Run php-cgi as app server
# processname: php-cgi
# config:      /etc/sysconfig/phpfastcgi (defaults RH style)
# pidfile:     /var/run/php_cgi.pid
# Note: See how to use this script :
# http://www.cyberciti.biz/faq/rhel-fedora-install-configure-nginx-php5/
# Source function library.
. /etc/rc.d/init.d/functions
 
# Source networking configuration.
. /etc/sysconfig/network
 
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
 
spawnfcgi="/usr/bin/spawn-fcgi"
php_cgi="/usr/bin/php-cgi"
prog=$(basename $php_cgi)
server_ip=127.0.0.1
server_port=9000
server_user=nobody
server_group=nobody
server_childs=5
pidfile="/var/run/php_cgi.pid"
 
# do not edit, put changes in /etc/sysconfig/phpfastcgi
[ -f /etc/sysconfig/phpfastcgi ] && . /etc/sysconfig/phpfastcgi
 
start() {
    [ -x $php_cgi ] || exit 1
    [ -x $spawnfcgi ] || exit 2
    echo -n $"Starting $prog: "
    daemon $spawnfcgi -a ${server_ip} -p ${server_port} -u ${server_user} 
           -g ${server_group} -P ${pidfile} -C ${server_childs} -f ${php_cgi}
    retval=$?
    echo
    return $retval
}
 
stop() {
    echo -n $"Stopping $prog: "
    killproc -p ${pidfile} $prog -QUIT
    retval=$?
    echo
    [ -f ${pidfile} ] && /bin/rm -f ${pidfile}
    return $retval
}
 
restart(){
        stop
        sleep 2
        start
}
 
rh_status(){
        status -p ${pidfile} $prog
}
 
case "$1" in
    start)
        start;;
    stop)
        stop;;
    restart)
        restart;;
    status)
        rh_status;;
    *)
        echo $"Usage: $0 {start|stop|restart|status}"
        exit 3
esac

And the last point you might want to configure: with nginx as front-end (reverse proxy), Apache will show all requests as coming from localhost, and PHP will also think that everything is coming from localhost. nginx will be passing the real IP using X-Real-IP header, and you can use the following Apache configuration to rewrite the remote client IP to the real one:

LoadModule rpaf_module modules/mod_rpaf-2.0.so

    RPAFenable On
    RPAFsethostname On
    RPAFproxy_ips 127.0.0.1
    RPAFheader X-Real-IP

You need to install Apache's mod_rpaf for this to work. You can find an RPM for it in various repositories.

Fixing Greenplum 'unresolved in-doubt transaction' errors

We had an issue with a database server running Greenplum (commercial Postgresql for large-scale datawarehousing). Greenplum was starting, but attempts to do anything in the database were resulting in the following errors:

INFO: Crash recovery broadcast of the distributed transaction 'Commit Prepared' broadcast succeeded for gid = 1265880453-0032866370.
INFO: Crash recovery broadcast of the distributed transaction 'Abort Prepared' broadcast succeeded for gid = 1265880453-0032866371��C
psql: FATAL: DTM Log recovery failed. There are still unresolved in-doubt transactions on some of the segment databaes that were not able to be resolved for an unknown reason. (cdbtm.c:2829)
DETAIL: Here is a list of in-doubt transactions in the system: List of In-doubt transactions remaining across the segdbs: ("1265880453-0032866371��C" , )
HINT: Try restarting the Greenplum Database array. If the problem persists an Administrator will need to resolve these transactions manually.

Of course, manuals/forums/Google did not provide any useful ideas as regards to how 'to resolve these transactions manually'. Morever there was no backup handy (and the db was huge). I didn't care about two lost transactions, I just wanted to start the database. After an hour of attempts, eventually I succeeded. The trick was to delete files from pg_twophase/ subdirectories.

I'm blogging this in the hope that when somebody else faces this problem, he would be able to find this post through Google, saving his nerves. :-)

Guaba Beach Bar Limassol

Great music but a boo for customer service. I don't like being told "I don't give a shit" by supervisors. Having spent there a few hundred euros in a week, the least I want is to be handled politely. Fix yourself or you'll end up only with drunken kids. The atmosphere has been degrading ever since the move to a new place.

What a wonderful day ... NOT

It's been a while I posted something here, so as well it might be a rant.

TODAY:

Confortel Atrium hotel in Madrid tried to put 60 EUR worth of bar/restaurant charges on me, ignoring the fact that they were dated BEFORE the date I arrived.
Aegean airlines screwed up web check-in Madrid-Athens-Larnaca. I managed to check in for Madrid-Athens, but then the system would not let me check-in to Larnaca.
Aegean airlines did not manage to register me to Larnaca even in Madrid airport registration desk. "We use different systems". It should be mentioned that my transfer time in Athens was planned to be just one hour, which is already tight, without having to obtain a boarding pass...
... and then the flight to Athens was delayed.
The duty-free in Madrid didn't put the bottle of wine I purchased in a sealed bag. So the bottle was happily confiscated by Athens airport security.
My passport got damaged (peeled off) just on the place where my photo is.
It seems I caught a cold.

Well, it could be worse, right ??

Kernel 2.6.28 for Fedora Core 8

In case someone wants to run a recent Linux kernel on an FC8 box, I have made an RPM for 2.6.28.7 and you can download it here. Should also install on CentOS 5/RHEL 5 if you use --force.

Ussuriysk space center

This breathtaking photo was taken next to a space communication facility near Ussuriysk, Russia. The biggest RT-70 radiotelescope has a 70m diameter antenna with an area of 2500 sq. m! The telescope stands over 100m high, can rotate 360 degrees and weights over 24 thousand tons [1] [2]. Impressive gadget :) Sadly, not used nowadays, due to lack of financing and deep space missions.

Waltz with Bashir

Just finished watching Waltz with Bashir. Emotional, original and very beautiful. Not for the faint of heart.

I hope that it will get the well-deserved Oscar in the best foreign movie category.

You think that SPAM is distributed? I did so too.

Shutdown of a single ISP, McColo, has reduced world spam levels by 70%. Check the full story here. Amazing.