Hazard's stuff

28 Aug, 2011

nginx as protection against DDoS to Apache

— Posted by hazard @ 2011-08-28 18:33
A few days ago I was asked to help with a DDoS attack against a website. The DDoS itself was pretty generic, a small zombie network hammering particular URLs from the websites with GET requests. The websites were running on Apache, and despite that the target page was static, the DDoS was bringing Apache to its knees. System administrators tried to utilize various Apache modules and configuration tricks to protect against DDoS, but to no avail.

There was only one solution to this on my mind - install nginx. And that really helped. nginx is asynchronous by design and therefore handles load much much better. Whilst Apache was failing with several hundred simultaneous connections, nginx easily scaled to 10k caused by DDoS, whilst using only 20% CPU.

The first website was completely moved to nginx, with PHP being served through PHP/FastCGI. For the second website, the nginx was configured in proxy mode, so that it would forward all requests to the Apache, whilst enforcing limits against DDoS - 1 unique page request per IP per second, as well as blocking certain user agents. Below is an example configuration I created, relevant for CentOS/RHEL.



# For better scalability - set this to the number of CPU cores
worker_processes  1;

events {
    # Max number of connections per workers
    worker_connections  16384;
}

http {
    # This is a classifier which will enforce one request per IP per sec. 
    # It is applied to individual locations later.
    limit_req_zone  $binary_remote_addr  zone=ddos:30m   rate=1r/s;

    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    keepalive_timeout  65;

    server {
        listen   80;
        # 1st website. This is a default webserver (_).
        # Runs completely from nginx.
        # PHP is served via PHP/FastCGI.
        server_name _;

        root /var/www/default;
        index  index.html;

        # Request-range protection fix using a regexp.
        if ($http_range ~ "(?:d*s*-s*d*s*,s*){5,}") {
                return 416;
        }

        location = / {   
            limit_req zone=ddos;
            expires 20m;
        }


        location ~ .php$ {
            # Forward all .php pages to PHP/FastCGI on port 9000. 
            root           html;
            fastcgi_pass   127.0.0.1:9000;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  /var/www/default/$fastcgi_script_name;
            include        fastcgi_params;
        }

        location / {
            expires 20m;
        }
    }

    server {
        listen   80;
        # Second website, responding to particular domains only.
        server_name [domain1] [domain2];
        root /var/www/secondweb;

        # Request-range protection fix.
        if ($http_range ~ "(?:d*s*-s*d*s*,s*){5,}") {
                return 416;
        }

        location = / {
            # Block particular user agent which was used for DDoS
            if ($http_user_agent ~* Opera) {
                return 403;
            }
            
            limit_req zone=ddos;
            expires 20m;

            index  index.html;
        }

        # Serve index.html statically, and limit it with anti-DDoS.
        # Note that = means full match only. 
        location = /index.html {    
            limit_req zone=ddos;
            expires 20m;

            index  index.html;
        }


        # Location without = means that any child URLs are also matched.
        location / {
            # Forward everything to upstream Apache server on port 8080.

            proxy_pass_header Server;
            proxy_set_header Host $http_host;
            proxy_redirect off;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Scheme $scheme;
            proxy_connect_timeout 10;
            proxy_read_timeout 10;
            proxy_set_header Range "";
            proxy_pass http://localhost:8080/;
            # if you want, limit it with anti-DDoS as well:
            # limit_req zone=ddos;
        }
    }
}


Now, you might want to know to how to start PHP/FastCGI on CentOS. You'll need a recent 5.x PHP for that. Below is an init script for CentOS, obviously you need to put it into /etc/init.d and configure using chkconfig to start on boot.

#!/bin/sh
#
# php-cgi - php-fastcgi swaping via  spawn-fcgi
#
# chkconfig:   - 85 15
# description:  Run php-cgi as app server
# processname: php-cgi
# config:      /etc/sysconfig/phpfastcgi (defaults RH style)
# pidfile:     /var/run/php_cgi.pid
# Note: See how to use this script :
# http://www.cyberciti.biz/faq/rhel-fedora-install-configure-nginx-php5/
# Source function library.
. /etc/rc.d/init.d/functions
 
# Source networking configuration.
. /etc/sysconfig/network
 
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
 
spawnfcgi="/usr/bin/spawn-fcgi"
php_cgi="/usr/bin/php-cgi"
prog=$(basename $php_cgi)
server_ip=127.0.0.1
server_port=9000
server_user=nobody
server_group=nobody
server_childs=5
pidfile="/var/run/php_cgi.pid"
 
# do not edit, put changes in /etc/sysconfig/phpfastcgi
[ -f /etc/sysconfig/phpfastcgi ] && . /etc/sysconfig/phpfastcgi
 
start() {
    [ -x $php_cgi ] || exit 1
    [ -x $spawnfcgi ] || exit 2
    echo -n $"Starting $prog: "
    daemon $spawnfcgi -a ${server_ip} -p ${server_port} -u ${server_user} 
           -g ${server_group} -P ${pidfile} -C ${server_childs} -f ${php_cgi}
    retval=$?
    echo
    return $retval
}
 
stop() {
    echo -n $"Stopping $prog: "
    killproc -p ${pidfile} $prog -QUIT
    retval=$?
    echo
    [ -f ${pidfile} ] && /bin/rm -f ${pidfile}
    return $retval
}
 
restart(){
        stop
        sleep 2
        start
}
 
rh_status(){
        status -p ${pidfile} $prog
}
 
case "$1" in
    start)
        start;;
    stop)
        stop;;
    restart)
        restart;;
    status)
        rh_status;;
    *)
        echo $"Usage: $0 {start|stop|restart|status}"
        exit 3
esac

And the last point you might want to configure: with nginx as front-end (reverse proxy), Apache will show all requests as coming from localhost, and PHP will also think that everything is coming from localhost. nginx will be passing the real IP using X-Real-IP header, and you can use the following Apache configuration to rewrite the remote client IP to the real one:
LoadModule rpaf_module modules/mod_rpaf-2.0.so

    RPAFenable On
    RPAFsethostname On
    RPAFproxy_ips 127.0.0.1
    RPAFheader X-Real-IP


You need to install Apache's mod_rpaf for this to work. You can find an RPM for it in various repositories.

Comments

  1. I knew that nginx handles load much better than apache but had no idea about the scale of the difference!

    In case where nginx is fronted to apache guess some extra performance could be gained by enabling caching in nginx.

    Posted by Sergey — 28 Aug 2011, 22:49

  2. Yep, that's an option. Generally, I found that nginx is quite flexible. I like that various kinds of matching (like regexp) and flow-controlling if statements are available straight within the configuration file.

    Posted by hazard — 29 Aug 2011, 14:25