//
you're reading...
Camping IO, HAProxy, Nginx

Nginx and HAProxy for Fun and Profit.

I’ve recently been playing around with Camping IO and Celluloid Reel-Rack (will be a different post) and I need to setup some proxy to the running apps and I finally got a working combo I really like.

I’m using Nginx on the frontend, mainly for SSL offload, but also to host a backup page.

Then I’m using HAProxy 1.4 as a middle layer, there’s a lot of features I’ll look at using later on when I build a bigger testbed, things like sticky cookies and context matching.

Finally the backend is my Camping IO / Reel-Rack service.

This idea is that normally, when all apps are up, connections go to Nginx -> HAProxy -> Rackapp.

If the Rackapp is down, then HAProxy will go to its backup server which is Nginx listening on port 81 just to serve a basic HTML page.

Finally if HAProxy is down, then Nginx will serve up a 50x.html error page.

Took a lot of trial and error to get the settings just right where:

1.) The Rackapp isn’t crashing everytime HAProxy does a http check.
2.) Nginx wouldn’t serve up the generic 503 Bad Gateway error page.

I originally had HAProxy listening on port 80 and serving pages directly, this is more ideal but I had issues with the connections getting to the Rackapp and I couldn’t figure it out, so in the end I moved the port 80 listener to Nginx.

Here’s the configs for the two services I’m currently using on my SmartOS server:

user www www;
worker_processes 1;

events {
    worker_connections 1024;
}

http {
    include /opt/local/etc/nginx/mime.types;
    default_type  application/octet-stream;
    sendfile on;
    keepalive_timeout  65;

    server_tokens off;
    server_name_in_redirect off;
    port_in_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Scheme $scheme;
    proxy_set_header X-NginX-Proxy true;
    proxy_redirect off;

    proxy_read_timeout 15s;
    proxy_connect_timeout 15s;

    root /home/web;
    fastcgi_buffers 8 16k;
    fastcgi_buffer_size 32k;
    error_page 404 /404.html;
    error_page 500 502 503 504 /50x.html;

    upstream haproxy {
        server 127.0.0.1:8000;
    }

    upstream haproxy-stats {
        server 127.0.0.1:8080;
    }

    server {
        listen 127.0.0.1:81;
        server_name  _;
        server_tokens off;

        location / {
            index index.html;
        }
    }

    server {
        listen 80;
        server_name  _;
        server_tokens off;

        location / { 
            proxy_pass http://haproxy;
        }
        location = /404.html {}
        location = /50x.html {}
    }

    server {
        listen 443 ssl;
        server_name   _;

        ssl on;
        ssl_certificate /opt/local/etc/nginx/ssl/server.crt;
        ssl_certificate_key /opt/local/etc/nginx/ssl/server.key;
 
        proxy_redirect http://$host/ https://$host/;
 
        location / { 
            proxy_pass http://haproxy;
        }

        location /stats { 
            proxy_pass http://haproxy-stats/stats;
        }
        location = /404.html {}
        location = /50x.html {}
    }
}
global
    maxconn 4096
    chroot /var/haproxy
    uid 99
    gid 99
    daemon

defaults
    mode http
    log global
    log 127.0.0.1 local0
    log 127.0.0.1 local7 debug
    option httplog
    option dontlognull
    option http-server-close
    option forwardfor except 127.0.0.0/8
    option httpchk  GET / HTTP/1.1\r\nHost:\ webapp
    retries 3
    timeout http-request 3s
    timeout queue 1m
    timeout connect 2s
    timeout client 5s
    timeout server 5s
    timeout http-keep-alive 1s

frontend main_http 127.0.0.1:8000
    default_backend web_app

backend web_app
    server webapp 127.0.0.1:8001 check
    server nginx 127.0.0.1:81 backup check

listen stats 127.0.0.1:8080
    stats enable  
    stats realm Haproxy\ Statistics  
    stats uri /stats
    stats auth admin:admin

NOTES:
This setup assumes SmartOS, why the /opt paths, adjust as needed for other OS’s, it also assumes that all 3 services (Nginx, HAProxy, and the Webapp) are on the same host, adjust the IP’s and listeners as needed.

Advertisements

Discussion

No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s