HAproxy TLS Load Balancing using CentOS 7, NGINX, and PHP Cookies

If you made it to this page after watching a YouTube video where someone explained basic HAProxy
HTTP load balancing while highlighting notepad text with terrible music…perhaps you’re in the right spot.
The point of this post is not to dumb down HAproxy to 0,1,n round robin but to provide a sandbox for some of the more
advanced features. In particular I was interested in TLS bridging and termination and how that actually appeared on the backend servers.

By no means would I consider myself an expert with HAproxy or NGINX (or PHP for that matter) and I wouldn’t expect these
templates to be a gold standard for anybody, but they can provide a framework to expand on some HAproxy features without starting from scratch.  Although if I’m lucky they’ll be heavily referenced in some YT videos by 2018.

A few possibilities based on this setup:
1) Use socat to control HAProxy
2) Connect to https://10.0.0.80:4433 and login as admin:admin
3) Witness TLS bridging and termination using HTTP headers via backend PHP
4) Use /terminate.php or /bridge.php to direct traffic through particular HAProxy TLS backends
5) Rename up.html on an individual web server to put it into maintenance mode
6) Examine which HAProxy backend is communicating with your servers via X headers
7) Use PHP to create 2 minute cookie validating session stickiness

Basic Deployment

HA1 = HAProxy Load Balancer (10.0.0.80)
WEB1 = NGINX + PHP (10.0.0.81)
WEB2 = NGINX + PHP (10.0.0.82)

WEBx Base Commands

yum install epel-release  -y
yum install php-fpm nginx  -y

firewall-cmd --permanent --add-service http
firewall-cmd --permanent --add-service https
firewall-cmd --reload

# used to test drain
touch /usr/share/nginx/html/up.html

# each file should match contents of index.php
touch /usr/share/nginx/html/index.php
touch /usr/share/nginx/html/terminate.php
touch /usr/share/nginx/html/bridge.php

systemctl enable nginx && systemctl start nginx
systemctl enable php-fpm && systemctl start php-fpm

cd /etc/pki/tls/certs
openssl req -x509 -newkey rsa:2048 -sha256 -keyout wwwpriv.pem -out wwwcert.pem -days 365 -nodes -subj /C=US/ST=State/L=City/O=infiniteloop.io/OU=Lab/CN=*.infiniteloop.io

sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g' /etc/php.ini
sed -i 's/user = apache/user = nginx/g' /etc/php-fpm.d/www.conf
sed -i 's/group = apache/group = nginx/g' /etc/php-fpm.d/www.conf
mkdir /var/lib/php/session
chown nginx:nginx /var/lib/php/session

 

WEBx /etc/nginx/nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       443 ssl;
        server_name  %SERVERNAME%;
        root         /usr/share/nginx/html;
        ssl_certificate "/etc/pki/tls/certs/wwwcert.pem";
        ssl_certificate_key "/etc/pki/tls/certs/wwwpriv.pem";

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
            index index.php index.html;
            try_files $uri $uri/ =404;
        }
		
        location ~\.php$ {
            try_files $uri =404;
            fastcgi_split_path_info ^(.+\.php)(/.+)$;
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            include fastcgi_params;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
}

 

WEBx Contents of /usr/share/nginx/html/index.php|bridge.php|terminate.php

<?php
$lifetime=120;
ini_set('session.gc_maxlifetime', $lifetime);
session_set_cookie_params($lifetime);
session_start();

header('Content-Type: text/plain');

if(!isset($_SESSION['visit']))
{
  echo "\nFirst Visit";
  $_SESSION['visit'] = 1;
}
else
{
  $_SESSION['visit']++;
  echo "\nPage Visits:      " . $_SESSION['visit'];
}

echo "\n\nServer Name:       " . $_SERVER['SERVER_NAME'];
echo "\nSession Name:      " . session_name();
echo "\nSession ID:        " . session_id();
echo "\nServer IP:         " . $_SERVER['SERVER_ADDR'];
echo "\nServer Port:       " . $_SERVER['SERVER_PORT'];
echo "\nClient IP:         " . $_SERVER['REMOTE_ADDR'];
echo "\nClient Port:       " . $_SERVER['REMOTE_PORT'];
echo "\nX-Forwarded-for:   " . $_SERVER['HTTP_X_FORWARDED_FOR'];
echo "\nX-Forwarded-proto: " . $_SERVER['HTTP_X_FORWARDED_PROTO'];
echo "\nX-Backend:         " . $_SERVER['HTTP_X_BACKEND'];
echo "\nHTTP-Cookie:       " . $_SERVER['HTTP_COOKIE'] . "\n";
?>

 

HA1 Base Commands

yum install haproxy -y
systemctl enable haproxy && systemctl start haproxy

firewall-cmd --permanent --add-service http
firewall-cmd --permanent --add-service https
firewall-cmd --permanent --add-port 4433/tcp
firewall-cmd --reload

# assuming SELinux is live you'll want the following
setsebool -P haproxy_connect_any 1 

cd /etc/pki/tls/certs
openssl req -x509 -newkey rsa:2048 -sha256 -keyout hapriv.pem -out hacert.pem -days 365 -nodes -subj /C=US/ST=State/L=City/O=infiniteloop.io/OU=Lab/CN=*.infiniteloop.io

cat hacert.pem hapriv.pem > hachain.pem

 

HA1 Contents of /etc/haproxy/haproxy.cfg

global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    stats socket /var/lib/haproxy/stats mode 600 level admin
    stats timeout 2m

    tune.ssl.default-dh-param 2048

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

    option httpchk GET /up.html
    http-check disable-on-404
    default-server inter  10000
    cookie PHPSESSID prefix nocache

listen stats
    bind *:4433 ssl crt /etc/pki/tls/certs/hachain.pem
    mode http
    stats enable
    stats hide-version
    stats realm HAproxy\ Statistics
    stats uri /
    stats auth admin:admin

frontend ft_http
    bind *:80
    default_backend     bk_http

frontend ft_https
    bind *:443 ssl crt /etc/pki/tls/certs/hachain.pem
    acl bridge path_beg /bridge
    acl terminate path_beg /terminate
    use_backend bk_https_bridge if bridge
    use_backend bk_https_terminate if terminate
    default_backend bk_https_bridge

backend bk_http
    reqadd X-Forwarded-Proto:\ http
    reqadd X-Backend:\ bk_http
    balance roundrobin
    server  web1 10.0.0.81:80 cookie w1 check
    server  web2 10.0.0.82:80 cookie w2 check

backend bk_https_bridge
    reqadd X-Forwarded-Proto:\ https
    reqadd X-Backend:\ bk_https_bridge
    balance roundrobin
    server web1 10.0.0.81:443 cookie w1 check ssl verify none
    server web2 10.0.0.82:443 cookie w2 check ssl verify none

backend bk_https_terminate
    reqadd X-Forwarded-Proto:\ http
    reqadd X-Backend:\ bk_https_terminate
    balance roundrobin
    server web1 10.0.0.81:80 cookie w1 check
    server web2 10.0.0.82:80 cookie w2 check

 

Keepalived Configuration

I deployed a second HAproxy server after the initial post to experiment with Keepalived and a floating IP.  I was expecting some hassles but the only thing I encountered was the check script you find everywhere used killall -0 haproxy, which was not part of my centos minimal install, so I had to swap it out with pidof instead.

#Run the following on both HAproxy servers
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens160 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface ens160 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --reload

# Primary HAproxy server /etc/keepalived/keepalived.conf contents
vrrp_script chk_haproxy {
  script "pidof haproxy"
  interval 2
  weight 2
}

vrrp_instance VI_1 {
  interface ens160
  state MASTER
  virtual_router_id 11
  priority 101
  virtual_ipaddress {
    10.0.0.180
  }
  track_script {
    chk_haproxy
  }
}


# Secondary HAproxy Server /etc/keepalived/keepalived.conf contents
vrrp_script chk_haproxy {
  script "pidof haproxy"
  interval 2
  weight 2
}

vrrp_instance VI_1 {
  interface ens160
  state BACKUP
  virtual_router_id 11
  priority 100
  virtual_ipaddress {
    10.0.0.180
  }
  track_script {
    chk_haproxy
  }
}

# restart keepalived on both servers after updating conf files
systemctl restart keepalived

With this in place you can now stop haproxy on HA1 and HA2 will add 10.0.0.180 as a virtual address.  You can verify with ip -a or using tail -f /var/log/messages | grep -i keepalived to watch status updates.