HAproxy TLS Load Balancing using CentOS 7, NGINX, and PHP Cookies
If you made it to this page after watching a YouTube video where someone explained basic HAProxy
HTTP load balancing while highlighting notepad text with terrible music…perhaps you’re in the right spot.
The point of this post is not to dumb down HAproxy to 0,1,n round robin but to provide a sandbox for some of the more
advanced features. In particular I was interested in TLS bridging and termination and how that actually appeared on the backend servers.
By no means would I consider myself an expert with HAproxy or NGINX (or PHP for that matter) and I wouldn’t expect these
templates to be a gold standard for anybody, but they can provide a framework to expand on some HAproxy features without starting from scratch. Although if I’m lucky they’ll be heavily referenced in some YT videos by 2018.
A few possibilities based on this setup:
1) Use socat to control HAProxy
2) Connect to https://10.0.0.80:4433 and login as admin:admin
3) Witness TLS bridging and termination using HTTP headers via backend PHP
4) Use /terminate.php or /bridge.php to direct traffic through particular HAProxy TLS backends
5) Rename up.html on an individual web server to put it into maintenance mode
6) Examine which HAProxy backend is communicating with your servers via X headers
7) Use PHP to create 2 minute cookie validating session stickiness
Basic Deployment
HA1 = HAProxy Load Balancer (10.0.0.80)
WEB1 = NGINX + PHP (10.0.0.81)
WEB2 = NGINX + PHP (10.0.0.82)
WEBx Base Commands
yum install epel-release -y yum install php-fpm nginx -y firewall-cmd --permanent --add-service http firewall-cmd --permanent --add-service https firewall-cmd --reload # used to test drain touch /usr/share/nginx/html/up.html # each file should match contents of index.php touch /usr/share/nginx/html/index.php touch /usr/share/nginx/html/terminate.php touch /usr/share/nginx/html/bridge.php systemctl enable nginx && systemctl start nginx systemctl enable php-fpm && systemctl start php-fpm cd /etc/pki/tls/certs openssl req -x509 -newkey rsa:2048 -sha256 -keyout wwwpriv.pem -out wwwcert.pem -days 365 -nodes -subj /C=US/ST=State/L=City/O=infiniteloop.io/OU=Lab/CN=*.infiniteloop.io sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g' /etc/php.ini sed -i 's/user = apache/user = nginx/g' /etc/php-fpm.d/www.conf sed -i 's/group = apache/group = nginx/g' /etc/php-fpm.d/www.conf mkdir /var/lib/php/session chown nginx:nginx /var/lib/php/session
WEBx /etc/nginx/nginx.conf
user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; } http { log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; server { listen 80 default_server; listen 443 ssl; server_name %SERVERNAME%; root /usr/share/nginx/html; ssl_certificate "/etc/pki/tls/certs/wwwcert.pem"; ssl_certificate_key "/etc/pki/tls/certs/wwwpriv.pem"; # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; location / { index index.php index.html; try_files $uri $uri/ =404; } location ~\.php$ { try_files $uri =404; fastcgi_split_path_info ^(.+\.php)(/.+)$; fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; include fastcgi_params; } error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } } }
WEBx Contents of /usr/share/nginx/html/index.php|bridge.php|terminate.php
<?php $lifetime=120; ini_set('session.gc_maxlifetime', $lifetime); session_set_cookie_params($lifetime); session_start(); header('Content-Type: text/plain'); if(!isset($_SESSION['visit'])) { echo "\nFirst Visit"; $_SESSION['visit'] = 1; } else { $_SESSION['visit']++; echo "\nPage Visits: " . $_SESSION['visit']; } echo "\n\nServer Name: " . $_SERVER['SERVER_NAME']; echo "\nSession Name: " . session_name(); echo "\nSession ID: " . session_id(); echo "\nServer IP: " . $_SERVER['SERVER_ADDR']; echo "\nServer Port: " . $_SERVER['SERVER_PORT']; echo "\nClient IP: " . $_SERVER['REMOTE_ADDR']; echo "\nClient Port: " . $_SERVER['REMOTE_PORT']; echo "\nX-Forwarded-for: " . $_SERVER['HTTP_X_FORWARDED_FOR']; echo "\nX-Forwarded-proto: " . $_SERVER['HTTP_X_FORWARDED_PROTO']; echo "\nX-Backend: " . $_SERVER['HTTP_X_BACKEND']; echo "\nHTTP-Cookie: " . $_SERVER['HTTP_COOKIE'] . "\n"; ?>
HA1 Base Commands
yum install haproxy -y systemctl enable haproxy && systemctl start haproxy firewall-cmd --permanent --add-service http firewall-cmd --permanent --add-service https firewall-cmd --permanent --add-port 4433/tcp firewall-cmd --reload # assuming SELinux is live you'll want the following setsebool -P haproxy_connect_any 1 cd /etc/pki/tls/certs openssl req -x509 -newkey rsa:2048 -sha256 -keyout hapriv.pem -out hacert.pem -days 365 -nodes -subj /C=US/ST=State/L=City/O=infiniteloop.io/OU=Lab/CN=*.infiniteloop.io cat hacert.pem hapriv.pem > hachain.pem
HA1 Contents of /etc/haproxy/haproxy.cfg
global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats mode 600 level admin stats timeout 2m tune.ssl.default-dh-param 2048 defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 option httpchk GET /up.html http-check disable-on-404 default-server inter 10000 cookie PHPSESSID prefix nocache listen stats bind *:4433 ssl crt /etc/pki/tls/certs/hachain.pem mode http stats enable stats hide-version stats realm HAproxy\ Statistics stats uri / stats auth admin:admin frontend ft_http bind *:80 default_backend bk_http frontend ft_https bind *:443 ssl crt /etc/pki/tls/certs/hachain.pem acl bridge path_beg /bridge acl terminate path_beg /terminate use_backend bk_https_bridge if bridge use_backend bk_https_terminate if terminate default_backend bk_https_bridge backend bk_http reqadd X-Forwarded-Proto:\ http reqadd X-Backend:\ bk_http balance roundrobin server web1 10.0.0.81:80 cookie w1 check server web2 10.0.0.82:80 cookie w2 check backend bk_https_bridge reqadd X-Forwarded-Proto:\ https reqadd X-Backend:\ bk_https_bridge balance roundrobin server web1 10.0.0.81:443 cookie w1 check ssl verify none server web2 10.0.0.82:443 cookie w2 check ssl verify none backend bk_https_terminate reqadd X-Forwarded-Proto:\ http reqadd X-Backend:\ bk_https_terminate balance roundrobin server web1 10.0.0.81:80 cookie w1 check server web2 10.0.0.82:80 cookie w2 check
Keepalived Configuration
I deployed a second HAproxy server after the initial post to experiment with Keepalived and a floating IP. I was expecting some hassles but the only thing I encountered was the check script you find everywhere used killall -0 haproxy, which was not part of my centos minimal install, so I had to swap it out with pidof instead.
#Run the following on both HAproxy servers firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens160 --destination 224.0.0.18 --protocol vrrp -j ACCEPT firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface ens160 --destination 224.0.0.18 --protocol vrrp -j ACCEPT firewall-cmd --reload # Primary HAproxy server /etc/keepalived/keepalived.conf contents vrrp_script chk_haproxy { script "pidof haproxy" interval 2 weight 2 } vrrp_instance VI_1 { interface ens160 state MASTER virtual_router_id 11 priority 101 virtual_ipaddress { 10.0.0.180 } track_script { chk_haproxy } } # Secondary HAproxy Server /etc/keepalived/keepalived.conf contents vrrp_script chk_haproxy { script "pidof haproxy" interval 2 weight 2 } vrrp_instance VI_1 { interface ens160 state BACKUP virtual_router_id 11 priority 100 virtual_ipaddress { 10.0.0.180 } track_script { chk_haproxy } } # restart keepalived on both servers after updating conf files systemctl restart keepalived
With this in place you can now stop haproxy on HA1 and HA2 will add 10.0.0.180 as a virtual address. You can verify with ip -a or using tail -f /var/log/messages | grep -i keepalived to watch status updates.