HAproxy TLS Load Balancing using CentOS 7, NGINX, and PHP Cookies
If you made it to this page after watching a YouTube video where someone explained basic HAProxy
HTTP load balancing while highlighting notepad text with terrible music…perhaps you’re in the right spot.
The point of this post is not to dumb down HAproxy to 0,1,n round robin but to provide a sandbox for some of the more
advanced features. In particular I was interested in TLS bridging and termination and how that actually appeared on the backend servers.
By no means would I consider myself an expert with HAproxy or NGINX (or PHP for that matter) and I wouldn’t expect these
templates to be a gold standard for anybody, but they can provide a framework to expand on some HAproxy features without starting from scratch. Although if I’m lucky they’ll be heavily referenced in some YT videos by 2018.
A few possibilities based on this setup:
1) Use socat to control HAProxy
2) Connect to https://10.0.0.80:4433 and login as admin:admin
3) Witness TLS bridging and termination using HTTP headers via backend PHP
4) Use /terminate.php or /bridge.php to direct traffic through particular HAProxy TLS backends
5) Rename up.html on an individual web server to put it into maintenance mode
6) Examine which HAProxy backend is communicating with your servers via X headers
7) Use PHP to create 2 minute cookie validating session stickiness
Basic Deployment
HA1 = HAProxy Load Balancer (10.0.0.80)
WEB1 = NGINX + PHP (10.0.0.81)
WEB2 = NGINX + PHP (10.0.0.82)
WEBx Base Commands
yum install epel-release -y yum install php-fpm nginx -y firewall-cmd --permanent --add-service http firewall-cmd --permanent --add-service https firewall-cmd --reload # used to test drain touch /usr/share/nginx/html/up.html # each file should match contents of index.php touch /usr/share/nginx/html/index.php touch /usr/share/nginx/html/terminate.php touch /usr/share/nginx/html/bridge.php systemctl enable nginx && systemctl start nginx systemctl enable php-fpm && systemctl start php-fpm cd /etc/pki/tls/certs openssl req -x509 -newkey rsa:2048 -sha256 -keyout wwwpriv.pem -out wwwcert.pem -days 365 -nodes -subj /C=US/ST=State/L=City/O=infiniteloop.io/OU=Lab/CN=*.infiniteloop.io sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g' /etc/php.ini sed -i 's/user = apache/user = nginx/g' /etc/php-fpm.d/www.conf sed -i 's/group = apache/group = nginx/g' /etc/php-fpm.d/www.conf mkdir /var/lib/php/session chown nginx:nginx /var/lib/php/session
WEBx /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen 443 ssl;
server_name %SERVERNAME%;
root /usr/share/nginx/html;
ssl_certificate "/etc/pki/tls/certs/wwwcert.pem";
ssl_certificate_key "/etc/pki/tls/certs/wwwpriv.pem";
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
index index.php index.html;
try_files $uri $uri/ =404;
}
location ~\.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
}
WEBx Contents of /usr/share/nginx/html/index.php|bridge.php|terminate.php
<?php
$lifetime=120;
ini_set('session.gc_maxlifetime', $lifetime);
session_set_cookie_params($lifetime);
session_start();
header('Content-Type: text/plain');
if(!isset($_SESSION['visit']))
{
echo "\nFirst Visit";
$_SESSION['visit'] = 1;
}
else
{
$_SESSION['visit']++;
echo "\nPage Visits: " . $_SESSION['visit'];
}
echo "\n\nServer Name: " . $_SERVER['SERVER_NAME'];
echo "\nSession Name: " . session_name();
echo "\nSession ID: " . session_id();
echo "\nServer IP: " . $_SERVER['SERVER_ADDR'];
echo "\nServer Port: " . $_SERVER['SERVER_PORT'];
echo "\nClient IP: " . $_SERVER['REMOTE_ADDR'];
echo "\nClient Port: " . $_SERVER['REMOTE_PORT'];
echo "\nX-Forwarded-for: " . $_SERVER['HTTP_X_FORWARDED_FOR'];
echo "\nX-Forwarded-proto: " . $_SERVER['HTTP_X_FORWARDED_PROTO'];
echo "\nX-Backend: " . $_SERVER['HTTP_X_BACKEND'];
echo "\nHTTP-Cookie: " . $_SERVER['HTTP_COOKIE'] . "\n";
?>
HA1 Base Commands
yum install haproxy -y systemctl enable haproxy && systemctl start haproxy firewall-cmd --permanent --add-service http firewall-cmd --permanent --add-service https firewall-cmd --permanent --add-port 4433/tcp firewall-cmd --reload # assuming SELinux is live you'll want the following setsebool -P haproxy_connect_any 1 cd /etc/pki/tls/certs openssl req -x509 -newkey rsa:2048 -sha256 -keyout hapriv.pem -out hacert.pem -days 365 -nodes -subj /C=US/ST=State/L=City/O=infiniteloop.io/OU=Lab/CN=*.infiniteloop.io cat hacert.pem hapriv.pem > hachain.pem
HA1 Contents of /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats mode 600 level admin
stats timeout 2m
tune.ssl.default-dh-param 2048
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
option httpchk GET /up.html
http-check disable-on-404
default-server inter 10000
cookie PHPSESSID prefix nocache
listen stats
bind *:4433 ssl crt /etc/pki/tls/certs/hachain.pem
mode http
stats enable
stats hide-version
stats realm HAproxy\ Statistics
stats uri /
stats auth admin:admin
frontend ft_http
bind *:80
default_backend bk_http
frontend ft_https
bind *:443 ssl crt /etc/pki/tls/certs/hachain.pem
acl bridge path_beg /bridge
acl terminate path_beg /terminate
use_backend bk_https_bridge if bridge
use_backend bk_https_terminate if terminate
default_backend bk_https_bridge
backend bk_http
reqadd X-Forwarded-Proto:\ http
reqadd X-Backend:\ bk_http
balance roundrobin
server web1 10.0.0.81:80 cookie w1 check
server web2 10.0.0.82:80 cookie w2 check
backend bk_https_bridge
reqadd X-Forwarded-Proto:\ https
reqadd X-Backend:\ bk_https_bridge
balance roundrobin
server web1 10.0.0.81:443 cookie w1 check ssl verify none
server web2 10.0.0.82:443 cookie w2 check ssl verify none
backend bk_https_terminate
reqadd X-Forwarded-Proto:\ http
reqadd X-Backend:\ bk_https_terminate
balance roundrobin
server web1 10.0.0.81:80 cookie w1 check
server web2 10.0.0.82:80 cookie w2 check
Keepalived Configuration
I deployed a second HAproxy server after the initial post to experiment with Keepalived and a floating IP. I was expecting some hassles but the only thing I encountered was the check script you find everywhere used killall -0 haproxy, which was not part of my centos minimal install, so I had to swap it out with pidof instead.
#Run the following on both HAproxy servers
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens160 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface ens160 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --reload
# Primary HAproxy server /etc/keepalived/keepalived.conf contents
vrrp_script chk_haproxy {
script "pidof haproxy"
interval 2
weight 2
}
vrrp_instance VI_1 {
interface ens160
state MASTER
virtual_router_id 11
priority 101
virtual_ipaddress {
10.0.0.180
}
track_script {
chk_haproxy
}
}
# Secondary HAproxy Server /etc/keepalived/keepalived.conf contents
vrrp_script chk_haproxy {
script "pidof haproxy"
interval 2
weight 2
}
vrrp_instance VI_1 {
interface ens160
state BACKUP
virtual_router_id 11
priority 100
virtual_ipaddress {
10.0.0.180
}
track_script {
chk_haproxy
}
}
# restart keepalived on both servers after updating conf files
systemctl restart keepalived
With this in place you can now stop haproxy on HA1 and HA2 will add 10.0.0.180 as a virtual address. You can verify with ip -a or using tail -f /var/log/messages | grep -i keepalived to watch status updates.