Test an HAProxy Graceful Reload Under Live Traffic
Last edited on August 10, 2026

A successful HAProxy reload message is only one half of a safe change. The new worker must accept the intended traffic, while the old worker must finish connections it already admitted. If you verify only systemctl reload haproxy, you can miss a route that now points at the wrong backend, a long request that was reset, or old workers that never drain.

Treat an HAProxy graceful reload as a release gate with two clocks: new-request convergence should happen quickly, but old-connection drain may legitimately take as long as the application protocol allows. Validate the candidate first, label the old and new generations, hold one real request open across the handoff, send a fixed traffic set, inspect process state, and keep a validated rollback ready.

In the reproduced lab behind this guide, HAProxy 3.0.11 ran on Debian 13. A malformed candidate failed before the running master changed. One slow request completed on generation v1 after fresh requests were already reaching v2. A separate parallel run saved 1,000 of 1,000 responses with zero curl error lines, and rollback restored v1 through the same validation and reload path.

Define the Reload Acceptance Contract First

“Zero downtime” is too vague to be a test. Write the observable contract before touching the configuration.

Evidence point Pass condition What a failure means
candidate parse installed HAProxy binary returns exit 0 for the complete candidate do not signal or reload the running master
active baseline request reaches the expected route and carries an old-generation marker the pre-change state is not known well enough to compare
in-flight continuity request started before reload finishes successfully on the old generation admitted work was interrupted or the test did not span the handoff
new-request convergence fresh request reaches the intended new generation and backend reload did not apply as intended even if the service is active
fixed traffic set every expected request completes; transport error count is zero investigate resets, refusals, timeouts, limits, or the test harness
old-worker drain old worker appears during the open request and disappears after it finishes a long-lived connection, stuck session, or repeated reload may retain it
rollback saved config validates, reloads, and returns the prior marker recovery path is not ready for a change window

This contract deliberately separates listener handoff from backend health. If a worker accepts the request but a server is marked down, continue with HAProxy probe-versus-service diagnosis. Likewise, a failed new connection can belong to Linux listen-queue pressure or conntrack rather than the reload itself.

Read the Installed Reload Path Before You Use It

HAProxy’s process model and your service manager must agree. Current packages commonly run a master process that starts workers and handles reloads, but unit files, binary paths, configuration includes, PID files, chroots, capabilities, and command-line options vary. Inspect the installed unit rather than copying a command from another distribution.

set -euo pipefail
command -v haproxy
command -v socat
haproxy -vv | sed -n '1,45p'
systemctl show haproxy -p FragmentPath -p ExecStart -p ExecReload
systemctl cat haproxy
systemctl show haproxy -p MainPID -p ActiveState -p SubState

This is an unmarked production preflight: the disposable evidence host deliberately did not install or start a system-wide HAProxy service, so those exact service-manager commands were not counted as reproduced input. The bounded run identified Debian 13, Linux 6.12.96, and an extracted HAProxy 3.0.11-1+deb13u3 binary built with systemd and multithreading support. Version identity matters: a 2017 HAProxy 1.8 unit example is useful history, but it is not authority for a current package.

HAProxy’s official configuration testing guide recommends haproxy -c before a service action. HAProxy’s hitless reload explanation describes listener file-descriptor transfer and the overlap between old and new workers. Those are two separate requirements: a parseable candidate and a handoff-capable runtime.

On production, use the installed package’s supported ExecReload path after validation. The direct USR2 signal below belongs to the isolated lab master that the same script started; it is not a universal substitute for systemctl reload haproxy.

Build a Bounded, Generation-Labeled Lab

Only two loopback listeners exist in the fixture: 127.0.0.1:19080 for HAProxy and 127.0.0.1:19081 for a Python backend. /slow waits three seconds. HAProxy adds X-Reload-Generation: v1 or v2, so the response proves which worker generation handled it without changing the backend body.

Create one backend and two candidate configurations

Use a literal lab path that does not already exist. The production runner kept the HAProxy package under the project directory and wrote evidence beneath one guarded lab directory; the shortened block below shows the complete behavioral fixture.

set -euo pipefail
LAB=/var/tmp/voxfor-haproxy-reload-116
[[ ! -e "$LAB" ]]
install -d -m 0700 "$LAB" "$LAB/run" "$LAB/requests"

cat >"$LAB/backend.py" <<'PY'
import json, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/slow": time.sleep(3)
        body = json.dumps({"backend":"fixture-a","path":self.path},
                          separators=(",", ":")).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers(); self.wfile.write(body)
    def log_message(self, fmt, *args): return
ThreadingHTTPServer(("127.0.0.1", 19081), Handler).serve_forever()
PY

cat >"$LAB/v1.cfg" <<EOF
global
    stats socket $LAB/run/admin.sock mode 600 level admin
    master-worker
defaults
    mode http
    timeout connect 2s
    timeout client 10s
    timeout server 10s
frontend lab_frontend
    bind 127.0.0.1:19080
    http-response set-header X-Reload-Generation v1
    default_backend lab_backend
backend lab_backend
    server fixture-a 127.0.0.1:19081 check
EOF

sed 's/X-Reload-Generation v1/X-Reload-Generation v2/' \
  "$LAB/v1.cfg" >"$LAB/v2.cfg"
sed 's/default_backend/default_backned/' \
  "$LAB/v1.cfg" >"$LAB/broken.cfg"
install -m 0600 "$LAB/v1.cfg" "$LAB/active.cfg"

This fixture isolates one variable: the worker-generation header. In production, identify the route with a health-neutral endpoint, response header, backend log field, deployment ID, or application build marker. Do not add a public debug header indefinitely merely to make a change window convenient.

Make the malformed candidate fail closed

Validate with the same binary and full configuration tree that the service will load. If production uses several -f arguments, a configuration directory, or generated files, reproduce that exact input set. Testing only the edited fragment can miss an error in an included file.

set -euo pipefail
: "${LAB:=/var/tmp/voxfor-haproxy-reload-116}"
HAPROXY="$(command -v haproxy)"

"$HAPROXY" -c -V -f "$LAB/v1.cfg"
"$HAPROXY" -c -V -f "$LAB/v2.cfg"
set +e
"$HAPROXY" -c -V -f "$LAB/broken.cfg" \
  >"$LAB/broken-validation.txt" 2>&1
BROKEN_RC=$?
set -e
test "$BROKEN_RC" -ne 0
grep -q "unknown keyword 'default_backned'" "$LAB/broken-validation.txt"

default_backned returned exit code 1 and a fatal parser error in the current run; v1 and v2 both returned Configuration file is valid. That failure is useful evidence because it proves the gate stops before the master receives a reload signal. A parser pass still cannot prove that ACL order, map contents, certificates, DNS answers, server reachability, or intended routing are correct.

Hold Old Work Open While Fresh Traffic Moves

Start the bounded backend and HAProxy master, prove v1, then begin /slow before applying v2. The held request is the old-work clock. A new ordinary request is the convergence clock.

Establish v1 and start the slow request

set -euo pipefail
: "${LAB:=/var/tmp/voxfor-haproxy-reload-116}"

python3 "$LAB/backend.py" >"$LAB/backend.log" 2>&1 &
BACKEND_PID=$!
printf '%s\n' "$BACKEND_PID" >"$LAB/run/backend.pid"
for _ in $(seq 1 50); do
  curl -fsS http://127.0.0.1:19081/health >/dev/null && break
  sleep 0.1
done

haproxy -W -db -S "$LAB/run/master.sock" \
  -f "$LAB/active.cfg" -p "$LAB/run/haproxy.pid" \
  >"$LAB/haproxy.log" 2>&1 &
HAPROXY_JOB_PID=$!
printf '%s\n' "$HAPROXY_JOB_PID" >"$LAB/run/haproxy-job.pid"
for _ in $(seq 1 50); do
  curl -fsS -D "$LAB/requests/baseline.headers" \
    http://127.0.0.1:19080/ -o "$LAB/requests/baseline.body" && break
  sleep 0.1
done
grep -qi '^x-reload-generation: v1' "$LAB/requests/baseline.headers"

curl -fsS -D "$LAB/requests/slow.headers" \
  http://127.0.0.1:19080/slow -o "$LAB/requests/slow.body" &
SLOW_PID=$!
sleep 0.5

Choose a slow endpoint that runs longer than the delay before reload but shorter than both client and server timeouts. If the application uses requests that legitimately last minutes, size the test to its real contract. Artificially raising HAProxy timeouts only to make the rehearsal pass changes the system under test.

Validate v2, signal only the lab master, and compare generations

set -euo pipefail
: "${LAB:=/var/tmp/voxfor-haproxy-reload-116}"

haproxy -c -V -f "$LAB/v2.cfg"
install -m 0600 "$LAB/v2.cfg" "$LAB/active.cfg"
MASTER_PID="$(cat "$LAB/run/haproxy.pid")"
kill -USR2 "$MASTER_PID"

for _ in $(seq 1 80); do
  if curl -fsS -D "$LAB/requests/post.headers" \
      http://127.0.0.1:19080/ -o "$LAB/requests/post.body" \
      && grep -qi '^x-reload-generation: v2' "$LAB/requests/post.headers"; then
    break
  fi
  sleep 0.1
done
grep -qi '^x-reload-generation: v2' "$LAB/requests/post.headers"

printf 'show proc\n' | socat - UNIX-CONNECT:"$LAB/run/master.sock" \
  >"$LAB/show-proc-during-drain.txt"
awk '/^# old workers/{seen=1; next} /^# programs/{seen=0} \
  seen && $2 == "worker"{found=1} END{exit found?0:1}' \
  "$LAB/show-proc-during-drain.txt"

wait "$SLOW_PID"
grep -qi '^x-reload-generation: v1' "$LAB/requests/slow.headers"

for _ in $(seq 1 50); do
  printf 'show proc\n' | socat - UNIX-CONNECT:"$LAB/run/master.sock" \
    >"$LAB/show-proc-after-drain.txt"
  if ! awk '/^# old workers/{seen=1; next} /^# programs/{seen=0} \
      seen && $2 == "worker"{found=1} END{exit found?0:1}' \
      "$LAB/show-proc-after-drain.txt"; then
    break
  fi
  sleep 0.1
done
! awk '/^# old workers/{seen=1; next} /^# programs/{seen=0} \
  seen && $2 == "worker"{found=1} END{exit found?0:1}' \
  "$LAB/show-proc-after-drain.txt"

The fresh response arrived with v2 before the already-open slow response completed with v1. The master CLI simultaneously listed a current worker and an old worker during that open request, then an empty old-worker section after it finished. That ordering is the handoff proof. Merely receiving two HTTP 200 responses would be weaker because both could have come from the same generation.

Measure Continuity and Watch the Old Worker Drain

One slow request proves an admitted request survived. It does not sample the new-connection path under concurrency. The second run therefore issued a fixed set of 1,000 requests with Connection: close, wrote one file per completed response, captured stderr separately, and refused to pass unless both the file count and error count matched expectations.

set -euo pipefail
: "${LAB:=/var/tmp/voxfor-haproxy-reload-116}"

seq 1 1000 | xargs -P 24 -I{} \
  curl --fail --silent --show-error --max-time 5 \
    -H 'Connection: close' \
    "http://127.0.0.1:19080/load?request={}" \
    -o "$LAB/requests/load-{}.json" \
    2>"$LAB/load-errors.txt"

LOAD_COUNT="$(find "$LAB/requests" -maxdepth 1 \
  -type f -name 'load-*.json' | wc -l)"
LOAD_ERRORS="$(wc -l <"$LAB/load-errors.txt")"
test "$LOAD_COUNT" -eq 1000
test "$LOAD_ERRORS" -eq 0

printf 'show info\n' | socat - UNIX-CONNECT:"$LAB/run/admin.sock" \
  | sed -n '/^Name:/p;/^Version:/p;/^Pid:/p;/^Uptime:/p;/^CurrConns:/p;/^CumConns:/p'

An independent first run made the process transition explicit: during the open request, master PID 775250 had current worker 775358 and old worker 775255. After /slow completed, only worker 775358 remained. Its fixed request series recorded 160 successes, zero failures, 20 BLUE responses before the handoff and 140 GREEN responses after it. The repeatable second run independently captured master PID 806332, current worker 806362, and old worker 806337 during drain; its post-drain master receipt retained only the current worker.

Run B recorded 1,000 expected response files, 1,000 completed, and zero curl error lines. These counts are not a production capacity benchmark; loopback removes real clients, TLS, network loss, and application latency. They are a transport-continuity receipt for this exact handoff.

environment=Debian_13 haproxy=3.0.11-1+deb13u3
broken_candidate_exit=1 running_generation_unchanged=yes
slow_request_generation=v1 new_request_generation=v2
process_during_drain=current_worker+old_worker
first_series=requests:160 ok:160 fail:0 blue:20 green:140
second_series=expected:1000 completed:1000 curl_error_lines:0
process_after_drain=old_worker_absent
rollback_generation=v1 cleanup_listeners=stopped

Do not mistake ActiveState=active for this receipt. A process can remain active while the route is wrong. Conversely, an old worker can remain visible because it is correctly preserving a WebSocket or upload. Judge it against declared maximum age and active-session evidence.

If fresh connections fail while the worker is healthy, conntrack capacity diagnosis separates host admission pressure from HAProxy generation state. If clients abort before the response returns, NGINX 499 timing analysis provides a useful adjacent ownership model even when HAProxy is the edge.

Roll Back Through the Same Gate

Rollback is a second release, not a file copy followed by hope. Preserve the exact prior configuration and metadata before the window. Validate it with the installed binary, apply it atomically through the same owner, use the same supported reload action, then prove the prior route marker.

set -euo pipefail
: "${LAB:=/var/tmp/voxfor-haproxy-reload-116}"
HAPROXY="$(command -v haproxy)"
MASTER_PID="$(cat "$LAB/run/haproxy.pid")"
HAPROXY_JOB_PID="$(cat "$LAB/run/haproxy-job.pid")"
BACKEND_PID="$(cat "$LAB/run/backend.pid")"
for PID in "$MASTER_PID" "$HAPROXY_JOB_PID" "$BACKEND_PID"; do
  [[ "$PID" =~ ^[0-9]+$ ]]
done
ps -p "$MASTER_PID" -o args= | grep -F -- "$LAB/active.cfg" >/dev/null
ps -p "$BACKEND_PID" -o args= | grep -F -- "$LAB/backend.py" >/dev/null

install -m 0600 "$LAB/v1.cfg" "$LAB/rollback.cfg"
"$HAPROXY" -c -V -f "$LAB/rollback.cfg"
install -m 0600 "$LAB/rollback.cfg" "$LAB/active.cfg"
kill -USR2 "$MASTER_PID"

for _ in $(seq 1 80); do
  if curl -fsS -D "$LAB/requests/rollback.headers" \
      http://127.0.0.1:19080/ -o "$LAB/requests/rollback.body" \
      && grep -qi '^x-reload-generation: v1' \
        "$LAB/requests/rollback.headers"; then
    break
  fi
  sleep 0.1
done
grep -qi '^x-reload-generation: v1' "$LAB/requests/rollback.headers"

The reproduced rollback returned Configuration file is valid; the master logged a new worker and Loading success; the response returned v1. Production rollback triggers should be declared before the change: wrong backend identity, any transport errors above the approved baseline, elevated 5xx, application acceptance failure, or an old-worker age that exceeds the protocol budget without an explained long session.

After rollback, diagnose rather than repeatedly alternating generations. Each reload can create another old worker. Repeated reloads during long-lived sessions can accumulate generations, file descriptors, memory, and confusing telemetry.

Expand the Gate for Production Protocols

Scope this evidence to HTTP/1.x request continuity under a bounded configuration change. It does not make every HAProxy deployment hitless by inheritance.

  • WebSockets and tunnels: hold a real connection open, exchange messages after reload, and set a maximum acceptable old-worker age. A process remaining alive may be correct.
  • HTTP/2: test concurrent streams and fresh connections. One TCP connection can carry many requests, so Connection: close is not the same workload.
  • QUIC/HTTP/3: validate the current build, listener ownership, UDP behavior, and supported reload semantics separately.
  • TLS termination: verify SNI names, certificate fingerprints, ALPN, OCSP behavior, and every listening address. Use live TLS endpoint comparison when the certificate, not the route, owns the change.
  • Stick tables, peers, and runtime API changes: determine which state transfers, synchronizes, or exists only in memory. A disk configuration rollback cannot restore every runtime-only mutation.
  • Maps and dynamic discovery: prove the effective map content, DNS resolution, or service-discovery result instead of relying on parser success.
  • Multiple HAProxy nodes: drain or reload one node at a time, preserve an out-of-band return path, and send acceptance traffic to each node instead of only the load-balanced VIP.

HAProxy’s current management guide documents master-worker signals and operational behavior; the 3.0 configuration manual defines the directives in the candidate. Pin those references to the deployed branch. If another reverse proxy owns the route, use its control path; Voxfor’s validated Caddy reload workflow is deliberately separate.

Verify the Receipt and Remove Only the Lab

Machine acceptance checks every central outcome together. It cannot be replaced by manually noticing one green line.

set -euo pipefail
: "${LAB:=/var/tmp/voxfor-haproxy-reload-116}"

grep -q "unknown keyword 'default_backned'" "$LAB/broken-validation.txt"
grep -qi '^x-reload-generation: v1' "$LAB/requests/baseline.headers"
grep -qi '^x-reload-generation: v1' "$LAB/requests/slow.headers"
grep -qi '^x-reload-generation: v2' "$LAB/requests/post.headers"
grep -qi '^x-reload-generation: v1' "$LAB/requests/rollback.headers"
awk '/^# old workers/{seen=1; next} /^# programs/{seen=0} \
  seen && $2 == "worker"{found=1} END{exit found?0:1}' \
  "$LAB/show-proc-during-drain.txt"
! awk '/^# old workers/{seen=1; next} /^# programs/{seen=0} \
  seen && $2 == "worker"{found=1} END{exit found?0:1}' \
  "$LAB/show-proc-after-drain.txt"
test "$(find "$LAB/requests" -maxdepth 1 -name 'load-*.json' | wc -l)" -eq 1000
test "$(wc -l <"$LAB/load-errors.txt")" -eq 0
printf 'verification=PASS invalid=rejected slow=v1 fresh=v2 load=1000/1000 rollback=v1\n'

Stop the literal PIDs started by the lab and refuse a broad path. Production cleanup is different: retain approved evidence and the prior configuration for the defined rollback window.

set -euo pipefail
: "${LAB:=/var/tmp/voxfor-haproxy-reload-116}"
[[ "$LAB" == /var/tmp/voxfor-haproxy-reload-116 ]]
MASTER_PID="$(cat "$LAB/run/haproxy.pid")"
HAPROXY_JOB_PID="$(cat "$LAB/run/haproxy-job.pid")"
BACKEND_PID="$(cat "$LAB/run/backend.pid")"
for PID in "$MASTER_PID" "$HAPROXY_JOB_PID" "$BACKEND_PID"; do
  [[ "$PID" =~ ^[0-9]+$ ]]
done
ps -p "$MASTER_PID" -o args= | grep -F -- "$LAB/active.cfg" >/dev/null
ps -p "$BACKEND_PID" -o args= | grep -F -- "$LAB/backend.py" >/dev/null

kill -TERM "$MASTER_PID" 2>/dev/null || true
kill -TERM "$BACKEND_PID" 2>/dev/null || true
wait "$HAPROXY_JOB_PID" 2>/dev/null || true
wait "$BACKEND_PID" 2>/dev/null || true
! ss -ltn '( sport = :19080 or sport = :19081 )' | grep -q LISTEN
rm -rf -- "$LAB"
test ! -e "$LAB"
printf 'cleanup=PASS listeners=stopped lab=absent\n'

In the complete project runner, evidence remained preserved while both listeners stopped; the public cleanup block removes the disposable copy only after verification.

FAQ: HAProxy Graceful Reload Change Windows

Does systemctl reload haproxy drop active connections?

A correctly configured master-worker reload is designed to let old workers finish admitted connections while a new worker accepts fresh traffic. Prove that behavior on the installed package and real protocol; a successful systemd exit alone is not connection evidence.

Is haproxy -c enough before a reload?

No. It proves the candidate parses with that binary and input set. It does not prove intended ACL order, backend identity, certificate correctness, server reachability, request continuity, or application success.

Why does an old HAProxy worker remain after reload?

It may still own a legitimate long-lived request, WebSocket, tunnel, or HTTP stream. Compare its age and connection evidence with the declared drain budget before forcing it to stop. An unexplained old worker beyond that budget requires diagnosis.

Should I send USR2 to HAProxy in production?

Use the installed package’s documented service-manager reload path. The direct signal in this guide targets only a lab master started by the same shell. Production units may add PID, binary, socket-transfer, privilege, or logging behavior that a copied signal bypasses.

Can HAProxy reload successfully but route traffic incorrectly?

Yes. Syntax can be valid while an ACL, map, backend name, weight, resolver result, or certificate is logically wrong. Send a fresh request carrying an expected route or build marker and check application behavior after reload.

How many requests prove a zero-downtime reload?

There is no universal count. Use a fixed set large enough to cross the handoff and expose the failure mode, plus at least one representative long-lived session. The 1,000-request lab proves this fixture, not an Internet-scale capacity claim.

When should rollback start?

Rollback when a predeclared criterion fails: wrong route identity, unexpected transport errors, elevated 5xx or latency, failed application journey, or an unexplained old-worker age beyond its budget. Validate the saved configuration and run the same acceptance checks after rollback.

Can repeated graceful reloads accumulate old workers?

Yes. A new generation can start while older generations still drain. Repeated changes during long-lived sessions can increase memory, file-descriptor use, and operational ambiguity. Stop the release sequence and identify which sessions own the remaining workers.

Close With a Go or No-Go Record

Record the candidate checksum, installed HAProxy version, exact validation command, reload mechanism, master/current/old worker identities, old-session result, new-route marker, expected and completed request counts, transport errors, application checks, rollback result, and cleanup state. Keep timestamps in one timezone.

A release becomes a go only when the invalid candidate was rejected without changing runtime state, the valid candidate reached the intended new route, old admitted work completed, the fixed traffic set met its error budget, and old workers drained within the protocol contract. Anything less is a reload request—not proof of a safe handoff.

Share this Post

Leave a Reply

Your email address will not be published. Required fields are marked *