HAProxy Backend DOWN: Separate Probe Failure from Service Failure

HAProxy marks a backend server DOWN when its configured health-check contract fails often enough. That result does not, by itself, prove the application is unavailable to every client. A direct browser request may use another hostname, port, protocol, network path, source address, TLS identity, HTTP method, URI, or accepted status code.

Treat the event as a disagreement that must be localized. Preserve HAProxy’s last check result, replay the same request from the load balancer’s network context, correct one proven mismatch, validate the configuration, and return traffic only after both the active check and a representative client request succeed. Forcing a server to ready can hide the evidence while leaving the original failure untouched.

Read DOWN as a probe transcript

Begin with the checker that made the routing decision. Application logs and a successful request from an administrator’s laptop are useful later, but neither explains what HAProxy actually sent and received.

HAProxy’s Runtime API show stat output includes server status and check fields. Query the local Runtime API socket configured on your host; the path varies by package and installation:

printf 'show stat\n' | sudo socat - UNIX-CONNECT:/run/haproxy/admin.sock > /tmp/haproxy-stat.csv
sed -n '1,8p' /tmp/haproxy-stat.csv

Keep the header rather than cutting fixed column numbers. Fields can differ across versions, while names such as status, check_status, check_code, check_duration, and last_chk describe the useful evidence when present. Current HAProxy Runtime API documentation for show stat documents filters and the CSV result.

Pair that snapshot with recent service logs:

sudo journalctl -u haproxy --since '-15 minutes' --no-pager
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

Do not reload yet. The validation command proves only that the file parses; it does not prove the existing check reaches the intended service. Preserve timestamps, backend/server names, transition reason, response code, duration, and whether every server failed together.

Patterns narrow the search quickly. L4CON or connection refusal points toward address, port, listener, route, or firewall. L6RSP suggests a TLS/protocol exchange failed. L7RSP identifies an invalid Layer 7 response or protocol error, while L7STS identifies an HTTP status rejected by the check. A timeout may belong to network loss, application saturation, or an unrealistically short check budget. Exact labels vary, so read them with the HAProxy health-check guidance and the configuration running on this node.

Replay exactly what HAProxy tests

A useful replay preserves the check’s destination and identity. Running curl https://public.example.com/ from a laptop tests public DNS, edge routing, and a client-facing certificate. HAProxy may instead connect to 10.0.2.17:8443, send SNI app.internal, attach Host: app.internal, request /ready, and require status 200.

Execute the probe from the same network namespace as HAProxy. For a containerized load balancer, enter its container or pod. On a host service, use the load-balancer host. If a firewall or application ACL distinguishes source addresses, confirm the process’s real egress source rather than assuming the shell and service are identical.

Prove transport before HTTP

First establish whether the destination accepts a TCP connection:

timeout 5 bash -c '</dev/tcp/10.0.2.17/8443'

A refusal proves the address is reachable but nothing accepts that port, or a policy actively rejects it. A timeout is different: routing, silent filtering, packet loss, or a saturated listener may be involved. When host receive-path counters move during the same interval, Linux softnet packet-drop diagnosis provides the deeper queue and NAPI evidence; it should not be inferred from one failed connect.

Preserve TLS SNI and Host independently

For HTTPS origins, replay the intended hostname while pinning the backend address:

curl --resolve app.internal:8443:10.0.2.17 \
  --header 'Host: app.internal' \
  --connect-timeout 3 --max-time 8 \
  --silent --show-error --output /dev/null \
  --write-out 'status=%{http_code} connect=%{time_connect} total=%{time_total}\n' \
  https://app.internal:8443/ready

--resolve keeps the URL hostname, so curl sends matching SNI while connecting to the selected IP. The explicit Host header reproduces HAProxy’s declared Host: app.internal without curl adding non-default port 8443. A request to https://10.0.2.17:8443/ready is not equivalent when virtual hosting or certificate selection depends on the name.

Inspect the certificate chain separately when verification fails:

openssl s_client -connect 10.0.2.17:8443 -servername app.internal \
  -verify_return_error -verify_hostname app.internal \
  -CAfile /etc/haproxy/ca/internal-ca.pem </dev/null

Never solve an unknown certificate failure by setting verify none as the first change. Wrong SNI, an expired leaf, an incomplete chain, an untrusted private CA, and a certificate for another hostname require different fixes.

Locate the contract that disagrees

Once transport and TLS are visible, compare the live backend stanza with the replay. The failure normally belongs to one of five contracts.

Port or protocol drift

Deployment changes often move readiness to another port while application traffic continues on the old one, or vice versa. A server line can also use a separate check port. Confirm whether check port 8081, http-check connect port 8081, or inherited default-server options override the address shown beside the server.

Protocol mismatch creates misleading symptoms. Plain HTTP sent to a TLS listener may close immediately; TLS sent to a clear-text listener cannot complete a handshake. A basic TCP check proves only that a listener accepts connections. It cannot prove the correct virtual host, URI, dependencies, or response policy.

SNI, Host, method, and path

Virtual hosts can return 404 or redirect when Host is absent. Authentication middleware may reject HAProxy’s default probe, and an endpoint that supports GET may reject HEAD. Treat 301, 401, 403, 404, and 405 as real evidence rather than broadening the accepted range immediately.

To make the HTTP contract reviewable, declare the request and accepted response explicitly:

backend api_pool
    option httpchk
    http-check connect ssl sni app.internal
    http-check send meth GET uri /ready ver HTTP/1.1 hdr Host app.internal
    http-check expect status 200
    default-server inter 5s fastinter 2s downinter 10s fall 3 rise 2
    server api1 10.0.2.17:8443 ssl verify required verifyhost app.internal ca-file /etc/haproxy/ca/internal-ca.pem check

Treat this as an example, not a universal stanza. Use the syntax supported by the installed HAProxy version, and keep health-check SNI, verifyhost, and HTTP Host aligned with the certificate and virtual host. HAProxy 3.3’s current configuration manual documents http-check connect, http-check send, http-check expect, server TLS options, and check intervals.

Timing and capacity

One slow check is not the same as sustained failure. Compare check_duration with the configured check timeout, application latency, connection backlog, CPU pressure, and dependency time. Raising a timeout can reduce false negatives, but it also delays removal of a genuinely failed backend.

Capacity matters when all servers become slow together. Keep enough CPU, memory, file descriptors, and network reserve on the load balancer and application nodes; VPS hosting capacity choices should reflect both normal requests and health-check/recovery overhead. Do not use a larger timeout to conceal saturation that already violates client latency.

Source identity and policy

Backend firewalls, service meshes, security groups, and application ACLs may permit administrators while rejecting the load balancer subnet. Record the source IP seen by the backend. Container bridges and policy routing can make it differ from the host address operators expect.

Proxy-aware security also depends on which hop owns the client address. If troubleshooting reveals bans or allowlists applied to the proxy identity, reverse-proxy client-IP enforcement explains the separate trust and reachable-enforcement problem. Do not copy forwarded-header trust settings into a health endpoint merely to make a check pass.

Change one condition, then validate and reload

Choose the smallest correction supported by the evidence: restore the listener, repair routing, install the right CA chain, set the check’s SNI and Host, correct the URI or method, or tune a timeout that is demonstrably below healthy latency. Do not change path, expected status, TLS verification, and thresholds together. A bundled edit destroys attribution.

Before deployment, save the current configuration in the host’s normal backup system, review the diff, and validate the complete file:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
sudo systemctl is-active haproxy

A validated reload preserves established connections when the service unit and package are configured for graceful reload, but verify the behavior of your distribution and unit. Caddy’s validate-before-reload workflow covers the same operational principle for a different proxy; HAProxy still requires its own syntax and runtime checks.

Watch multiple check intervals. With fall 3, three consecutive failures are required to mark the server down; with rise 2, two consecutive successes are required to mark it up. Those counters prevent one transient result from flipping state, but oversized values also extend removal or recovery time. Calculate the approximate transition window from the applicable interval, then confirm the real timestamps instead of treating the estimate as a guarantee.

Runtime state changes are temporary and can bypass intended safety. Avoid set server ... state ready as a substitute for repairing the check: Runtime API changes live in memory, may disappear at reload, and can place traffic on a server whose probe contract still fails.

Return traffic only after both views agree

Green server state is one receipt, not the whole acceptance test. Capture show stat again after enough successful intervals, confirm the expected UP transition, and ensure check duration stays comfortably inside its timeout. Then send one representative request through the client-facing HAProxy listener and verify the chosen backend’s application log, response status, and latency.

Keep an observer outside the same failure domain. Uptime Kuma placement guidance helps prevent a monitor from disappearing with the load balancer or application host it is meant to watch.

Close the incident when the check contract, backend log, Runtime API state, real client path, and external observation tell the same story. Record the original mismatch, exact change, validation output, reload timestamp, recovery intervals, and rollback boundary. HAProxy UP should mean the routed service is ready—not merely that an operator found a way to paint the row green.