Fail2ban Is Banning Your Proxy, Not the Attacker

A Fail2ban ban can look successful while blocking nothing. Behind Cloudflare or another reverse proxy, the web server may receive the proxy as the TCP peer, recover a visitor address from a trusted header, write that visitor into a log, and then ask a host firewall to ban an address that never appears as the packet source. Detection and enforcement must agree on the same reachable address boundary.

Safe repair has four parts: prove the socket peer, trust client-IP headers only from current proxy networks, verify which address reaches the log and Fail2ban filter, then place the ban where traffic can actually be rejected. Never solve attribution by trusting X-Forwarded-For from every source; that turns an attacker-controlled header into security evidence.

Four addresses decide whether a ban can work

One request can carry four different addresses. The transport peer is visible to the kernel. The original peer remains available to NGINX after real-IP rewriting. The restored client becomes NGINX $remote_addr when a trusted proxy supplies an accepted header. Finally, the ban target is the address extracted by Fail2ban.

Confusion begins when those roles are collapsed into one phrase such as “real IP.” A direct visitor has the same transport and client address. A visitor passing through Cloudflare does not: the origin socket sees a Cloudflare edge address, while CF-Connecting-IP carries the visitor address. Multi-hop X-Forwarded-For can contain several addresses and needs an explicit trust-chain model.

Trace the socket peer before changing NGINX

Capture current listener, active configuration and a short access-log sample before editing anything. The commands below are read-only:

sudo ss -ntp '( sport = :443 )'
sudo nginx -T 2>&1 | sudo tee /root/nginx-effective-before.txt >/dev/null
sudo tail -n 30 /var/log/nginx/access.log
sudo fail2ban-client status

Compare a known request timestamp with proxy analytics or a controlled external request. If every TLS connection comes from a proxy network but the log also shows the proxy address, attribution is missing. If logs show unique client addresses yet bans have no effect, enforcement is probably at the wrong layer.

Operators using a different proxy should confirm its documented header and trusted egress ranges. Caddy configuration validation and safe reload practice remains useful for Caddy deployments, but NGINX directives in this article do not transfer verbatim.

Make the header trustworthy before using it

NGINX ngx_http_realip_module changes the client address only when the current peer matches set_real_ip_from. Official NGINX real-IP documentation also explains that $realip_remote_addr preserves the original peer. Trust must be narrow enough that an Internet client cannot connect directly and nominate any address it wants.

For Cloudflare-proxied HTTP, Cloudflare documents CF-Connecting-IP as a single client address sent from its edge to the origin. Cloudflare recommends it over parsing X-Forwarded-For when the origin needs one consistent address. Current edge networks belong in a maintained include sourced from the definitive Cloudflare IP range list, not copied once and forgotten.

Cloudflare Workers can change that meaning. Same-zone Worker subrequests derive CF-Connecting-IP from a Worker-controlled x-real-ip, while cross-zone subrequests use a fixed Cloudflare address. Test Worker routes as a separate trust path before treating the field as an end-user identity.

Pseudo IPv4 also needs an explicit check. When Overwrite Headers is enabled, Cloudflare replaces CF-Connecting-IP and X-Forwarded-For with a pseudo IPv4 value and preserves the original IPv6 address in CF-Connecting-IPv6. Dual-stack bans must therefore use the field and enforcement action that represent the intended identity; do not assume the normal header still contains the visitor IPv6 address.

Restore one documented address, from known senders only

Keep provider CIDRs in a separate file so range updates are reviewable. The placeholders below are intentionally not deployable; replace every placeholder with current provider-published IPv4 and IPv6 networks before testing.

# /etc/nginx/conf.d/trusted-proxies.conf
set_real_ip_from PROXY_IPV4_CIDR;
set_real_ip_from PROXY_IPV6_CIDR;

real_ip_header CF-Connecting-IP;
real_ip_recursive off;

Do not use set_real_ip_from 0.0.0.0/0 or ::/0. With that configuration, a direct origin request can forge the header and make logs, rate limits and bans blame another host. Restrict public origin ports to approved proxy networks when the application should never be reached directly. Keycloak proxy-header boundary guidance demonstrates the same broader principle: forwarded identity is trustworthy only when sender, header contract and public origin are defined together.

Some stacks have another hop between edge and NGINX, such as a load balancer, container bridge or ingress controller. In that case, document the complete chain before enabling real_ip_recursive. Its NGINX behavior selects the last non-trusted address from the configured header chain; guessing which hops are trusted can expose spoofing or misattribute shared proxies.

Turn logs into an attribution receipt

During the change window, log both perspectives. $realip_remote_addr records the original peer that reached NGINX, while $remote_addr records the address after trusted real-IP processing. A temporary audit format makes disagreements visible without replacing every production log immediately.

log_format ban_audit '$time_iso8601 peer=$realip_remote_addr client=$remote_addr request="$request" status=$status';
access_log /var/log/nginx/ban-audit.log ban_audit;

After nginx -t passes and configuration is reloaded, make controlled requests through the proxy and, where origin policy permits, one direct negative test. Expected evidence has three properties: peer= belongs to an approved proxy CIDR, client= matches the controlled external source, and a direct untrusted source cannot change client= by submitting the proxy header.

Prove header spoofing fails at the origin boundary

Use a documentation-only address in the header and replace the host plus origin placeholders with a test endpoint. A secure origin either rejects the direct connection before NGINX or logs the actual direct source as client, not the forged value.

curl --resolve app.example.net:443:ORIGIN_IP -H 'CF-Connecting-IP: 198.51.100.77' -o /dev/null -sS -w '%{http_code}\n' https://app.example.net/protected-test
sudo tail -n 10 /var/log/nginx/ban-audit.log

Do not run an origin-bypass test against an endpoint whose change policy forbids it. Use a staging hostname or a pre-approved maintenance route. Remove the temporary audit log after the evidence is retained, unless its storage and privacy impact have been accepted.

Test Fail2ban against evidence, not production traffic

Fail2ban needs a log event that represents a real failure, not merely any 401, 403 or 404. Applications can emit those statuses during normal operation, and broad patterns create false bans. Prefer a maintained filter that matches the authentic service event; custom filters should anchor the surrounding message and extract exactly one <HOST> value.

Official Fail2ban configuration documentation recommends keeping distributed .conf files unchanged and placing local overrides in .local files. Preserve that upgrade boundary. For WordPress, combine network controls with application evidence such as login rate limits, 2FA and WAF signals described in WordPress brute-force protection guidance; one noisy access-log status is not proof of an authentication attack.

Run the filter offline first

Test the exact production log and filter before enabling or reloading a jail. fail2ban-regex exists specifically to show matched, missed and ignored lines, as documented by the Fail2ban regex manual.

sudo fail2ban-regex /var/log/nginx/error.log /etc/fail2ban/filter.d/nginx-http-auth.conf --print-all-matched
sudo fail2ban-regex /var/log/nginx/ban-audit.log /etc/fail2ban/filter.d/APP_FILTER.conf --print-all-matched

The second command is a template for an application-specific filter, not a universal file name. Review missed malicious samples and matched benign samples. A high match count is not success if the extracted address is the proxy or if ordinary users match the same expression.

Inspect the effective jail after reload

Once offline results are defensible, validate configuration and inspect the running jail. Keep a separate administration session open for rollback.

sudo fail2ban-client -t
sudo fail2ban-client reload APP_JAIL
sudo fail2ban-client status APP_JAIL
sudo fail2ban-client get APP_JAIL logpath
sudo fail2ban-client get APP_JAIL ignoreip

Confirm the expected log path, filter, findtime, maxretry, bantime, action and ignore list from effective configuration rather than memory. Avoid placing every proxy CIDR in ignoreip as a substitute for correct attribution; doing so may hide the only address currently reaching a broken filter.

Choose an enforcement layer that matches the packet path

Detection answers “who failed?” Enforcement answers “where can that identity be stopped?” When Cloudflare terminates the visitor connection, the origin host firewall sees Cloudflare as the packet source. An nftables or iptables rule banning the restored visitor address will not match those proxied packets because header rewriting occurs inside NGINX after the kernel accepted the connection.

Three enforcement patterns can be valid:

1. Edge enforcement updates a provider firewall, WAF or rate-limit rule so the visitor is rejected before the proxy opens an origin connection.
2. Request-layer enforcement rejects the restored address inside NGINX or the application after a strictly trusted real-IP transformation.
3. Host-firewall enforcement works when the banned visitor address is also the transport source, such as direct client traffic or a transparent path that preserves source addressing.

Pick one pattern deliberately. Edge API actions need least-privilege credentials, bounded scope, rate-limit handling, auditable unban behavior and a fallback when the provider API is unavailable. Request-layer deny files need atomic updates, syntax validation and a safe reload. Host-firewall actions need IPv4 and IPv6 parity plus the correct input chain.

Large floods should be mitigated upstream before they consume origin bandwidth and connection state. Fail2ban is better suited to repeated abusive events that the origin can identify reliably.

Verify the ban at both control and data planes

A controlled test must prove more than a name appearing in fail2ban-client status. Record the extracted client, selected action, target layer, ban timestamp and expiry. Then verify that the same controlled source is rejected at that layer while an unrelated source and proxy health checks still succeed.

If the status says “banned” but requests continue, stop increasing bantime. Recheck whether the action targets the host firewall while traffic arrives from a proxy, whether an IPv6 client bypasses an IPv4-only rule, whether another hostname follows a different proxy route, or whether cached responses hide origin access.

Run a bounded cutover and leave an acceptance record

Stage the change with two sessions: one performs the work, while another preserves administrative access and rollback. For management-plane isolation, Tailscale SSH access controls can keep operator access separate from the public web path being tested.

Use this order:

1. Save effective NGINX and Fail2ban configuration plus sample pre-change logs.
2. Update reviewed proxy CIDRs and the selected single-address header.
3. Run nginx -t, reload NGINX, and verify peer/client evidence through proxy plus direct negative path.
4. Test the filter offline, validate Fail2ban configuration, then reload only Fail2ban.
5. Trigger one approved failure sequence from a controlled IPv4 source and one controlled IPv6 source when the service supports both.
6. Confirm rejection at the chosen enforcement point, continued proxy health, a successful unrelated request and a working unban.
7. Remove temporary diagnostics or assign retention, then record the range-refresh owner and next test date.

Rollback means restoring the prior NGINX include and log format, validating and reloading NGINX, restoring the prior jail override, validating and reloading Fail2ban, and removing only rules created by the change. Do not flush a whole firewall or delete every ban to undo one failed test.

The final receipt should name transport peer, trusted sender list, restored header, monitored log, filter result, action, enforcement layer, IPv4 and IPv6 outcome, negative spoof test, unban proof, proxy health and rollback artifact. A ban counter is only a control-plane claim; the incident closes when unwanted traffic is actually rejected without sacrificing the proxy.