An NGINX mTLS Gate Must Reject the Wrong Client CA

Successful admission is the wrong place to stop an NGINX mutual TLS test. The useful proof is the three-outcome contract around that request: a client with no certificate is rejected, a client signed by the intended client CA reaches the upstream, and a certificate signed by another CA is rejected before the upstream sees it. In the reproduced lab, those outcomes were 400, 200, and 400, while the upstream recorded exactly one request.

This guide is for Linux and reverse-proxy operators who already understand HTTPS, can read an NGINX server block, and are authorized to handle private keys. It uses an isolated loopback listener, short-lived lab certificates, and a marker-owned directory. Nothing touches the system NGINX service, public DNS, firewall rules, or a production certificate store.

Start with the 400, 200, 400 contract

Mutual TLS, usually shortened to mTLS, adds client authentication to ordinary TLS. The client still validates the server certificate and hostname. NGINX then requests a client certificate and validates that certificate against a separate trust store. Encryption alone is therefore not the acceptance condition; the server must also prove that it rejected identities outside the intended client CA.

Three results define the minimum useful receipt:

• No client certificate: NGINX returns 400 No required SSL certificate was sent, and the upstream request count stays unchanged.
• Trusted client certificate: NGINX reports $ssl_client_verify as SUCCESS, forwards the request, and the upstream sees the client subject supplied by NGINX.
• Wrong client CA: NGINX returns 400 The SSL certificate error, and the upstream request count again stays unchanged.

Those negative controls close a gap left by many setup pages. A single curl --cert ... response proves only that one credential worked. It does not prove that ssl_client_certificate names the intended CA, that verification is mandatory, or that an untrusted client is stopped before application code.

Keep server trust and client trust separate

An mTLS connection contains two independent certificate decisions. Curl uses the lab server CA to authenticate mtls-lab.internal. NGINX uses the trusted-client CA to authenticate trusted-client. The server leaf is not a client issuer, and the client leaf is not a server trust anchor.

NGINX documents ssl_client_certificate as the PEM bundle used to verify client certificates. The same official HTTP SSL module reference explains that ssl_verify_client on requires verification and that $ssl_client_verify exposes the result. ssl_verify_depth limits intermediate certificates below the trusted root; it does not repair a missing chain or make the server certificate a client CA.

Server identity still has its own checks. Before debugging client admission on a public endpoint, compare the certificate served by the live TLS endpoint to confirm that NGINX actually presents the intended leaf. The network endpoint, protocol mode, presented chain, and expected name must all agree before client admission becomes the next question.

Certificate purpose matters as well. The lab server leaf carries serverAuth; both client leaves carry clientAuth. RFC 5280 defines extended key usage processing, while OpenSSL exposes those extensions during issuance and inspection. A production CA policy should enforce those roles instead of relying only on a recognizable common name.

Build three CA paths in a disposable workspace

Start a fresh unprivileged Bash session and run every block in that one disposable session. This keeps the lab-only shell options away from an administrative working shell. The first block atomically creates an unpredictable private workspace, refuses occupied loopback ports, records ownership, and installs an exit trap. Port 18443 is the isolated NGINX listener; port 18081 belongs to the disposable upstream.

set -euo pipefail
nginx_port=18443
backend_port=18081

command -v nginx openssl curl python3 ss >/dev/null
if ss -H -ltn "sport = :$nginx_port or sport = :$backend_port" | grep -q .; then
  printf 'A required loopback port is already listening.\n' >&2
  exit 1
fi
lab_root=$(mktemp -d "${TMPDIR:-/tmp}/voxfor-nginx-mtls-163.XXXXXX")
chmod 0700 "$lab_root"
install -d -m 0700 "$lab_root/certs"
printf 'voxfor-nginx-mtls-163\n' > "$lab_root/OWNER"

validate_lab_owner() {
  [[ -d "$lab_root" ]]
  [[ "$lab_root" == "${TMPDIR:-/tmp}"/voxfor-nginx-mtls-163.* ]]
  [[ -O "$lab_root" ]]
  [[ "$(<"$lab_root/OWNER")" == voxfor-nginx-mtls-163 ]]
}

stop_owned_process() {
  local pidfile=$1 required_arg=$2 pid cmdline
  [[ -f "$pidfile" ]] || return 0
  pid=$(<"$pidfile")
  [[ "$pid" =~ ^[0-9]+$ ]] || return 1
  [[ -r "/proc/$pid/cmdline" ]] || return 0
  cmdline=$(tr '\0' ' ' < "/proc/$pid/cmdline")
  [[ " $cmdline " == *" $required_arg "* ]] || {
    printf 'Refusing PID %s with unexpected command: %s\n' "$pid" "$cmdline" >&2
    return 1
  }
  kill -TERM "$pid"
  for _ in {1..50}; do
    kill -0 "$pid" 2>/dev/null || return 0
    sleep 0.1
  done
  printf 'Owned PID %s did not exit.\n' "$pid" >&2
  return 1
}

cleanup_processes() {
  validate_lab_owner || return 1
  stop_owned_process "$lab_root/nginx.pid" "$lab_root/nginx.conf"
  stop_owned_process "$lab_root/backend.pid" "$lab_root/backend.py"
}
trap cleanup_processes EXIT
printf 'nginx=%s openssl=%s curl=%s\n' \
  "$(nginx -v 2>&1 | sed 's#^nginx version: nginx/##')" \
  "$(openssl version | awk '{print $2}')" \
  "$(curl --version | awk 'NR==1 {print $2}')"

Three short-lived roots come next: one for the server, one NGINX should trust for clients, and one deliberate wrong issuer. Lab keys are unencrypted only to keep this disposable sequence non-interactive; the directory is mode 0700, keys become 0600, and cleanup removes them. Do not copy that key-storage choice into production.

make_ca() {
  local stem=$1 subject=$2
  openssl req -x509 -newkey rsa:2048 -nodes -days 2 -sha256 \
    -subj "$subject" \
    -keyout "$lab_root/certs/$stem.key" \
    -out "$lab_root/certs/$stem.crt" >/dev/null 2>&1
}

make_ca server-ca '/CN=Voxfor mTLS lab server CA'
make_ca trusted-client-ca '/CN=Voxfor mTLS trusted client CA'
make_ca wrong-client-ca '/CN=Voxfor mTLS wrong client CA'

Leaf issuance is explicit about role and hostname. subjectAltName=DNS:mtls-lab.internal lets curl verify the lab name, while extendedKeyUsage=clientAuth prevents the two client credentials from pretending to be server certificates under a conforming verifier.

make_leaf() {
  local stem=$1 subject=$2 ca=$3 eku=$4 san=$5
  openssl req -newkey rsa:2048 -nodes -sha256 \
    -subj "$subject" \
    -keyout "$lab_root/certs/$stem.key" \
    -out "$lab_root/certs/$stem.csr" >/dev/null 2>&1
  {
    printf 'basicConstraints=critical,CA:FALSE\n'
    printf 'keyUsage=critical,digitalSignature,keyEncipherment\n'
    printf 'extendedKeyUsage=%s\n' "$eku"
    [[ -n "$san" ]] && printf 'subjectAltName=%s\n' "$san"
  } > "$lab_root/certs/$stem.ext"
  openssl x509 -req -days 2 -sha256 \
    -in "$lab_root/certs/$stem.csr" \
    -CA "$lab_root/certs/$ca.crt" \
    -CAkey "$lab_root/certs/$ca.key" \
    -CAcreateserial -extfile "$lab_root/certs/$stem.ext" \
    -out "$lab_root/certs/$stem.crt" >/dev/null 2>&1
}

make_leaf server '/CN=mtls-lab.internal' server-ca serverAuth 'DNS:mtls-lab.internal'
make_leaf trusted-client '/CN=trusted-client' trusted-client-ca clientAuth ''
make_leaf wrong-client '/CN=wrong-client' wrong-client-ca clientAuth ''
chmod 0600 "$lab_root"/certs/*.key
openssl verify -CAfile "$lab_root/certs/trusted-client-ca.crt" "$lab_root/certs/trusted-client.crt"
! openssl verify -CAfile "$lab_root/certs/trusted-client-ca.crt" "$lab_root/certs/wrong-client.crt"

One negated command is intentional. It confirms that the wrong leaf cannot chain to the trusted client root before NGINX is involved. OpenSSL’s verification command documents the trust-store and purpose controls behind that check.

Make NGINX the admission point, not the application

To prove rejection happens before application code, the lab uses a tiny upstream that records every request and echoes only the verification headers inserted by NGINX. A real application must accept those headers only from a protected proxy path; an internet client must not be able to reach the upstream directly and forge them.

cat > "$lab_root/backend.py" <<'PY'
from http.server import BaseHTTPRequestHandler, HTTPServer
import json
import sys

log_path = sys.argv[1]

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        body = json.dumps({
            "upstream": "reached",
            "client_verify": self.headers.get("X-Client-Verify"),
            "client_subject": self.headers.get("X-Client-Subject"),
        }, sort_keys=True).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        with open(log_path, "a", encoding="utf-8") as stream:
            stream.write((fmt % args) + "\n")

HTTPServer(("127.0.0.1", 18081), Handler).serve_forever()
PY

Isolation matters here: the NGINX instance has its own prefix, PID, logs, and loopback socket. ssl_client_certificate points only to the trusted client root. The wrong root never appears in the configuration.

cat > "$lab_root/nginx.conf" <<EOF
pid $lab_root/nginx.pid;
error_log $lab_root/error.log info;
events { worker_connections 64; }
http {
    access_log $lab_root/access.log combined;
    server {
        listen 127.0.0.1:$nginx_port ssl;
        server_name mtls-lab.internal;
        ssl_certificate $lab_root/certs/server.crt;
        ssl_certificate_key $lab_root/certs/server.key;
        ssl_client_certificate $lab_root/certs/trusted-client-ca.crt;
        ssl_verify_client on;
        ssl_verify_depth 1;
        ssl_protocols TLSv1.2 TLSv1.3;
        location / {
            proxy_set_header X-Client-Verify \$ssl_client_verify;
            proxy_set_header X-Client-Subject \$ssl_client_s_dn;
            proxy_pass http://127.0.0.1:$backend_port;
        }
    }
}
EOF

Syntax validation comes before either listener starts. nginx -t checks file readability and directive validity with the same prefix and configuration that the lab will run.

nginx -t -p "$lab_root/" -c "$lab_root/nginx.conf"
python3 "$lab_root/backend.py" "$lab_root/backend-access.log" >"$lab_root/backend.stdout" 2>"$lab_root/backend.stderr" &
printf '%s\n' "$!" > "$lab_root/backend.pid"
nginx -p "$lab_root/" -c "$lab_root/nginx.conf"

for _ in {1..20}; do
  if ss -H -ltn "sport = :$nginx_port" | grep -q . && \
     ss -H -ltn "sport = :$backend_port" | grep -q .; then
    break
  fi
  sleep 0.1
done
ss -H -ltn "sport = :$nginx_port or sport = :$backend_port"
curl_base=(--silent --show-error \
  --cacert "$lab_root/certs/server-ca.crt" \
  --resolve "mtls-lab.internal:$nginx_port:127.0.0.1")

Run the negative controls before you trust the 200

Begin without a client certificate. Curl still verifies the server CA and hostname; it merely omits the client identity NGINX requires.

no_cert_status=$(curl "${curl_base[@]}" \
  -o "$lab_root/no-cert.body" -w '%{http_code}' \
  "https://mtls-lab.internal:$nginx_port/")
printf 'no_client_cert http=%s\n' "$no_cert_status"
[[ "$no_cert_status" == 400 ]]
[[ ! -e "$lab_root/backend-access.log" ]]
printf 'no_client_cert http=%s upstream_requests=0_expected\n' "$no_cert_status"

Next, present the leaf signed by the trusted client CA. The response comes from the upstream, not a static NGINX success page, and includes SUCCESS plus the client subject NGINX derived after verification.

trusted_status=$(curl "${curl_base[@]}" \
  --cert "$lab_root/certs/trusted-client.crt" \
  --key "$lab_root/certs/trusted-client.key" \
  -o "$lab_root/trusted.body" -w '%{http_code}' \
  "https://mtls-lab.internal:$nginx_port/")
python3 -m json.tool "$lab_root/trusted.body"
trusted_verify=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["client_verify"])' "$lab_root/trusted.body")
trusted_subject=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["client_subject"])' "$lab_root/trusted.body")
[[ "$trusted_status" == 200 ]]
[[ "$trusted_verify" == SUCCESS ]]
[[ "$trusted_subject" == CN=trusted-client ]]

Finally, send the structurally valid client leaf signed by the wrong CA. This is stronger than testing a missing file or malformed certificate: the credential can sign the handshake, but its issuer is outside the trust store.

set +e
wrong_status=$(curl "${curl_base[@]}" \
  --cert "$lab_root/certs/wrong-client.crt" \
  --key "$lab_root/certs/wrong-client.key" \
  -o "$lab_root/wrong.body" -w '%{http_code}' \
  "https://mtls-lab.internal:$nginx_port/")
wrong_exit=$?
set -e

backend_requests=$(wc -l < "$lab_root/backend-access.log")
printf 'trusted_client http=%s verify=%s subject=%s\n' "$trusted_status" "$trusted_verify" "$trusted_subject"
printf 'wrong_client http=%s curl_exit=%s\n' "$wrong_status" "$wrong_exit"
printf 'backend_request_count=%s expected=1\n' "$backend_requests"
printf 'nginx_access_statuses=%s\n' "$(awk '{print $9}' "$lab_root/access.log" | paste -sd, -)"
[[ "$wrong_status" == 400 || "$wrong_status" == 000 ]]
[[ "$backend_requests" == 1 ]]
grep -F 'client SSL certificate verify error: (21:unable to verify the first certificate)' "$lab_root/error.log" >/dev/null
printf 'wrong_issuer_error=unable_to_verify_first_certificate\n'

Read the receipt as a security boundary

Reproduction used Debian 13, NGINX 1.26.3, OpenSSL 3.5.6, and curl 8.14.1. Its representative output was:

nginx=1.26.3 openssl=3.5.6 curl=8.14.1
no_client_cert http=400 upstream_requests=0_expected
trusted_client http=200 verify=SUCCESS subject=CN=trusted-client
wrong_client http=400 curl_exit=0
backend_request_count=1 expected=1
nginx_access_statuses=400,200,400
wrong_issuer_error=unable_to_verify_first_certificate
cleanup=complete path_absent=yes

The mTLS gate is accepted only when the no-certificate and wrong-CA requests are rejected, the trusted client returns 200 with SUCCESS, and the upstream log contains exactly one request. That combination proves both trust selection and pre-application rejection; a successful handshake, open port, or 200 alone does not.

NGINX access status 400,200,400 provides the edge view. The upstream count supplies the ownership boundary. When proxy and application timings later diverge, NGINX request and upstream timing evidence helps separate admission time from application work without weakening client verification.

Carry the lab into production without carrying its shortcuts

Short-lived self-issued lab credentials are evidence fixtures, not deployment advice. Production mTLS needs a named CA owner, protected offline or hardware-backed signing keys, per-client identity, issuance approval, expiry monitoring, revocation, and an emergency removal path. The OpenSSL x509 reference can inspect dates, purposes, fingerprints, and extensions, but a command does not create governance.

Prefer one client certificate per workload or device. Sharing one private key across a fleet erases attribution and makes one compromise a fleet-wide event. Authorization should use a stable reviewed identity—often a SAN URI or another constrained field—not an unparsed subject string blindly forwarded to application code.

Keep the upstream private. If clients can reach it directly, they can bypass NGINX and forge X-Client-Verify or X-Client-Subject. Network policy, a Unix socket, a loopback binding, or mTLS on the proxy-to-upstream leg can enforce the boundary. This is related to, but distinct from, correct client-IP ownership behind a proxy; reverse-proxy client-IP enforcement shows why applications and security tools must trust only the component that truly owns the signal.

Monitoring must exercise the protected path with an authorized synthetic identity and protect that probe key like any other client key. A generic HTTPS probe can validate the server certificate yet still fail client admission. Build the mTLS-specific test beside HTTP, DNS, and TLS probes rather than treating those network checks as an application authorization result.

Cut over only with a rollback owner

Before production reload, save the active NGINX configuration, certificate references, permissions, current server-chain receipt, and a known-good client test. Run nginx -t, stage a second operator session, and decide whether rollback restores the previous server block or temporarily removes only the new mTLS requirement. Do not improvise by trusting both the intended and wrong CA; that converts a test failure into broader access.

Before exposing the endpoint, assign one operator to issuance, NGINX reloads, revocation, logging, and emergency access. If the team lacks that owner, compare those responsibilities with the documented managed server operations scope, which covers server configuration, security work, updates, and network diagnostics. That is a delegation checkpoint, not a substitute for the certificate acceptance tests above.

Cleanup kills only PIDs recorded under the marker path, confirms the marker, removes that exact directory, and checks both listeners are absent.

cleanup_processes
trap - EXIT
validate_lab_owner
cleanup_target=$lab_root
rm -rf -- "$lab_root"
[[ ! -e "$cleanup_target" ]]
! ss -H -ltn "sport = :$nginx_port or sport = :$backend_port" | grep -q .
printf 'cleanup=complete path_absent=yes\n'

If any validation or request result differs, stop the isolated NGINX and backend PIDs recorded in the marker directory, keep production unchanged, and inspect the lab logs and certificate chains before retrying. On a production rollout, restore the backed-up server block and certificate references, run nginx -t, reload, and repeat the previous known-good external acceptance path; never delete unfamiliar keys, CAs, or configuration files during rollback.