Test SMTP STARTTLS Certificate Trust and Hostname

An SMTP server can advertise STARTTLS, complete a TLS handshake, and still present an identity that a careful client must reject. Encryption only proves that a session became unreadable in transit. It does not, by itself, prove that the peer certificate chains to a trusted root, covers the hostname the operator intended to reach, or leaves SMTP usable after the upgrade.

This audit treats those as separate gates. It inventories the endpoint, records the plaintext SMTP capability, enforces chain and hostname validation, reads the certificate actually served, runs two deliberate negative controls, and confirms that EHLO still succeeds inside TLS. The result is a cutover receipt rather than a reassuring screenshot of a padlock.

Reproduction ran on Debian 13.6 with OpenSSL 3.5.6, Node.js 24.18.0, and a loopback-only SMTP fixture on port 2525. That fixture creates a disposable root and a two-day certificate for the reserved name smtp.audit.test. It sends no mail, changes no production MTA, and removes only its marker-owned temporary directory.

Encryption Is Not Authenticated SMTP Identity

SMTP begins as a plaintext application protocol on ports such as 25 and 587. RFC 3207 defines the STARTTLS extension: the server advertises the capability, the client requests it, the server returns 220, and the parties negotiate TLS. After the handshake, the client must discard knowledge learned from the earlier session and issue EHLO again.

That sequence answers an upgrade question, not every identity question. A useful acceptance test needs four independent results:

1. The intended SMTP listener advertises STARTTLS before credentials or message content are sent.
2. The served chain validates against the trust store selected for the test.
3. The certificate’s Subject Alternative Name covers the hostname the client is meant to verify.
4. SMTP works after TLS, demonstrated by a new successful EHLO response.

OpenSSL’s current s_client documentation matters here because -verify alone is diagnostic: verification errors can be printed while the connection continues. -verify_return_error makes the command stop on an error, while -verify_hostname applies the hostname assertion. A line that merely says the handshake completed is therefore not a pass.

Inventory Every Endpoint You Expect to Trust

Start with the delivery route rather than a hostname copied from a control panel. MX records identify inbound delivery hosts; submission on port 587 or implicit TLS on 465 may use different names, certificates, and load balancers. Record each service separately, including its port, expected verification name, IP family, and observing network.

set -Eeuo pipefail
DOMAIN=voxfor.com
MX=$(dig +short MX "$DOMAIN" | sort -n)
printf 'domain=%s\n%s\n' "$DOMAIN" "$MX"
grep -Eq '^[0-9]+[[:space:]]+[^[:space:]]+\.$' <<< "$MX"

Our tested public lookup returned priority 10 and mail.voxfor.com.. That public fact is only an inventory example; it is not permission to scan or reconfigure the server. For another domain, replace DOMAIN, preserve every returned MX hostname, and add the separately documented submission endpoint.

MX delivery and message submission are different products

For ordinary MX delivery, verify the actual MX hostname reached by the client unless a stronger policy such as MTA-STS or DANE defines a different reference identity. For authenticated submission, verify the hostname configured in the mail client. Do not test example.com merely because it is the address domain if the client connects to smtp.example.net.

Test every relevant backend. A certificate can pass on IPv4 and fail on IPv6, or pass on the first MX while a lower-priority listener serves an expired chain. This is the same reason a TLS renewal endpoint audit must inspect the certificate at the live termination point rather than only the certificate file on disk.

Build a Disposable STARTTLS Listener

One loopback port and one exact directory define the fixture’s ownership boundary. It generates a local root, signs a leaf certificate whose SAN is smtp.audit.test, writes a minimal Node SMTP state machine, and records the server PID. Its marker makes every later block refuse to run against an unrelated path.

set -Eeuo pipefail
LAB=/tmp/voxfor-smtp-starttls-140
MARKER=voxfor-smtp-starttls-lab-140
[[ ! -e "$LAB" ]]
! ss -Hln 'sport = :2525' | grep -q .
install -d -m 0700 "$LAB"
printf '%s\n' "$MARKER" > "$LAB/.marker"
openssl req -x509 -newkey rsa:2048 -nodes -days 2 \
  -subj '/CN=Voxfor SMTP Lab Root' \
  -keyout "$LAB/root.key" -out "$LAB/root.pem" >/dev/null 2>&1
openssl req -newkey rsa:2048 -nodes \
  -subj '/CN=smtp.audit.test' \
  -addext 'subjectAltName=DNS:smtp.audit.test' \
  -keyout "$LAB/server.key" -out "$LAB/server.csr" >/dev/null 2>&1
printf 'subjectAltName=DNS:smtp.audit.test\nextendedKeyUsage=serverAuth\n' > "$LAB/server.ext"
openssl x509 -req -days 2 -sha256 -in "$LAB/server.csr" \
  -CA "$LAB/root.pem" -CAkey "$LAB/root.key" -CAcreateserial \
  -extfile "$LAB/server.ext" -out "$LAB/server.pem" >/dev/null 2>&1
cat > "$LAB/server.mjs" <<'EOF'
import fs from 'node:fs'; import net from 'node:net'; import tls from 'node:tls';
const dir=process.argv[2];
const secure=tls.createServer({key:fs.readFileSync(`${dir}/server.key`),cert:fs.readFileSync(`${dir}/server.pem`)},s=>protocol(s,true));
function protocol(s,encrypted){let b='';s.setEncoding('utf8');s.on('error',()=>{});s.on('data',c=>{b+=c;while(b.includes('\n')){const i=b.indexOf('\n');const q=b.slice(0,i).replace(/\r$/,'').trim();b=b.slice(i+1);if(/^EHLO\b/i.test(q))s.write(encrypted?'250-smtp.audit.test\r\n250-SIZE 10485760\r\n250 HELP\r\n':'250-smtp.audit.test\r\n250-STARTTLS\r\n250 SIZE 10485760\r\n');else if(q==='STARTTLS'&&!encrypted){s.removeAllListeners('data');s.write('220 2.0.0 Ready to start TLS\r\n',()=>secure.emit('connection',s));return}else if(q==='QUIT')s.end('221 2.0.0 Bye\r\n');else if(q)s.write('502 5.5.2 Not implemented\r\n')}})}
const server=net.createServer(s=>{s.write('220 smtp.audit.test ESMTP Voxfor STARTTLS Lab\r\n');protocol(s,false)});
server.listen(2525,'127.0.0.1',()=>console.log('ready=127.0.0.1:2525'));
const close=()=>server.close(()=>secure.close(()=>process.exit(0)));process.on('SIGTERM',close);process.on('SIGINT',close);
EOF
node "$LAB/server.mjs" "$LAB" > "$LAB/server.log" 2>&1 &
echo $! > "$LAB/server.pid"
for _ in {1..40}; do grep -q '^ready=' "$LAB/server.log" && break; sleep 0.05; done
grep -qx 'ready=127.0.0.1:2525' "$LAB/server.log"
kill -0 "$(<"$LAB/server.pid")"

A two-day leaf lifetime is deliberate: this is a lab identity, not a reusable certificate. Keep the root key inside the mode-0700 directory and never install it into the system trust store. Production tests should use the operating system’s existing CA path or the explicit private CA bundle approved for that service.

Read EHLO Before You Upgrade

Before TLS, the receipt proves only the banner and capability of the listener currently owning the address and port. It does not authenticate the peer, so do not send credentials or message data during this stage.

set -Eeuo pipefail
LAB=/tmp/voxfor-smtp-starttls-140
[[ "$(<"$LAB/.marker")" == voxfor-smtp-starttls-lab-140 ]]
OUT=$(printf 'EHLO probe.audit.test\r\nQUIT\r\n' | nc -w 2 127.0.0.1 2525)
printf '%s\n' "$OUT"
grep -q '^220 smtp.audit.test ESMTP' <<< "$OUT"
grep -q '^250-STARTTLS' <<< "$OUT"

250-STARTTLS means the server offered an upgrade in this session. It does not prove that a remote policy requires TLS, that the certificate is usable, or that a downstream relay will accept a message. When a queue incident is already in progress, preserve this transport evidence before following the Postfix queue triage workflow; queue state and certificate identity are different ownership layers.

Make Chain and Hostname Checks Fail Closed

Four often-implicit inputs become explicit in the positive command: protocol upgrade, network endpoint, TLS SNI name, and verification hostname. The lab uses its private CA file. On a typical public Linux host, replace that with the distribution trust path, commonly -CApath /etc/ssl/certs.

set -Eeuo pipefail
LAB=/tmp/voxfor-smtp-starttls-140
[[ "$(<"$LAB/.marker")" == voxfor-smtp-starttls-lab-140 ]]
OUT=$(printf 'EHLO probe.audit.test\r\nQUIT\r\n' | openssl s_client \
  -starttls smtp -connect 127.0.0.1:2525 -servername smtp.audit.test \
  -verify_hostname smtp.audit.test -verify_return_error -CAfile "$LAB/root.pem" \
  -brief -ign_eof 2>&1)
printf '%s\n' "$OUT"
grep -q 'Verification: OK' <<< "$OUT"
grep -q 'Verified peername: smtp.audit.test' <<< "$OUT"
grep -q '^250-SIZE 10485760' <<< "$OUT"

SNI and hostname verification solve separate problems

-servername smtp.audit.test sends SNI so a TLS terminator can choose a virtual certificate. -verify_hostname smtp.audit.test asks OpenSSL to reject a certificate that does not cover that name. Supplying one flag does not imply the other. -verify_return_error then converts a verification defect into a nonzero command result instead of a warning buried above a continuing session.

This distinction is especially useful after renewal: the certificate file may be correct while a proxy, MTA process, IPv6 listener, or secondary node still serves an older object. Postfix’s TLS documentation also emphasizes correct certificate-chain construction and server-side TLS logging, but client-side endpoint verification is what proves the listener presented the expected identity.

Inspect the Certificate the Listener Actually Serves

A successful automated assertion should still produce a human-readable receipt for scope and expiry. Pipe the certificate presented by the live listener into openssl x509; do not inspect only a local PEM path and assume the network service loaded it.

set -Eeuo pipefail
LAB=/tmp/voxfor-smtp-starttls-140
[[ "$(<"$LAB/.marker")" == voxfor-smtp-starttls-lab-140 ]]
FIELDS=$(printf 'QUIT\r\n' | openssl s_client \
  -starttls smtp -connect 127.0.0.1:2525 -servername smtp.audit.test \
  -verify_hostname smtp.audit.test -verify_return_error -CAfile "$LAB/root.pem" \
  -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer -dates -ext subjectAltName)
printf '%s\n' "$FIELDS"
grep -q 'subject=CN=smtp.audit.test' <<< "$FIELDS"
grep -q 'DNS:smtp.audit.test' <<< "$FIELDS"

Record the issuer, SAN, notBefore, and notAfter values in UTC. OpenSSL’s hostname check can fall back to the subject Common Name when a certificate has no supported SAN, while current RFC 9525 service-identity guidance deprecates CN-ID use. This audit therefore does not accept the positive -verify_hostname line alone: the separate field assertion must also find the expected dNSName in SAN. Automate an expiry threshold appropriate to the renewal process, but do not confuse remaining days with trust or hostname coverage.

Run Negative Controls Before the Cutover

A positive-only probe can be accidentally permissive. Two negative controls prove that the command actually enforces the assertions the receipt claims.

First, keep the connection and SNI name correct but ask OpenSSL to verify a hostname the certificate does not cover. Exit status must be nonzero, and the output must identify a hostname mismatch.

set -Eeuo pipefail
LAB=/tmp/voxfor-smtp-starttls-140
[[ "$(<"$LAB/.marker")" == voxfor-smtp-starttls-lab-140 ]]
set +e
OUT=$(printf 'QUIT\r\n' | openssl s_client \
  -starttls smtp -connect 127.0.0.1:2525 -servername smtp.audit.test \
  -verify_hostname wrong.audit.test -verify_return_error -CAfile "$LAB/root.pem" \
  -brief 2>&1)
STATUS=$?
set -e
printf '%s\nexit=%s\n' "$OUT" "$STATUS"
[[ "$STATUS" -ne 0 ]]
grep -qi 'hostname mismatch' <<< "$OUT"

Second, keep the expected hostname correct but validate against an unrelated root. This separates name coverage from chain trust. The unrelated root is created only inside the owned lab directory.

set -Eeuo pipefail
LAB=/tmp/voxfor-smtp-starttls-140
[[ "$(<"$LAB/.marker")" == voxfor-smtp-starttls-lab-140 ]]
openssl req -x509 -newkey rsa:2048 -nodes -days 2 \
  -subj '/CN=Unrelated SMTP Lab Root' \
  -keyout "$LAB/unrelated.key" -out "$LAB/unrelated.pem" >/dev/null 2>&1
set +e
OUT=$(printf 'QUIT\r\n' | openssl s_client \
  -starttls smtp -connect 127.0.0.1:2525 -servername smtp.audit.test \
  -verify_hostname smtp.audit.test -verify_return_error -CAfile "$LAB/unrelated.pem" \
  -brief 2>&1)
STATUS=$?
set -e
printf '%s\nexit=%s\n' "$OUT" "$STATUS"
[[ "$STATUS" -ne 0 ]]
grep -qi 'unable to get local issuer certificate' <<< "$OUT"

If either negative control exits zero, the audit command is not fail-closed and the positive result is not acceptable. Do not weaken production verification to make the test green. Repair the served chain, the client reference hostname, SNI routing, or the selected trust bundle according to the failed control.

Turn the Probe Into an Endpoint Receipt

Run the positive sequence against every endpoint from the inventory and retain one row per address family and vantage point. This matrix keeps a passing primary MX from hiding a failing secondary or submission listener.

Here is the representative receipt from the disposable run:

environment: Debian 13.6; OpenSSL 3.5.6; Node.js 24.18.0; 127.0.0.1:2525 only
plaintext: banner=smtp.audit.test; STARTTLS advertised
positive: TLSv1.3; TLS_AES_256_GCM_SHA384; Verification=OK
identity: Verified peername=smtp.audit.test; SAN=DNS:smtp.audit.test
application: post-TLS EHLO returned SIZE and HELP
hostname control: verify error 62 hostname mismatch; exit=1
chain control: verify error 20 unable to get local issuer certificate; exit=1
verdict: upgrade, trust, identity, and post-TLS SMTP all proved fail closed
cleanup boundary: verified Node process identity plus exact marker-owned directory

Accept an endpoint only when the intended listener advertises or directly provides the correct TLS mode; the fail-closed positive command returns zero; chain validation reports Verification: OK; the verified peer name equals the documented reference hostname; the expected dNSName is explicitly present in SAN; issuer and validity are recorded from the served leaf; a fresh post-TLS EHLO succeeds; both negative controls return nonzero for the expected reason; and every relevant MX, submission, IPv4, IPv6, proxy, and secondary path produces a consistent receipt from a representative network.

These checks prove endpoint identity and SMTP continuity, not successful delivery or inbox placement. If a delivered message later fails body integrity, use the DKIM body-hash diagnosis. If forwarding changes the authentication path, evaluate SPF, SRS, and ARC ownership separately. A healthy TLS receipt should narrow the incident, not absorb unrelated mail controls such as authentication, DKIM, DMARC, and recipient acceptance.

When certificate deployment, MTA configuration, DNS/PTR, and renewal monitoring belong to another team, compare its written duties with the managed mail-server scope before handoff; an undefined owner is itself a failed cutover condition.

Remove the Fixture and Preserve the Decision

Cleanup first proves that the recorded PID still belongs to the exact Node executable, script path, and lab argument created by the fixture. Only then does it send a signal, wait for the loopback listener to disappear, and delete only that exact directory. A dead fixture needs no signal; a reused PID with different command-line identity makes cleanup fail closed.

set -Eeuo pipefail
LAB=/tmp/voxfor-smtp-starttls-140
[[ -f "$LAB/.marker" ]]
[[ "$(<"$LAB/.marker")" == voxfor-smtp-starttls-lab-140 ]]
PID=$(<"$LAB/server.pid")
if [[ -d "/proc/$PID" ]]; then
  mapfile -d '' -t ARGV < "/proc/$PID/cmdline"
  [[ "$(readlink -f "/proc/$PID/exe")" == "$(readlink -f "$(command -v node)")" ]]
  [[ "${#ARGV[@]}" -eq 3 ]]
  [[ "${ARGV[1]}" == "$LAB/server.mjs" ]]
  [[ "${ARGV[2]}" == "$LAB" ]]
  kill "$PID"
  for _ in {1..30}; do [[ ! -d "/proc/$PID" ]] && break; sleep 0.1; done
fi
[[ ! -d "/proc/$PID" ]]
! ss -Hln 'sport = :2525' | grep -q .
rm -rf --one-file-system "$LAB"
[[ ! -e "$LAB" ]]
printf 'cleanup=owned_pid_stopped_marker_scope_absent\n'

If a reviewed production certificate, listener, SNI map, or trust-bundle change causes acceptance to regress, restore only the previous backed-up object for the affected endpoint, reload the exact owning service with its normal validation command, and repeat the same positive and negative receipts. Do not disable certificate verification, replace the system trust store broadly, remove unrelated certificates, open relay access, or roll back DNS and mail-authentication records that the evidence did not implicate; retain the failed output and change record for the responsible owner. If fixture PID identity does not match, stop and inspect it instead of signaling that process.