A Full Linux Accept Queue Is Not Always a SYN Flood

TcpExtListenOverflows rose by 4,812 during a five-minute traffic burst. The kernel log mentioned a possible SYN flood, yet the packet capture showed ordinary clients completing handshakes. The counter proves that a listening socket could not admit another connection; it does not identify the cause by itself.

Linux keeps incomplete handshakes and completed-but-unaccepted connections behind different limits. A busy application can fill the accept queue because its workers stop calling accept() fast enough. Hostile clients can instead hold pressure in the incomplete queue. Packet loss may even happen earlier in the receive path, so listener counters must be compared with host and network evidence rather than treated as the whole path.

Treat this as an admission incident, not a reason to paste a large somaxconn value into sysctl.conf. Capture a short counter delta, find the affected listener, decide which queue owns the pressure, and change the smallest responsible contract.

Start with a delta, not the warning label

Lifetime totals in /proc/net/netstat survive long after the incident. Take two samples around the same workload window and compare them with connection errors and application latency. nstat from iproute2 exposes the named counters without a fragile column-number parser:

nstat -az \
  TcpExtListenOverflows \
  TcpExtListenDrops \
  TcpExtSyncookiesSent \
  TcpExtTCPReqQFullDoCookies \
  TcpExtTCPReqQFullDrop

According to the current Linux SNMP counter documentation, an accept-queue-full event increments both TcpExtListenOverflows and TcpExtListenDrops. ListenDrops can also rise without ListenOverflows, including when allocation fails. Therefore, unequal values are expected and neither counter should be relabeled as “attacks blocked.”

If ListenDrops rises alone while the host is under memory pressure, preserve allocation and cgroup evidence before restarting the service. Voxfor’s Linux OOM ownership trace provides the separate method for deciding whether the kernel or systemd-oomd acted; listener counters cannot answer that question.

Log text is version-dependent. Red Hat’s listener-overflow guidance notes that RHEL 8 and later do not necessarily emit the older “Possible SYN flooding” message when the accept backlog is full. Build alerts on counters, socket occupancy, and application symptoms rather than one kernel string.

Pair the delta with the listener surface:

ss -lntp
ss -lntp 'sport = :443'

For a LISTEN socket, Recv-Q shows connections waiting to be accepted and Send-Q shows the listener backlog surface; the netstat manual documents the same queue semantics. A momentary zero does not clear the incident because the burst may have ended between samples. Repeated sampling or socket-level telemetry must overlap the failure window.

One listener has two admission queues

Linux changed the meaning of listen(backlog) in kernel 2.2. The listen(2) manual states that the argument now describes completely established sockets waiting for the application to accept them. Incomplete connection requests have a separate control.

The incomplete queue waits for the final ACK

After receiving SYN, Linux replies with SYN-ACK and remembers the request while waiting for the client’s ACK. net.ipv4.tcp_max_syn_backlog controls remembered incomplete requests when normal state is used. The current kernel IP sysctl guide describes SYN cookies as a fallback when that queue overflows, not a mechanism for supporting ordinary legal connection rates.

Many SYN-RECV sockets, repeated SYN-ACKs without final ACKs, concentrated unexpected sources, or rising cookie counters can support an incomplete-queue diagnosis. None is conclusive alone: lossy networks, overloaded clients, asymmetric paths, and legitimate connection bursts can produce part of the same picture.

The accept queue waits for application progress

Once the handshake completes, the socket waits for the server process to call accept() or accept4(). Its effective ceiling is constrained by the application’s listen() backlog and net.core.somaxconn; Linux silently caps an application request above somaxconn.

An application pause, exhausted worker pool, stop-the-world runtime event, file-descriptor pressure, accept mutex behavior, or scheduler starvation can let this queue fill even when every client is legitimate. Low host CPU utilization does not exclude container CPU throttling that delays the owning process. A reverse proxy may also be healthy at the process level while its downstream handoff stalls; use HAProxy probe-versus-service evidence when the listener is only one hop in that chain.

Attribute the pressure to a socket and process

Global counters do not name the port. During the incident, capture every listener’s queue occupancy and owning process at a short interval. Do not rely on a single screenshot:

for i in $(seq 1 12); do
  date -Ins
  ss -H -lntp
  sleep 5
done

Record local address, port, Recv-Q, Send-Q, PID, executable, container or network namespace, and deployment revision. If a container owns the socket, repeat the inspection in its network namespace; host and container views can attribute different listeners.

When safe in a controlled window, a bounded syscall trace can answer whether the process is draining the queue:

timeout 15s strace -ff -ttT \
  -e trace=accept,accept4 -p 12345

Tracing adds overhead and may expose process timing, so use it briefly and never as permanent monitoring. Application metrics are preferable when they already report accept rate, event-loop lag, active workers, rejected connections, and file-descriptor use.

Correlate downstream symptoms by timestamp. Clients may time out before the application accepts them; later, the proxy may record a client-aborted request. nginx 499 timing attribution helps separate that downstream observation from the earlier listener admission failure.

Decide whether traffic or drain rate changed

The useful question is not “Was there a SYN?” Every TCP connection starts with one. Ask whether arrival shape, handshake completion, and application drain changed together.

A full conntrack table can drop new flows while established sessions continue, producing a symptom that resembles listener saturation from the client side. Likewise, OpenSSH MaxStartups is an sshd pre-authentication admission policy, not the kernel accept backlog. Keep these boundaries separate before tuning.

Capture packet evidence only for the affected port and retention window. For example, a short SYN sample can show source spread and whether final ACKs return, but a packet capture contains client addresses and must follow incident-data controls.

When captures prove a distributed hostile SYN pattern and traffic saturates the path before the host, host sysctls cannot recover that upstream bottleneck; review whether upstream DDoS filtering can stop the traffic before it reaches the origin. If valid clients complete handshakes and the accept queue fills, stay with the application path instead of calling the traffic malicious.

Run two bounded experiments before a permanent tune

Capacity should absorb a measured burst while the application catches up. It should not hide a process that stopped draining indefinitely.

Restore accept progress before granting more wait time

Resolve blocked accept loops, exhausted workers, descriptor ceilings, or long pauses first. A larger queue can reduce drops during short bursts, but it also lets more clients wait and can increase tail latency. Queue depth is borrowed time, not throughput.

For nginx, the official HTTP core listen documentation exposes backlog=number. After adding a value such as listen 443 ssl backlog=2048; to the correct existing server block, validate and inspect the rendered configuration before reload:

nginx -t
nginx -T 2>/dev/null | grep -nE 'listen .*backlog='

Reload through the normal service manager only after validation, then confirm the new Send-Q. Other runtimes may set backlog in application code, a systemd socket unit, a framework option, or a hard-coded default. Changing only the kernel ceiling does nothing when the application still asks for a smaller value.

Match the application request to the kernel ceiling

Read the current controls before editing:

sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog
sysctl net.ipv4.tcp_syncookies

Choose a value from observed burst rate, acceptable queueing delay, memory budget, and application drain rate. Apply one temporary change, restart or reload the listener if its backlog is set at listen() time, and confirm the effective queue. Persist the value only after the acceptance test.

Avoid enabling tcp_abort_on_overflow as a generic fix. It changes how Linux responds when the accept queue is full and can convert waiting/retry behavior into immediate client failures. The safer correction is to restore drain capacity and size both application and kernel contracts deliberately.

Keep three numbers in the incident record

Close the incident with evidence that another operator can reproduce: counter deltas, affected listener, observed Recv-Q and Send-Q, SYN completion pattern, accept rate, application backlog, kernel caps, source distribution, and the exact change window. Include rollback thresholds for latency, memory, error rate, and queue depth.

Replay a representative burst from an approved test source. Acceptance requires ListenOverflows to stop increasing, the queue to drain after the burst, successful connections to stay within their latency budget, and the application to retain headroom. A quiet counter without a client success check is incomplete proof.

The decisive statement should name one owner: hostile traffic exceeded the perimeter, incomplete handshakes exceeded the SYN-state budget, or completed connections waited longer than the application could accept them. Only then does a backlog change become an engineering decision instead of a guess.