Linux Conntrack Is Full While Existing Sessions Still Work

One SSH session remains responsive, a database connection keeps serving queries, yet every fresh health check times out. That combination can look impossible until the Linux host reports nf_conntrack: table full, dropping packet. Existing tracked flows already own state; a new flow needs a new conntrack entry. When the table has no capacity, Netfilter can drop the packet that would create that entry while established traffic continues through its recorded path.

Treat that kernel message as a capacity incident, not permission to flush the table or paste a larger limit from another server. First prove occupancy and packet drops, preserve a return path, identify who is creating entries, then add only enough controlled headroom to recover. A complete repair ends with a workload test, a memory-aware limit and an alert that fires before allocation fails again.

The failure signature is wider than one error message

Conntrack is Linux kernel state for flows seen by Netfilter. Stateful firewall rules, source or destination NAT, container publishing and many Kubernetes service paths depend on it. The kernel exposes the current allocation as nf_conntrack_count and the ceiling as nf_conntrack_max; current kernel documentation defines both values and documents protocol-specific state timeouts. Red Hat’s incident summary confirms that the table-full message applies across Netfilter-based iptables, firewalld and nftables hosts.

A full table often produces an uneven outage. Reused keep-alive connections may survive. A fresh TCP handshake, new DNS request or short-lived API call may fail. Monitoring from inside the same persistent session can therefore stay green while new clients cannot connect. A live shell proves only that one path still has state, not that the host can admit another flow.

Other failures can create the same external symptom. A process may not be listening, a socket backlog may be full, policy may reject the port, an upstream load balancer may be unhealthy, or the provider path may be dropping traffic. Conntrack becomes the leading diagnosis only when table occupancy or Netfilter failure counters agree with the timing.

Capture capacity before touching firewall state

Before any firewall change, secure console or provider-panel access. If the only administrative path is the same SSH connection that still works, open an out-of-band console and keep the current shell alive. Do not restart networking, firewall services, container runtimes or the host merely to see whether the symptom clears; that destroys the most useful time correlation and may remove the surviving return path.

Read count, ceiling and kernel evidence together

A timestamped baseline keeps the later relief action measurable. The following commands are read-only; conntrack -S requires the conntrack package and sufficient privileges.

date -u
uname -r
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_buckets
sudo journalctl -k --since '-30 min' --no-pager | grep -Ei 'nf_conntrack|table full|dropping packet'
sudo conntrack -S

Occupancy is nf_conntrack_count / nf_conntrack_max, but one instantaneous ratio is not the verdict. A count of 98% with rising insert_failed, drop or early_drop counters is much stronger evidence than a quiet 98% snapshot. Record nf_conntrack_buckets at the same time: current kernel documentation notes that entries are placed in the hash twice, once for each direction, and the default maximum equals the bucket count. Raising only the maximum can deepen hash chains and increase lookup work even when it restores allocation headroom. Per-CPU output from conntrack -S can also show failures concentrated on a busy path when the aggregate view is less obvious.

Counter names and availability vary with kernel configuration. When /proc/net/stat/nf_conntrack is absent on a newer build, current and maximum entries can still be available through the sysctl interface. Record what the host actually exposes instead of treating a missing legacy file as proof that conntrack is inactive.

Prove a new flow from outside

From another network, open a fresh TCP or HTTPS connection while watching kernel counters. Reusing the browser tab or shell that survived the incident is not a new admission test.

curl --connect-timeout 5 --max-time 10 -fsS https://example.com/health
nc -vz -w 5 203.0.113.10 443

Documentation addresses must be replaced with the real service endpoint. A timeout aligned with rising conntrack drops supports the diagnosis. Connection refused instead points toward listener or reject behavior, while a successful TCP connect followed by HTTP 500 moves attention to the application.

Prove which traffic is consuming the table

Raising the ceiling without finding demand converts a hard outage into a later, larger outage. Entry growth can be legitimate—more clients, more NAT traffic, short-lived microservice calls—or hostile, such as scanning and floods. A retry loop inside the host can be just as damaging as an external source.

Sample states without dumping everything

The Netfilter conntrack manual supports filtered listing and live events. Start with statistics and short, scoped samples. Avoid an unfiltered conntrack -L on a table already holding hundreds of thousands of entries; serializing the entire state can add pressure during the incident.

sudo conntrack -L -p tcp --state SYN_SENT -o extended | sed -n '1,40p'
sudo conntrack -L -p tcp --state TIME_WAIT -o extended | sed -n '1,40p'
sudo timeout 5 conntrack -E -e NEW,DESTROY -o timestamp,extended

Samples answer different questions. Many repeated new tuples toward one destination can identify a failing dependency or retry storm. Large NAT fan-out from one internal source points toward a workload owner. Persistent UDP entries may reflect expected streaming behavior rather than TCP leakage. State name alone is not guilt; correlate addresses, ports, rate and workload logs.

Network namespaces complicate attribution. A container host can create state for published ports and forwarded traffic even when the application process lives elsewhere. On Kubernetes nodes, kube-proxy has explicit conntrack configuration, so check the cluster’s node policy and component ownership before treating a local sysctl file as the source of truth.

Separate growth from attack traffic

Compare source diversity and request rate with reverse-proxy, load-balancer, firewall and application logs. Organic growth usually follows a workload curve and produces accepted requests. A scan or flood may create many incomplete tuples, broad port fan-out or high new-flow rates with little completed work. Neither pattern is proven by one IP address or one TCP state.

If sustained hostile traffic is creating the demand, compare host evidence with DDoS protection services before treating a larger table as the whole defense. Upstream filtering can prevent unwanted packets from consuming host bandwidth and state; a bigger local ceiling only gives the same traffic more room.

Recover without destroying healthy sessions

Recovery has two simultaneous goals: create admission headroom and stop the source that is consuming it. Choose the smallest reversible action that satisfies both. Record count, max, memory headroom and drop counters before and after every change.

Add temporary headroom only with a memory budget

Increasing nf_conntrack_max can be a valid emergency action when the host has memory capacity and the current limit is genuinely too low for measured demand. It is not free. Conntrack entries live in kernel memory, and their actual cost depends on kernel build, enabled extensions and traffic. Old fixed “bytes per entry” formulas should not replace measurement on the running host.

Use an approved incident value, then verify allocation and memory behavior immediately:

CONNTRACK_APPROVED_LIMIT=262144
sudo sysctl -w net.netfilter.nf_conntrack_max="$CONNTRACK_APPROVED_LIMIT"
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_buckets
grep -E 'MemAvailable|Slab|SReclaimable|SUnreclaim' /proc/meminfo
sudo conntrack -S

262144 is an example, not a universal recommendation. If it is below the current host limit, above the host’s approved memory budget or inconsistent with a cluster-managed setting, do not use it. Compare the new maximum with the recorded bucket count and watch CPU, softirq load, latency and packet-failure counters during the acceptance window. nf_conntrack_buckets is writable only from the initial network namespace, so container or Kubernetes operators must not improvise a second owner from another namespace. Temporary relief passes only when new-flow success returns without dangerous memory pressure or lookup-cost regression.

Stop the producer instead of deleting shared state

Rate-limit or pause the identified retrying client, remove an accidental public exposure, scale the responsible NAT gateway, or apply upstream filtering. Keep scope narrow. Restarting every container or globally reloading the firewall may create another burst of fresh connections exactly when admission is fragile.

A global conntrack -F flush is not routine recovery. It deletes useful state for NAT and stateful firewall decisions, can interrupt healthy sessions and removes evidence about the demand source. Filtered deletion is also an incident change: use it only when one confirmed tuple set is harmful, the NAT/firewall consequence is understood and rollback means allowing affected clients to reconnect.

Prove both old and new traffic after relief

For acceptance, keep one pre-incident session under observation and create new connections from at least two independent locations. Test the real protocol—not only ICMP—and include a write/read action when the application is stateful. A website needs successful TLS and an expected HTTP response; a database needs authenticated query behavior; a NAT gateway needs forwarded return traffic.

sudo watch -n 2 'sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max net.netfilter.nf_conntrack_buckets; conntrack -S 2>/dev/null | grep -E "insert_failed|drop|early_drop"'

Exit watch after the observation window. Success means occupancy is falling or stable below the chosen boundary, failure counters stop increasing, preserved sessions remain valid and repeated new-flow probes pass. One green request is not a recovery trend.

Set a limit your host can defend

Once demand is understood, derive the persistent limit from observed peak entries, expected growth, failure-detection time and tested memory headroom. Short bursty NAT workloads need a different margin from long-lived MQTT, VPN or database connections. Dedicated firewalls and Kubernetes nodes also have different ownership and failover consequences.

Persist one owner and document rollback

Before writing a sysctl file, search for existing maximum or bucket declarations and orchestration:

grep -R --line-number -E 'nf_conntrack_(max|buckets)' /etc/sysctl.conf /etc/sysctl.d 2>/dev/null
systemctl cat systemd-sysctl.service

If configuration management, kube-proxy or an image build owns the value, change that source rather than creating a competing file. For a standalone host, a reviewed file might contain:

# /etc/sysctl.d/60-conntrack-capacity.conf
net.netfilter.nf_conntrack_max = 262144

Load only after peer review with sudo sysctl --system, then confirm the effective maximum and the separately recorded bucket count. A materially larger max-to-bucket ratio needs workload testing for CPU cost and packet latency; changing the hash size itself is a kernel- and ownership-sensitive operation, not an automatic companion command. Rollback is the previous documented limit, not deletion of random files. Do not shorten established TCP, UDP stream or generic timeouts globally just to lower count; kernel defaults reflect protocol state, and aggressive expiry can break valid long-lived traffic. Change a timeout only when flow-lifetime evidence, application behavior and reconnect testing support it.

Alert before allocation fails

Prometheus node_exporter exposes current entries and the table limit on supported Linux kernels. Its maintained conntrack collector also exposes statistics where the kernel provides them. A practical warning watches sustained occupancy rather than a single spike:

max_over_time(node_nf_conntrack_entries[10m])
  / node_nf_conntrack_entries_limit > 0.80

Choose the threshold from growth rate and response time. A fast NAT burst may need an earlier alert than a stable host. Add counter-rate alerts when insert_failed, drop or early_drop metrics exist, and keep an outside probe that creates a genuinely new connection. Monitoring built from private Uptime Kuma checks can provide an independent admission signal without exposing the administration dashboard.

Close the incident with an evidence record

Document the first failed new-flow timestamp, peak count/max ratio, relevant kernel counters, dominant states and tuples, responsible workload or traffic source, temporary relief, persistent owner, memory result and acceptance probes. That record turns the next alert into a comparison instead of another guess.

Conntrack recovery is complete when healthy old flows remain controlled, repeated new flows succeed and the table has defensible headroom under observed demand. Keep the record beside Linux capacity incident guidance so networking, memory, storage and service failures use the same evidence-first discipline.