Packets Can Drop Inside Linux Before the Application Sees Them

An interface can remain UP, the process can keep listening, and Linux can still discard receive packets before they reach the application socket. The decisive evidence is not a lifetime drop total. It is which counter increases during the failing interval: a generic interface RX counter, a driver-specific ring/missed/error counter, the per-CPU backlog-admission fields in /proc/net/softnet_stat, the NAPI budget-pressure field, or only an application metric.

Take two snapshots around the same traffic burst. If softnet dropped rises on one CPU while driver-specific ring, missed, no-buffer, and error counters stay flat, evidence moves from the hardware/driver queue toward CPU-backlog admission. Generic ip -s link RX-dropped cannot make that exclusion because Linux may increment it for the same software backlog rejection. Column two alone still does not say why admission failed. Compare the flow_limit delta: a matching rise identifies RPS flow limiting, while a supported flat flow_limit value makes backlog-capacity overflow the leading inference. If only time_squeeze rises, receive work exceeded a polling budget, but that value does not itself count lost packets.

One interval can place the loss inside Linux

An untouched baseline is essential: changing sysctls, interrupt affinity, queue count, or offloads would contaminate attribution. Identify the interface used for the affected destination and preserve device-level evidence.

ip route get 1.1.1.1
IFACE=eth0
date -u +'%Y-%m-%dT%H:%M:%SZ'
ip -s -s link show dev "$IFACE"
ethtool -S "$IFACE" 2>/dev/null | grep -Ei 'rx.*(drop|miss|no_buffer|error|discard)' || true

The interface from the route output belongs in IFACE; eth0 is only a placeholder. Driver statistic names vary, so keep the complete ethtool -S output in the incident record even when the filtered view is empty. A virtual NIC may expose fewer ring counters than physical hardware.

Softnet interval deltas form the next boundary. Each row of /proc/net/softnet_stat belongs to one CPU and each field is hexadecimal. This read-only Python helper waits ten seconds and prints only changed values for processed packets, backlog-admission drops, budget pressure, received RPS packets, and the separate flow-limit subset.

from pathlib import Path
from time import sleep

def read_softnet():
    rows = {}
    for ordinal, line in enumerate(Path("/proc/net/softnet_stat").read_text().splitlines()):
        fields = [int(value, 16) for value in line.split()]
        cpu = fields[12] if len(fields) > 12 else ordinal
        source = "field13" if len(fields) > 12 else "row_ordinal"
        if cpu in rows:
            raise RuntimeError(f"duplicate CPU key: {cpu}")
        rows[cpu] = (fields, source)
    return rows

before_online = Path("/sys/devices/system/cpu/online").read_text().strip()
before = read_softnet()
sleep(10)
after = read_softnet()
after_online = Path("/sys/devices/system/cpu/online").read_text().strip()

if before_online != after_online or set(before) != set(after):
    raise RuntimeError("online CPU set changed during the sample; discard and retry")

for cpu in sorted(before):
    old, old_source = before[cpu]
    new, new_source = after[cpu]
    if old_source != new_source:
        raise RuntimeError("softnet row layout changed during the sample; discard and retry")
    pick = lambda values, index: values[index] if len(values) > index else 0
    delta = {
        "processed": pick(new, 0) - pick(old, 0),
        "dropped": pick(new, 1) - pick(old, 1),
        "time_squeeze": pick(new, 2) - pick(old, 2),
        "received_rps": pick(new, 9) - pick(old, 9),
    }
    flow_limit = new[10] - old[10] if len(old) > 10 and len(new) > 10 else None
    if any(delta.values()):
        flow_text = str(flow_limit) if flow_limit is not None else "unsupported"
        print(f"cpu={cpu} cpu_source={old_source} " + " ".join(f"{key}={value}" for key, value in delta.items()) + f" flow_limit={flow_text}")

A representative slow or loss interval is required. A quiet ten seconds proves only that the sampled window was quiet. On current layouts, cpu_source=field13 means the exported CPU ID was used; row_ordinal is an older-layout fallback and should be treated as row identity when offline CPU gaps make ordinal-to-CPU mapping uncertain. The helper rejects an online-CPU-set change during the interval. Preserve request rate, packet rate, p95/p99 latency, retransmits, errors, completed work, and the UTC boundaries beside the output.

Decode softnet counters without turning them into a diagnosis

Linux processes receive traffic through driver queues and NAPI polling, then continue through protocol processing toward sockets. /proc/net/softnet_stat reports per-CPU network receive work, not a complete end-to-end packet-loss ledger.

The kernel network sysctl reference defines netdev_max_backlog as the maximum input-side queue when packets arrive faster than the kernel can process them. It defines netdev_budget and netdev_budget_usecs as packet and time limits for a NAPI polling cycle. These are separate boundaries.

Compare column two with the flow-limit field before naming overflow. The kernel scaling guide explains that optional per-CPU RPS Flow Limit can preferentially drop packets from a dominant flow once the backlog passes a threshold. Those packets increase the general dropped total and the separate flow_limit total. When the field is present, dropped rises, flow_limit stays flat, and driver-specific ring/missed/error evidence stays flat, queue-capacity overflow becomes the stronger inference. If the kernel’s exported row lacks the flow-limit field, the sampler prints flow_limit=unsupported; column two alone cannot make that distinction.

Generic interface RX-dropped remains useful corroboration, not a boundary label. Linux can account a CPU-backlog admission rejection in the interface statistic as well as softnet, so ip -s link must not be grouped with driver-specific ethtool -S ring or missed-buffer evidence.

Do not diagnose from an old nonzero hex value. Counters are cumulative and may include a boot-time burst, migration, backup window, or yesterday’s attack. Only an increasing counter during the symptom window belongs in the current causal chain.

Another loss path can sit later. When new connections fail while established flows continue, compare the interval with Linux conntrack saturation recovery instead of assuming the receive backlog is full. Size-specific stalls across an encrypted tunnel belong in WireGuard MTU black-hole diagnosis, especially when small packets succeed and large packets hang.

Find the CPU and RX queue that own the delta

A hot softnet row identifies a CPU, not yet the owner. Compare receive interrupts, available RX queues, steering masks, and softirq activity without changing them.

grep -E "$IFACE|virtio|ena|mlx|ixgbe|i40e" /proc/interrupts
ls -d "/sys/class/net/$IFACE/queues/"rx-* 2>/dev/null
grep -H . "/sys/class/net/$IFACE/queues/"rx-*/rps_cpus 2>/dev/null || true
grep -E 'NET_RX|NET_TX' /proc/softirqs
systemctl is-active irqbalance 2>/dev/null || true

Capture this twice around the same traffic interval. Interrupt totals that climb almost entirely on the same CPU as dropped or time_squeeze are strong ownership evidence. On virtual hardware, the hypervisor and virtual NIC may determine queue exposure, so do not assume that host-side physical queues are visible in the guest.

The kernel scaling guide explains that RSS selects hardware receive queues, while RPS can enqueue packets onto another CPU’s backlog. A zero rps_cpus mask means RPS is disabled for that queue. That can be correct when RSS already distributes flows; blindly enabling every CPU can harm cache locality and create needless IPIs.

Check whether the receive CPU is actually busy rather than trusting one overall CPU graph.

mpstat -P ALL 1 10 2>/dev/null || true
sar -n SOFT 1 10 2>/dev/null || true
cat /proc/net/softnet_stat

sar -n SOFT requires sysstat and may not exist. Keep the raw file and portable sampler as the primary evidence. A single elephant flow can remain on one RSS queue by design, while many flows should usually distribute more broadly when queue and hash configuration permit it.

Hardware topology becomes relevant only after measurement. Dedicated server hardware guidance helps connect NIC queue count, CPU cores, NUMA locality, and sustained throughput without treating hardware purchase as the first troubleshooting step.

Change only the boundary the evidence names

Record every current value before an experiment.

sysctl net.core.netdev_max_backlog
sysctl net.core.netdev_budget
sysctl net.core.netdev_budget_usecs
ethtool -l "$IFACE" 2>/dev/null || true
ethtool -g "$IFACE" 2>/dev/null || true

Do not change backlog, NAPI budgets, ring size, IRQ affinity, and RPS in one maintenance window. That destroys attribution. Record the old value, chosen test value, workload, start/end time, and exact rollback condition. Persist a sysctl only after the temporary test survives a representative peak.

Increasing netdev_max_backlog can provide burst headroom, but Rocky Linux IRQ and packet-drop guidance also requires measuring receive processing and distribution rather than treating one setting as universal. If the CPU cannot drain the queue, a larger backlog merely moves the symptom into latency.

Hostile or volumetric traffic belongs at a different control boundary. Upstream DDoS protection can filter before the guest spends receive-path capacity. For legitimate sustained demand, preserve the measured traffic envelope and queue/CPU evidence for a separate capacity decision rather than sizing from one unexplained spike.

Close the incident at the socket, not the counter

Replay the same request rate, flow mix, packet sizes, and duration. Capture generic interface counters, driver-specific ring/missed/error counters, softnet deltas, interrupts, CPU/softirq time, retransmits, service errors, tail latency, and useful completions inside one UTC window. If possible, compare both server-side receipt and client-side success.

Pass only when the expected application delivery improves and the identified loss boundary stays quiet without unsafe CPU, memory, or latency growth. If softnet drops flatten but client errors continue, restore the previous setting and continue at conntrack, MTU, firewall, socket, application, or upstream-network boundaries. If time_squeeze falls but useful completion does not rise, the budget was observable but not causal.

Keep the incident receipt: interface, queue count, CPU map, before/after counters, traffic envelope, changed control, old value, new value, rollback trigger, and application result. Continue through the Linux operations library for adjacent system evidence, but preserve this receive-path fingerprint as its own monitoring rule: alert on interval deltas and user impact, not cumulative totals alone.