Linux Audit Lost Events Mean the Evidence Is Already Incomplete

When auditctl -s shows a larger lost value than the previous sample, the Linux Audit trail already has a gap. Those discarded records cannot be recovered by emptying the queue, restarting auditd, or increasing -b. The safe response is to preserve the loss boundary, find why records stopped draining, repair that owner, and prove that new events reach durable storage.

That distinction matters on a VPS, dedicated host, or security-sensitive Linux machine. Audit data may support incident response, privileged-change review, or compliance evidence. A quiet console after the burst is not proof that the trail became complete again.

Read lost and backlog as different receipts

The current auditctl manual defines backlog as records queued in the kernel waiting for auditd. lost counts records discarded because that queue overflowed. One value describes work waiting now; the other records evidence already dropped.

Capture status without resetting either counter:

sudo auditctl -s
sudo journalctl -k --since "15 minutes ago" --no-pager | grep -E 'audit(_backlog|_lost)|backlog limit exceeded'

An empty backlog with lost=47 means the consumer caught up after 47 records were discarded. Conversely, a high backlog with an unchanged lost counter means pressure exists, but the kernel has not yet reported additional loss. Use deltas across timed samples, not one isolated number.

Record the UTC time, hostname, kernel release, audit package version, and complete status output in the incident file. Do not run auditctl --reset-lost during diagnosis; resetting the counter before preserving its terminal value erases the easiest system-level boundary for the gap.

Trace the path instead of tuning one queue

Linux Audit has more than one place where work can accumulate. Auditable activity creates records, the kernel backlog holds them until auditd reads them, auditd writes the local trail, and optional plugins receive copies through a separate dispatcher queue.

Kernel status identifies the first visible boundary

auditctl -s reports the registered daemon PID, current backlog, limit, rate limit, loss count, and wait-time fields. A pid of 0 means no audit daemon is registered with the kernel. A nonzero PID proves registration, not healthy disk writes or plugin delivery.

The backlog wait setting adds an availability tradeoff. When the queue is full, the kernel can delay processes that generate auditable events while it waits for room. Upstream plugin documentation warns that this can become visible as general application slowdown. Setting the wait to zero removes that delay but can make the kernel discard at capacity sooner.

auditd has its own sinks and queues

Local storage can block on a full, read-only, frozen, or failing filesystem. Red Hat documents a case where a frozen filesystem stopped auditd writes, the kernel backlog overflowed, and backlog limit exceeded was only the downstream symptom. Follow the same fail-closed storage ownership used in ext4 read-only recovery before treating the warning as queue sizing.

Plugins add another boundary. The q_depth and overflow_action settings in auditd.conf control auditd’s dispatcher queue. A slow child process can fill that queue; auditd then stops emptying the kernel backlog quickly enough. Kernel lost=0 also does not prove every plugin received every record, because dispatcher overflow is a separate loss surface.

Freeze three samples around the same workload

One snapshot cannot distinguish a recovered burst from an active drain failure. Capture three samples at a fixed interval while workload conditions remain comparable. The commands below read status and request auditd’s state report; they do not clear counters or reload policy.

incident_dir="/root/audit-incident-$(date -u +%Y%m%dT%H%M%SZ)"
sudo install -d -m 0700 "$incident_dir"
for sample in 1 2 3; do
  { date -u --iso-8601=seconds; sudo auditctl -s; } | sudo tee "$incident_dir/kernel-$sample.txt" >/dev/null
  sudo auditctl --signal state
  sleep 1
  sudo cp --preserve=mode,timestamps /run/audit/auditd.state "$incident_dir/auditd-$sample.state" 2>/dev/null || true
  sleep 19
done

Now correlate daemon, storage, and kernel evidence from the same window:

sudo systemctl status auditd --no-pager
sudo journalctl -u auditd --since "10 minutes ago" --no-pager
sudo journalctl -k --since "10 minutes ago" --no-pager
sudo findmnt -T /var/log/audit/audit.log -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo df -hT /var/log/audit
sudo df -i /var/log/audit

Compare lost, backlog, and backlog_wait_time_actual across the three kernel files. In the state files, inspect auditd queue depth, peak depth, plugin information, and write progress available on that package version. Missing fields are a compatibility observation, not permission to invent zeroes.

Find what stopped the drain

No registered daemon

If pid=0, preserve systemctl status, the auditd journal, exit status, unit limits, and memory evidence before taking a lifecycle action. Process disappearance may belong to kernel versus systemd-oomd evidence, while repeated activation can later be masked by systemd restart-limit evidence. Fix the first failure rather than clearing a later counter.

Avoid assuming that every distribution supports or recommends the same systemctl restart auditd workflow. Red Hat-family systems intentionally preserve service semantics that generate audit records for administrator actions. Use the distribution’s documented service operation after the evidence is saved.

Storage or rotation cannot keep up

Disk bytes are only one check. Inspect inodes, mount flags, kernel storage errors, log rotation activity, space_left_action, admin_space_left_action, disk_full_action, and disk_error_action. A large backlog is expected if auditd is suspended or waiting on a blocked write path.

Queue headroom can absorb a short rotation pause, but it cannot repair a frozen filesystem or a sink left suspended. Restore the sink first, then decide whether measured peak depth justifies more buffer capacity.

A plugin is holding the dispatcher

List active files under /etc/audit/plugins.d/, identify each executable and owner, and compare its queue or error logs with /run/audit/auditd.state. Current upstream guidance says plugins should dequeue quickly and perform heavier work behind their own internal queues.

Temporarily disabling a security feed may create another evidence gap. Any containment decision needs an owner, timestamp, alternate collection path, and explicit re-enable test. Increasing q_depth buys burst time only when the plugin eventually catches up.

Rules generate more than the host can process

Use keys to locate event-producing rule families before editing coverage:

sudo auditctl -l > /root/audit-rules-before.txt
sudo aureport --start today --key --summary -i
sudo ausearch --start recent -k suspicious-key --raw | sudo aureport --event --summary

The audit rules manual recommends combining related syscalls in one rule and declaring the architecture before syscall names. Missing architecture filters can make all system calls cross a rule and reduce performance. Deprecated -w watches also deserve migration to syscall-based path or directory rules, but required security coverage must not be deleted merely to silence loss.

During login floods, separate event production from admission control. OpenSSH pre-authentication limits can explain connection drops and source concentration, while the Audit incident still requires its own queue and sink proof.

Change the owner, then size the burst boundary

Choose one change that addresses the measured constraint:

• repair writable storage, inode reserve, or a frozen mount when auditd cannot write;
• restore a missing daemon only after its exit owner is known;
• fix, isolate, or right-size a slow plugin queue;
• narrow an accidentally broad rule while preserving its security purpose;
• increase kernel or dispatcher headroom only from measured peak depth and drain time.

Before considering a rule load, preserve the current list and check whether the source files would change the generated rules:

sudo auditctl -l > /root/audit-rules-before-change.txt
sudo augenrules --check
sudo auditctl -s

If status shows enabled 2, the audit configuration is immutable. Live rule changes are rejected until reboot, so use a controlled maintenance plan rather than trying repeated reload commands. For mutable systems, load only through the distribution’s supported audit rule workflow and immediately compare the effective auditctl -l output with the intended file.

Treat -b, --backlog_wait_time, -r, and -f as a policy set. A larger backlog consumes kernel memory and extends burst tolerance. A nonzero rate limit can itself trigger the configured failure action. Failure mode 2 can panic the host on critical Audit failures; changing it is a security and availability decision, not an incident shortcut.

Prove the trail from a new marker to durable storage

After the queue is draining and the sink is healthy, record a fresh status baseline and emit one controlled marker:

marker="voxfor-audit-acceptance-$(date -u +%Y%m%dT%H%M%SZ)"
sudo auditctl -s
sudo auditctl -m "$marker"
sleep 2
sudo ausearch -m USER --start recent -i | grep -F "$marker"
sudo auditctl -s

The acceptance window passes only when the marker is found in the intended durable trail, lost does not increase, backlog returns toward its normal range, auditd and required plugins remain healthy, and representative workload latency is acceptable. Send separate signals for lost-counter growth, sustained backlog pressure, and missing collection data; Grafana missing-data and error policies provide a useful model for keeping those states distinct.

Close the incident with two explicit statements: records before the repaired boundary may be incomplete, and controlled records after the acceptance marker were observed end to end. That wording protects both operational truth and the value of the evidence that remains.