Disk Space Is Missing: Find Deleted Files Still Held Open

df says a Linux filesystem is nearly full, yet du cannot find enough visible files to explain the used space. That contradiction often means a process still has a deleted file open. Removing the pathname changed what directory tools can see; it did not release the file’s allocated blocks while an open descriptor still refers to them.

The safe recovery is not a blind reboot or another deletion spree. Confirm that both commands measured the same mounted filesystem, find unlinked descriptors with lsof +L1, identify the owning service, and make that service close or reopen the file through its supported lifecycle. Space is recovered only after the final reference disappears.

A pathname can disappear before its blocks do

Linux separates a directory entry from an open file description. The Linux unlink(2) manual states that removing the last name does not delete the file’s contents while a process still has it open. Once the last open reference closes, the kernel can release the file.

That distinction explains the apparent disagreement. du walks reachable directory entries and totals their blocks. df reports allocation for the whole filesystem. An unlinked file has no pathname for du to visit, but its blocks remain allocated and therefore visible to df.

Several other conditions can also create a gap: mount points hidden by another mount, filesystem-reserved space, snapshots outside the visible tree, sparse-file accounting choices, permission errors during a du walk, or different block-size units. Treat an open deleted file as a hypothesis until the descriptor evidence proves it.

First prove both numbers describe the same filesystem

Start with the affected path rather than /. A server can have separate filesystems for /, /var, container data, databases, or network mounts, and a broad du command can cross boundaries that df reports separately.

target=/var/log
findmnt -T "$target" -o TARGET,SOURCE,FSTYPE,OPTIONS
df -hT "$target"
df -i "$target"
sudo du -xsh "$target"

findmnt -T names the mount that contains the path. df -hT measures allocated and available blocks on that mount; df -i checks inode capacity. du -x stays on one filesystem, which prevents a nested mount from distorting the comparison.

If df -i is full while byte capacity remains, deleted-open data may not be the incident owner. File-heavy workloads can exhaust inode count first; Maildir inode diagnosis shows how that failure differs from missing blocks. Likewise, Kubernetes operators should use DiskPressure filesystem attribution before assuming the node’s root path owns container storage.

Capture the numbers in bytes when the gap matters operationally:

df -B1 --output=source,size,used,avail,pcent,target "$target"
sudo du -x -B1 -s "$target"

Some difference is normal because the tools ask different questions and filesystem metadata or reserved blocks are not directory content. A multi-gigabyte gap that appeared after log deletion, rotation, or an application upgrade is a stronger signal than a small stable difference.

Ask the kernel which deleted objects remain open

The lsof(8) manual documents +L1 as selection for open files whose link count is below one. Run it with privilege because an unprivileged user cannot inspect every process:

sudo lsof -nP +L1

Relevant rows usually show REG, a link count of 0, a nonzero SIZE/OFF, and a name ending in (deleted). Record COMMAND, PID, USER, FD, DEVICE, SIZE/OFF, NODE, and NAME. Do not simply sum every displayed size: a process can expose the same open file through more than one descriptor, and shared references can duplicate rows.

Map the device back to the affected mount and inspect one candidate without reading its contents:

pid=REPLACE_WITH_PID
fd=REPLACE_WITH_FD_NUMBER

ps -o pid,ppid,user,lstart,etime,cmd -p "$pid"
sudo readlink "/proc/$pid/fd/$fd"
sudo stat -Lc 'device=%D inode=%i links=%h size=%s blocks=%b block_unit=%B' "/proc/$pid/fd/$fd"
sudo cat "/proc/$pid/fdinfo/$fd"

The Linux kernel’s /proc documentation describes per-process descriptor directories and fdinfo. These paths are live kernel views, so the process can exit or replace a descriptor between commands. Recheck PID start time and descriptor target immediately before any action; PID and FD numbers are not durable identifiers.

Allocated blocks, not logical file size alone, determine the expected return. Multiply blocks by block_unit from stat for that descriptor, then account for duplicate inode/device pairs. If the holder continues writing, both logical and allocated size may change while you investigate.

Attribute the descriptor to its real service

A process name is only the first clue. A worker may be supervised by systemd, a container runtime, a database cluster manager, or an application-specific master process. Stopping a child can cause immediate respawn while the parent or another worker keeps the same file open.

systemctl status REPLACE_WITH_SERVICE --no-pager
systemctl show REPLACE_WITH_SERVICE -p MainPID -p ExecMainStartTimestamp -p ActiveState -p SubState
cat "/proc/$pid/cgroup"

Compare MainPID with the lsof row and read the unit’s own operational documentation. For containers, identify the container and logging driver before touching the host path. Container log owners should pair incident recovery with Docker log rotation controls so the next rotation uses a bounded, supported lifecycle.

Database files need stronger caution. A deleted temporary or transaction-related file may be intentionally held by the engine; restarting merely to return space can create a longer outage or recovery cycle. Database-generated growth may instead follow PostgreSQL WAL retention by replication slots, which is a visible retention problem and needs a different owner and repair.

Release the reference through the least disruptive lifecycle

Choose the action according to what created the file and how the application reopens it:

Application-supported reopen is normally narrower than a restart. Web servers and logging daemons often react to a documented signal or reload, but signals are application-specific; do not copy HUP from one daemon to another. Confirm the installed service’s manual and use its service manager so supervision, dependencies, and audit logs remain intact.

When a restart is required, record current health, active connections, replication or queue state, rollback authority, and the expected recovery time. Keep console access if the full filesystem could interfere with SSH, package operations, or service startup. Free a small, known-safe reserve only when necessary to let the controlled action complete.

Avoid truncating /proc/PID/fd/FD as a generic shortcut. It mutates an object the application still owns, can violate application invariants, and may leave the write offset beyond the new end. Killing the PID with -9 is also not a diagnostic method; it skips orderly shutdown and can turn a space incident into data recovery.

When lsof +L1 finds nothing

Absence of unlinked descriptors narrows the case rather than proving df is wrong. Confirm that lsof ran with sufficient privilege and while the discrepancy still existed. A short-lived process may close the file between measurements.

Next, test other filesystem explanations:

• inspect findmnt for a path hidden beneath a mounted filesystem;
• compare filesystem snapshots or subvolumes using the owning storage tool;
• check filesystem-reserved capacity and metadata with filesystem-specific utilities;
• review du permission errors instead of discarding stderr;
• inspect open-but-deleted files inside the correct mount or container namespace;
• verify that df and du used the same block size and target.

Do not respond by deleting random files below container, package, or database data roots. CI builders need builder-specific cache attribution and pruning because Docker metadata owns those objects. Kernel I/O errors or a forced read-only remount point toward ext4 read-only evidence and offline repair, not an open-file cleanup.

Close with a space-recovery receipt

Repeat the same commands after the service action rather than trusting a success message:

sudo lsof -nP +L1
df -B1 --output=source,size,used,avail,pcent,target "$target"
sudo du -x -B1 -s "$target"
systemctl is-active REPLACE_WITH_SERVICE

Save the before/after mount, device and inode, allocated-block estimate, owning PID and service, exact release action, returned bytes, health result, and observation timestamp. Also verify that new writes land on the intended current file instead of another unlinked object.

The durable fix belongs at the lifecycle boundary that failed: log rotation must tell the writer to reopen, supervisors must manage the correct process, and capacity monitoring should alert before the filesystem loses enough headroom to restart safely. Recovered bytes close the immediate incident; a corrected ownership and rotation contract prevents its return.