When an LVM Thin Pool Fills Up, Free Space Is Not the Whole Recovery

An LVM thin pool can fail while a filesystem still reports free space, because pool data blocks, pool metadata, volume-group extents and filesystem blocks are different resources. Recovery starts by identifying which resource is exhausted. Extend a healthy data or metadata component from real VG extents; if the pool reports needs_check, stop its consumers and treat the event as offline metadata repair followed by upper-layer validation.

At the console, resist the fastest-looking actions. Deleting a snapshot, forcing activation or running lvconvert --repair without a controlled outage can turn a capacity incident into irreversible loss. First contain new allocations, save the LVM and kernel evidence, then choose one branch.

Freeze allocation before investigating capacity

New thin snapshots, clones, guest writes and image imports can consume the last mappings while an operator is still reading status. Pause automation that creates volumes, stop backup jobs that snapshot the pool, and reduce high-write workloads through their normal application shutdown path. Do not detach storage from a running database or virtual machine merely to make LVM commands quiet.

Record the current LVM layout and recent device-mapper messages before changing anything:

sudo lvs -a -o lv_name,vg_name,lv_attr,lv_size,pool_lv,data_percent,metadata_percent,lv_health_status,seg_monitor
sudo vgs -o vg_name,vg_size,vg_free,pv_count,lv_count
sudo pvs -o pv_name,vg_name,pv_size,pv_free,pv_attr
sudo journalctl -k --since '-2 hours' | grep -Ei 'device-mapper|thin|lvm'

Names in this article use VG/POOL as placeholders. Substitute the verified thin-pool LV, never a thin volume that happens to have a similar name. Hidden _tdata and _tmeta components appear with lvs -a, but normal pool extension should target the visible pool unless current LVM documentation or vendor support directs otherwise.

Four numbers define the incident

Data% is the proportion of mapped blocks used in the pool’s data LV. Meta% measures space used by metadata that maps thin virtual blocks to physical pool blocks. VFree is unallocated physical capacity in the volume group. Filesystem free space sits inside a thin LV and may not return mappings to the pool until discard reaches device-mapper.

The current lvmthin manual says a pool should be extended before either data or metadata reaches 100%. Read health and kernel state with those percentages; a low number after repair does not clear a needs_check flag by itself.

Read the device-mapper state, not only percentages

The Linux kernel’s thin-provisioning documentation distinguishes out_of_data_space, read-only operation, Fail and needs_check. Data exhaustion may queue or error I/O according to pool configuration and timeout. Metadata exhaustion can abort the current metadata transaction and require offline repair before normal operation.

Inspect the mapped pool without trying to repair it:

sudo dmsetup status
sudo lvs -a -o name,segtype,lv_attr,data_percent,metadata_percent,lv_health_status,seg_monitor VG
sudo lvmconfig --type full activation/thin_pool_autoextend_threshold activation/thin_pool_autoextend_percent
systemctl status lvm2-monitor --no-pager

lvm2-monitor and setting names vary across distributions and LVM2 builds, so confirm local unit and package documentation. A threshold of 100 disables automatic extension. Even an enabled monitor cannot allocate extents that the VG does not own.

Branch one: healthy metadata, exhausted data space

When Data% is critical, Meta% remains healthy, no repair flag appears and the VG has free extents, extending the visible thin pool is the direct capacity action. Size the increment from growth rate and a safe operating window instead of consuming every remaining extent.

Preview what LVM would do, then run the reviewed command:

sudo lvextend --test -L +20G VG/POOL
sudo lvextend -L +20G VG/POOL
sudo lvs -o name,lv_size,data_percent,metadata_percent,lv_health_status VG/POOL

The lvextend manual warns that test mode suppresses metadata writes but can still produce unusual multi-stage results. Treat it as a preview, not proof. Keep some VG extents in reserve for metadata, emergency growth and LVM metadata-spare needs.

If VFree is zero, the incident moves below LVM. Add an approved physical volume, extend the existing backing device and PV through the platform’s documented path, or migrate workloads to verified capacity. Never invent free extents by shrinking a mounted filesystem or removing an unknown PV. Teams without a tested storage outage procedure should use managed infrastructure support before attempting the lower-layer change.

Branch two: metadata pressure without a repair flag

High Meta% with a healthy pool needs metadata headroom before mappings reach 100%. Current lvmthin(7) and lvextend(8) support growing only pool metadata with --poolmetadatasize; the plus sign matters because it requests a relative increase.

sudo lvextend --test --poolmetadatasize +1G VG/POOL
sudo lvextend --poolmetadatasize +1G VG/POOL
sudo lvs -o name,metadata_percent,lv_health_status,seg_monitor VG/POOL

Choose the amount from current metadata size, growth and supported limits rather than copying 1 GiB blindly. Large snapshot populations and small chunk sizes change metadata demand. Preserve a pool metadata spare and verify the actual extent allocation after the command.

Branch three: needs_check or failed metadata

A repair flag changes the safety model. The kernel documentation states that metadata exhaustion or a failed metadata operation can error I/O until the pool is offline and repaired; it also recommends consistency checks on upper layers because cached writes may have been acknowledged before the metadata transaction aborted.

Before repair, back up LVM configuration and collect versioned evidence:

sudo vgcfgbackup VG
sudo lvm version
sudo thin_check --version
sudo lvs -a -o name,lv_attr,lv_health_status,data_percent,metadata_percent VG

Stop every consumer cleanly, confirm no mount, guest or process owns the thin LVs, and arrange access that does not depend on the affected pool. Only then follow current distribution/vendor guidance for deactivation and lvconvert --repair VG/POOL. The lvconvert manual supports repair for thin pools, but the command may swap in pool metadata spare space and preserve damaged metadata for analysis. Do not improvise direct thin_repair, metadata-device replacement or force activation from a generic web recipe. Escalate when backups, spare metadata or consumer shutdown are uncertain.

Validate every layer before restoring writes

Capacity recovery is not application recovery. After the pool activates cleanly, verify LVM health and watch kernel messages while each dependent layer returns. Filesystem checks must match filesystem type and normally run unmounted; virtual machines require guest-aware checks rather than host-side fsck against an active guest disk.

For virtualization hosts, follow storage recovery with a timed Proxmox restore drill using an isolated target. That test proves more than successful pool activation: it demonstrates that backups can produce an accepted guest when the original storage cannot.

Prevent another silent overcommit

Monitor Data%, Meta%, pool health and VFree together. One practical policy alerts well before the configured extension threshold, reserves enough VG extents for at least one policy action, and pages immediately on read-only, Fail or needs_check. Thresholds should come from peak growth and response time; a universal percentage cannot represent every write rate.

Review current policy without editing it:

sudo lvmconfig --type current activation/thin_pool_autoextend_threshold activation/thin_pool_autoextend_percent
sudo lvs -o name,data_percent,metadata_percent,lv_health_status,seg_monitor VG/POOL
sudo vgs -o name,free VG

Example planning logic is simple: if the pool can grow 8% during the longest unattended interval, a 95% alert leaves too little time. Configure earlier alerting, test that dmeventd actually extends a non-production pool, and prove that extension leaves reserve extents. Automatic growth without a capacity ceiling only postpones the incident.

Close on allocation proof, not a lower percentage

Save before/after LVM reports, the exact capacity or repair action, VG extents remaining, pool health, filesystem or guest checks, application acceptance and the next alert threshold. A falling percentage is useful, but successful new allocation plus an upper-layer transaction is the recovery proof.

Thin provisioning remains valuable because virtual capacity can be assigned without reserving every block on day one. That benefit creates an operational obligation: data mappings, metadata mappings and physical extents need independent budgets. Continue through Voxfor’s Linux operations library for related incident runbooks, and keep this three-state fork attached to the storage alert.