Restore Sparse Files With GNU tar, Then Verify the Holes

Two restored files can have the same length and SHA-256 hash yet consume radically different disk space. In the reproduced GNU tar 1.35 lab below, both candidates were byte-identical to a 256 MiB source. The ordinary restore allocated 268,435,456 bytes; the sparse restore allocated 12,288 bytes and preserved the source’s three data extents.

That result defines the real acceptance rule. tar exit status and a matching hash prove the byte stream, but they do not prove the filesystem recreated the holes. Create the archive with sparse metadata, extract into a new target, then measure apparent size, allocated blocks and data extents on the destination before replacing anything.

This how-to is for a Linux backup operator or developer with an owned test shell. It uses no production data, mounts or privileged device access. Run the five blocks in order in one Bash session; the first block installs an ownership-checked cleanup trap, and the last removes only the random lab scope and its explicit receipt.

One File Can Have a Logical Size and a Physical Footprint

A sparse file contains unwritten ranges called holes. Its logical or apparent size includes those ranges, while the filesystem may allocate blocks only for the real data islands. Applications read zeros from a hole, so the file behaves like a fully populated byte stream even though many physical blocks do not exist.

That transparent zero behavior creates the trap: a copy tool can read every hole as zeros and write those zeros back as ordinary allocated data. The restored bytes still match. Storage consumption does not. GNU tar’s sparse-file documentation demonstrates the same distinction between a file’s reported length and the blocks stored in its archive.

Do not mix this regular-file shape with a virtual disk’s internal allocation. A QCOW2 image adds guest filesystems, image clusters and storage-backend behavior; use the separate QCOW2 allocated-space workflow when those layers are involved. Likewise, a df/du gap caused by an unlinked file still held by a process belongs to deleted-open-file diagnosis, not sparse archive handling.

Start by creating an unpredictable temporary directory. The marker and path pattern prevent the trap from deleting an unrelated location if a variable is changed or the sequence is interrupted.

set -Eeuo pipefail

LAB_DIR="$(mktemp -d /tmp/voxfor-gnu-tar-sparse-171.XXXXXX)"
MARKER="$LAB_DIR/.voxfor-owned"
SOURCE_DIR="$LAB_DIR/source"
RAW_DIR="$LAB_DIR/restore-ordinary"
SPARSE_DIR="$LAB_DIR/restore-sparse"
SOURCE="$SOURCE_DIR/volume.bin"
RAW_RESTORE="$RAW_DIR/volume.bin"
SPARSE_RESTORE="$SPARSE_DIR/volume.bin"
RAW_ARCHIVE="$LAB_DIR/ordinary.tar"
SPARSE_ARCHIVE="$LAB_DIR/sparse-pax.tar"
RECEIPT_COPY="$PWD/gnu-tar-sparse-receipt-171.txt"

cleanup() {
  if [[ -n "${LAB_DIR:-}" && -d "$LAB_DIR" && -f "$MARKER" ]] \
    && [[ "$(<"$MARKER")" == "voxfor-gnu-tar-sparse-171" ]] \
    && [[ "$LAB_DIR" == /tmp/voxfor-gnu-tar-sparse-171.* ]]; then
    find "$LAB_DIR" -depth -mindepth 1 -delete
    rmdir "$LAB_DIR"
  fi
}
trap cleanup EXIT

for command_name in tar python3 truncate dd stat findmnt sha256sum; do
  command -v "$command_name" >/dev/null
done
[[ ! -e "$RECEIPT_COPY" ]]
printf '%s\n' 'voxfor-gnu-tar-sparse-171' > "$MARKER"
mkdir -p "$SOURCE_DIR" "$RAW_DIR" "$SPARSE_DIR"

printf 'environment\ttar=%s\tkernel=%s\tfilesystem=%s\n' \
  "$(tar --version | sed -n '1s/^tar (GNU tar) //p')" \
  "$(uname -r)" \
  "$(findmnt -T "$LAB_DIR" -no FSTYPE)"

Our published receipt comes from GNU tar 1.35 on Linux 6.12 with tmpfs as the test filesystem. A production record should capture the producer and consumer versions plus both source and destination filesystems. Sparse support is an end-to-end property, not a flag that makes every target behave identically.

Build Three Data Islands and Record the Source Truth

Build an apparent length of 256 MiB with nonzero text near byte zero, 96 MiB and the end. truncate creates the logical length without writing the gaps. Each small dd then allocates one filesystem block around its offset.

Inside the embedded inspector, Linux SEEK_DATA and SEEK_HOLE list every data extent. It also records st_blocks × 512, the portable unit Linux exposes through stat, instead of assuming the filename’s length equals its storage use.

cat > "$LAB_DIR/inspect.py" <<'PY'
import hashlib, os, sys

def inspect(label, path):
    st = os.stat(path)
    with open(path, 'rb') as fh:
        digest = hashlib.file_digest(fh, 'sha256').hexdigest()
    extents = []
    fd = os.open(path, os.O_RDONLY)
    try:
        pos = 0
        while pos < st.st_size:
            try:
                data = os.lseek(fd, pos, os.SEEK_DATA)
            except OSError:
                break
            hole = os.lseek(fd, data, os.SEEK_HOLE)
            extents.append(f"{data}-{hole}")
            pos = hole
    finally:
        os.close(fd)
    print(
        f"{label}\tapparent={st.st_size}\tallocated={st.st_blocks * 512}"
        f"\tsha256={digest}\textents={','.join(extents)}"
    )
    return st.st_size, st.st_blocks * 512, digest, extents

if len(sys.argv) == 3:
    inspect(sys.argv[1], sys.argv[2])
else:
    source = inspect('source', sys.argv[1])
    ordinary = inspect('ordinary_restore', sys.argv[2])
    sparse = inspect('sparse_restore', sys.argv[3])
    assert source[0] == ordinary[0] == sparse[0] == 256 * 1024 * 1024
    assert source[2] == ordinary[2] == sparse[2]
    assert ordinary[1] >= 250 * 1024 * 1024
    assert sparse[1] < 128 * 1024
    assert source[3] == sparse[3]
    print('receipt\tbytes_equal=yes\tordinary_fully_allocated=yes\tsparse_extents_preserved=yes')
PY

truncate --size=256M "$SOURCE"
printf 'VOXFOR-SPARSE-START\n' | dd of="$SOURCE" bs=1 seek=0 conv=notrunc status=none
printf 'VOXFOR-SPARSE-MIDDLE\n' | dd of="$SOURCE" bs=1 seek=$((96 * 1024 * 1024)) conv=notrunc status=none
printf 'VOXFOR-SPARSE-END\n' | dd of="$SOURCE" bs=1 seek=$((256 * 1024 * 1024 - 21)) conv=notrunc status=none
sync "$SOURCE"

python3 "$LAB_DIR/inspect.py" source "$SOURCE"

In the reproduced source, the receipt reports 268,435,456 apparent bytes, 12,288 allocated bytes and three 4 KiB data extents. On a filesystem that does not implement accurate SEEK_HOLE/SEEK_DATA, the inspector may return a coarser map or no map. That is a compatibility finding, not permission to invent the expected extents.

Archive the Same Source With and Without Sparse Metadata

GNU tar’s --sparse option tells the creator to detect holes and avoid writing their zero-filled contents into the archive. --hole-detection=seek uses the filesystem’s hole interface when available; the default can fall back to raw scanning. The GNU tar manual page also documents sparse format versions 0.0, 0.1 and 1.0.

This comparison deliberately creates one ordinary archive and one POSIX/PAX archive with GNU sparse version 1.0. Both use the same source path and no compression, making format behavior visible instead of letting compression hide the cost of stored zeros.

tar --create --file "$RAW_ARCHIVE" \
  -C "$SOURCE_DIR" volume.bin

tar --create --format=pax --sparse-version=1.0 \
  --file "$SPARSE_ARCHIVE" \
  -C "$SOURCE_DIR" volume.bin

printf 'archives\tordinary=%s\tsparse_pax=%s\n' \
  "$(stat -c %s "$RAW_ARCHIVE")" \
  "$(stat -c %s "$SPARSE_ARCHIVE")"
tar --list --verbose --file "$SPARSE_ARCHIVE"

Measured archive size was 268,441,600 bytes for the ordinary archive and 20,480 bytes for the PAX sparse archive. Those are lab results, not universal ratios. Header blocks, data-island count, filesystem granularity, tar version and chosen format all influence the exact archive size.

Red Hat’s remote sparse-copy case connects detection performance to FIEMAP and SEEK_HOLE/SEEK_DATA, while its broader oversized tar migration symptom shows why a same-sized destination can unexpectedly fill. Treat older vendor examples as context: rerun the producer/consumer pair you actually deploy.

Restore Both Candidates and Read All Three Signals

Extract into two new empty directories. Do not overwrite the source or an earlier recovery candidate: an in-place extraction can destroy the very rollback artifact needed to investigate a format or allocation mismatch.

Python assertions enforce the central claims. All three apparent sizes must be 256 MiB; all hashes must match; the ordinary restore must allocate at least 250 MiB; the sparse restore must remain below 128 KiB; and the sparse restore’s data extents must equal the source map exactly.

tar --extract --file "$RAW_ARCHIVE" -C "$RAW_DIR"
tar --extract --file "$SPARSE_ARCHIVE" -C "$SPARSE_DIR"

python3 "$LAB_DIR/inspect.py" \
  "$SOURCE" "$RAW_RESTORE" "$SPARSE_RESTORE" \
  | tee "$RECEIPT_COPY"

[[ "$(stat -c %s "$RAW_ARCHIVE")" -ge $((250 * 1024 * 1024)) ]]
[[ "$(stat -c %s "$SPARSE_ARCHIVE")" -lt $((128 * 1024)) ]]
[[ "$(sha256sum "$SOURCE" | awk '{print $1}')" == \
   "$(sha256sum "$SPARSE_RESTORE" | awk '{print $1}')" ]]

Representative output from the reproduced run:

source            apparent=268435456 allocated=12288     sha256=bff780f37c96bb3a3b67154c9ec4c14044e63a089b65bf59cad6e033f9d25856 extents=0-4096,100663296-100667392,268431360-268435456
ordinary_restore  apparent=268435456 allocated=268435456 sha256=bff780f37c96bb3a3b67154c9ec4c14044e63a089b65bf59cad6e033f9d25856 extents=0-268435456
sparse_restore    apparent=268435456 allocated=12288     sha256=bff780f37c96bb3a3b67154c9ec4c14044e63a089b65bf59cad6e033f9d25856 extents=0-4096,100663296-100667392,268431360-268435456
receipt           bytes_equal=yes ordinary_fully_allocated=yes sparse_extents_preserved=yes

Here the ordinary restore is the negative control that makes the decision useful. Its matching hash proves that content verification alone cannot detect lost sparseness. The sparse candidate proves more: byte equality, allocation bound and exact extent equality all agree in the tested environment.

Accept this lab result when all three apparent sizes are exactly 268435456, all three SHA-256 values match, the ordinary control allocates at least 250 MiB, the sparse candidate allocates less than 128 KiB, the source and sparse extent strings are identical, and the final receipt says both byte and sparse-extent checks succeeded. For production, replace the numeric allocation threshold with one derived from the source’s measured blocks, metadata overhead and documented tolerance.

Sparse Format and Destination Support Are One Contract

An archive is not sparse merely because its source was sparse. The producer must encode a sparse map, the consumer must understand that representation, and the destination filesystem must be able to create holes. Move any one of those variables and rerun extraction before scheduling the real recovery.

PAX improves the container’s extensibility, but GNU sparse keys are still a compatibility surface. Oracle’s GNU tar manual mirror exposes the same sparse-version controls. IBM’s AIX sparse-file guidance reaches different tar/pax conclusions for AIX tools, which is exactly why a Linux GNU tar result must not be presented as a universal tar guarantee.

Compression answers another question. gzip or xz may compress long zero runs into very few archive bytes even when the archive contains no sparse map. Extraction can still write a fully allocated file because the compressor reconstructs the zeros and the archive reader writes them. Measure the extracted target, not only the compressed archive.

Hole detection also deserves a real preflight. --hole-detection=seek is efficient when the filesystem reports data and holes accurately. raw scans content and can treat sufficiently long zero regions as holes, but that may not preserve an intentionally allocated zero range as allocation. If an application distinguishes allocation for performance, reservation or copy-on-write behavior, record that requirement before selecting the detector.

Control-panel exports have their own layouts, exclusions and restore rules. Inspect those artifacts with a dedicated cPanel archive preflight instead of adding -S to an opaque platform workflow and assuming the platform will retain it.

Convert the Lab Into a Production Restore Rule

Start from a stable source. GNU tar can preserve a sparse byte stream and its holes; it cannot make an actively changing database transactionally consistent. Stop writes, take a documented application snapshot, or use the engine’s backup interface before archiving. For SQLite in WAL mode, the online backup and restore verification path demonstrates why copying one visible database file is not enough.

Next, reserve destination headroom for the failure case. A compact archive can expand toward apparent size if the consumer or filesystem loses sparse metadata. Extract into a separate filesystem or staging path with enough room for that possibility, monitor free blocks during the test, and abort before a shared production volume reaches its safety margin.

Keep one receipt containing source path or immutable snapshot ID, GNU tar version, archive format and sparse version, source/destination filesystems, apparent and allocated bytes before and after, hash algorithm, extent method, extraction command, application check and rollback deadline. Broader restore-data verification can then add ownership, ACLs, xattrs, symlinks and application reads without confusing them with this allocation gate.

Clean the disposable fixture only after the receipt has been reviewed. The block verifies both ownership markers and refuses any path outside the random article prefix.

[[ "$(<"$MARKER")" == "voxfor-gnu-tar-sparse-171" ]]
[[ "$LAB_DIR" == /tmp/voxfor-gnu-tar-sparse-171.* ]]
[[ -s "$RECEIPT_COPY" ]]
sha256sum "$RECEIPT_COPY"

cleanup
trap - EXIT
[[ ! -e "$LAB_DIR" ]]
rm -- "$RECEIPT_COPY"
[[ ! -e "$RECEIPT_COPY" ]]
printf 'cleanup\tlab_absent=yes\treceipt_copy_absent=yes\n'

If any hash, allocation or extent assertion fails, leave the original source and archives unchanged, reject the candidate extraction, and collect the producer version, consumer version and destination-filesystem evidence before retrying in another empty directory. The disposable sequence removes only a directory with the exact random prefix and owner marker; production rollback is to keep using the preserved original or previous restore target, never to run fallocate --dig-holes against the sole copy just to reduce du output.

Preserve the Original Until the Application Uses the Restore

This lab proves a narrow but consequential result: the ordinary and sparse restores can match every byte while only one preserves the source’s physical shape. Production acceptance therefore needs two storage signals—allocated bytes and extents—beside the content hash.

Application behavior remains the final boundary. Start the candidate in isolation, run the database, service or image-specific read check, and retain the original until the rollback window closes. Capacity planning must also count directory depth and object count; a website restore-time file-count test covers that separate bottleneck when millions of small files outweigh one sparse object’s apparent length.

Use GNU tar sparse metadata because the tested producer and consumer support it, not because the archive looks small. A compact archive is an input; a byte-identical, allocation-bounded, application-readable extraction is the restore.