zstd Frame Integrity Does Not Verify Your Restored Files

A successful zstd integrity test answers one narrow question: can the compressed frame be decoded without an error? It does not prove that the file received is the object the sender intended, that an archive inside it is safe to extract, that every restored file matches a source baseline, or that an application can use those files.

Treat those as separate admission boundaries. Record a sender-side SHA-256 digest, test the received frame without modifying it, inspect the decoded archive before isolated extraction, and compare the restored tree with a manifest created before compression. A known-bad duplicate should also fail. Only that layered receipt supports an admit-or-reject decision.

The reproduced lab below ran on Debian 13.6 with zstd 1.5.7, GNU tar 1.35 and GNU coreutils 9.7. Four deterministic files totaling 143,469 bytes became a 7,945-byte checksum-bearing frame. The received copy matched its expected SHA-256, all four restored files matched the retained manifest, and a copy truncated by 64 bytes was rejected. The original remained unchanged.

Treat One Green Exit as One Boundary

A zstd frame is the formatted compressed unit that carries blocks plus optional fields such as decoded content size, dictionary identity and content checksum. Upstream Zstandard format specification defines that checksum as optional. When present, it stores the low 32 bits of an XXH64 digest calculated over decoded content.

Current zstd CLI documentation defines --test as decompression with output discarded. That is valuable: it reads the full frame and returns nonzero on detected decode or checksum failure. It deliberately does not create a restored file tree.

Four claims must therefore remain separate:

1. Transport identity: expected and received compressed objects share a content digest.
2. Frame decodability: zstd can decode the complete frame and validate its checksum when one exists.
3. Container scope: archive members can be admitted to an isolated extraction root.
4. Restored payload identity: restored regular files match a manifest recorded before compression.

File names, sizes and storage-provider receipts are not substitutes for the first claim. For example, multipart S3 ETag behavior can reflect upload shape rather than a conventional full-object MD5. Record an explicit SHA-256 beside the backup instead.

Write Expected File Identity Before Compression

This lab creates its own disposable source so every expected byte is known. For a real backup, create the manifest on the authoritative source before archiving. If no source manifest exists, you can still prove transport identity and frame decodability, but you cannot retroactively prove that the restored file set is what the source was supposed to contain.

Admit tools and marker-owned scope

Run the sequence in Bash on a disposable Linux host. It requires zstd, GNU tar, sha256sum and standard core utilities. The setup refuses to overwrite an existing receipt and defines cleanup that accepts only the random lab prefix plus the exact marker content.

set -Eeuo pipefail
umask 077

LAB_DIR=$(mktemp -d /tmp/voxfor-zstd-184.XXXXXX)
MARKER="$LAB_DIR/.voxfor-owned"
SOURCE_DIR="$LAB_DIR/source"
RESTORE_DIR="$LAB_DIR/restore"
RECEIPT="$LAB_DIR/receipt.txt"
RECEIPT_COPY="$PWD/zstd-verification-184.txt"

for tool in zstd tar sha256sum awk find sort xargs stat   truncate grep sed cmp wc; do
  command -v "$tool" >/dev/null
done
[[ ! -e "$RECEIPT_COPY" ]]

printf '%s\n' 'voxfor-zstd-184' > "$MARKER"
install -d -m 0700 "$SOURCE_DIR/config" "$SOURCE_DIR/data"   "$SOURCE_DIR/logs" "$RESTORE_DIR"

cleanup_zstd_lab() {
  if [[ -d "${LAB_DIR:-}" && -f "${MARKER:-}" ]] &&
     [[ "$(<"$MARKER")" == "voxfor-zstd-184" ]] &&
     [[ "$LAB_DIR" == /tmp/voxfor-zstd-184.* ]]; then
    find "$LAB_DIR" -depth -mindepth 1 -delete
    rmdir "$LAB_DIR"
  else
    printf 'cleanup=refused-unowned-path\n' >&2
    return 1
  fi
}
trap cleanup_zstd_lab EXIT

printf 'environment\tzstd=%s\ttar=%s\tsha256sum=%s\tkernel=%s\n'   "$(zstd --version | sed -n 's/^.* v\([^,]*\),.*$/\1/p')"   "$(tar --version | sed -n '1s/^tar (GNU tar) //p')"   "$(sha256sum --version | sed -n '1s/^sha256sum (GNU coreutils) //p')"   "$(uname -r)" | tee "$RECEIPT"

Keep the receipt outside the disposable directory. Give each real run a new receipt path rather than deleting old evidence. The trap will clean the lab after any failed assertion, but it will not touch the external receipt or an unrelated path.

Create four files and retain their manifest

Source material includes configuration, structured text, a short log and an empty marker. Fixed timestamps make the tar reproducible; sorted null-delimited paths make manifest order deterministic.

printf '%s\n' 'app_mode=production' 'retention_days=14'   > "$SOURCE_DIR/config/app.conf"
awk 'BEGIN {
  for (i=1; i<=4096; i++)
    printf "record=%04d tenant=%02d state=%s\\n",
      i, i%17, (i%11 ? "ready" : "retry")
}' > "$SOURCE_DIR/data/records.txt"
printf '%s\n'   '2026-08-15T04:20:00Z backup-start'   '2026-08-15T04:20:02Z backup-complete'   > "$SOURCE_DIR/logs/backup.log"
install -m 0600 /dev/null "$SOURCE_DIR/data/empty.marker"
chmod 0644 "$SOURCE_DIR/config/app.conf" "$SOURCE_DIR/data/records.txt" \
  "$SOURCE_DIR/logs/backup.log"

find "$SOURCE_DIR" -type f   -exec touch -d '2026-08-15 04:20:00 UTC' {} +

(
  cd "$LAB_DIR"
  find source -type f -print0 |
    sort -z |
    xargs -0 sha256sum > source.sha256
)

SOURCE_MANIFEST_SHA=$(sha256sum "$LAB_DIR/source.sha256" |
  awk '{print $1}')
printf 'fixture\tfiles=%s\tbytes=%s\tmanifest_sha256=%s\n'   "$(find "$SOURCE_DIR" -type f | wc -l)"   "$(find "$SOURCE_DIR" -type f -printf '%s\n' |
     awk '{sum+=$1} END {print sum+0}')"   "$SOURCE_MANIFEST_SHA" | tee -a "$RECEIPT"

Manifest ownership matters: the checksum list is verification evidence, not part of the source tree being checked. Store a real sender manifest separately from the compressed object and protect it with access controls appropriate to the backup.

Admit the Received Frame Without Changing It

Next, create a deterministic tar, request a zstd content checksum, simulate a transfer into a new file and test only that received copy. Nothing overwrites backup.tar.zst.

Build a checksum-bearing frame

tar --sort=name   --mtime='2026-08-15 04:20:00 UTC'   --owner=0 --group=0 --numeric-owner   -C "$LAB_DIR" -cf "$LAB_DIR/backup.tar" source

zstd -q -T1 -6 --check   "$LAB_DIR/backup.tar"   -o "$LAB_DIR/backup.tar.zst"

EXPECTED_ZST_SHA=$(sha256sum "$LAB_DIR/backup.tar.zst" |
  awk '{print $1}')
printf 'compressed\ttar_bytes=%s\tzst_bytes=%s\tzst_sha256=%s\n'   "$(stat -c %s "$LAB_DIR/backup.tar")"   "$(stat -c %s "$LAB_DIR/backup.tar.zst")"   "$EXPECTED_ZST_SHA" | tee -a "$RECEIPT"

--check makes the checksum premise explicit for this lab. Do not assume that every inherited .zst contains one: inspect metadata and document the producer settings. Even without a frame checksum, a full decode can catch structural errors, but it lacks that final content-checksum comparison.

Compare transport identity and inspect metadata

cp --reflink=auto   "$LAB_DIR/backup.tar.zst"   "$LAB_DIR/received.tar.zst"

RECEIVED_ZST_SHA=$(sha256sum "$LAB_DIR/received.tar.zst" |
  awk '{print $1}')
[[ "$EXPECTED_ZST_SHA" == "$RECEIVED_ZST_SHA" ]]

zstd -lv "$LAB_DIR/received.tar.zst"   > "$LAB_DIR/frame-list.txt" 2>&1
grep -Eq 'Check|checksum|XXH64' "$LAB_DIR/frame-list.txt"

printf 'transfer\texpected_sha256=%s\treceived_sha256=%s\tbytes=%s\n'   "$EXPECTED_ZST_SHA"   "$RECEIVED_ZST_SHA"   "$(stat -c %s "$LAB_DIR/received.tar.zst")" |
  tee -a "$RECEIPT"

sed -n '1,8p' "$LAB_DIR/frame-list.txt" |
  sed 's/^/frame_metadata\t/' |
  tee -a "$RECEIPT"

SHA-256 equality means the received compressed bytes match the sender-recorded object in this run. It says nothing yet about whether the sender archived a consistent database, included every tenant, preserved sparse allocation or chose the correct retention point.

Test complete decoding with discarded output

zstd --test -v "$LAB_DIR/received.tar.zst"   > "$LAB_DIR/frame-test.txt" 2>&1

[[ -s "$LAB_DIR/frame-test.txt" ]]
printf 'frame_test\tstatus=decoded_without_error\tchecksum_flag=present\n' |
  tee -a "$RECEIPT"

Exit zero is the decisive result; diagnostic wording varies by zstd version. The current Debian zstd man page also describes test mode as decompression with output discarded. Preserve stderr when a test fails, but do not weaken the gate by matching one preferred success phrase.

Inspect the Container, Then Restore into Isolation

Frame success exposes decoded bytes. In this fixture those bytes form a tar archive, so the next decision belongs to tar rather than zstd. A .zst could instead contain a database dump, disk image or single file; route that object to its owning format checker.

Decode to a new tar and screen member paths

zstd -q -dc "$LAB_DIR/received.tar.zst"   > "$LAB_DIR/received.tar"

cmp -s "$LAB_DIR/backup.tar" "$LAB_DIR/received.tar"
tar -tf "$LAB_DIR/received.tar" > "$LAB_DIR/members.txt"

awk '
  /^\// { bad=1 }
  /(^|\/)\.\.($|\/)/ { bad=1 }
  END { exit bad ? 1 : 0 }
' "$LAB_DIR/members.txt"

tar -xf "$LAB_DIR/received.tar"   -C "$RESTORE_DIR"   --no-same-owner

printf 'restore\tmembers=%s\ttar_sha256=%s\tunsafe_paths=0\n'   "$(wc -l < "$LAB_DIR/members.txt")"   "$(sha256sum "$LAB_DIR/received.tar" | awk '{print $1}')" |
  tee -a "$RECEIPT"

The current Debian GNU tar man page documents that --absolute-names stops stripping leading slashes from member names. This lab rejects absolute and parent-traversal names, extracts into an empty isolated directory, and avoids ownership restoration.

That predicate is not a complete hostile-archive sandbox. It does not validate link targets, device nodes, extended attributes or behavior of programs opened afterward. For untrusted customer input, inspect verbose member types and use stronger process/filesystem isolation. For a cPanel-specific package, follow a cPanel archive preflight that understands that container’s structure instead of treating generic tar success as account acceptance.

Let Restored Files Answer to the Manifest

Run the retained sender manifest from the restore root. Paths in source.sha256 are relative to that root, so every expected file must be present with the exact content hash.

(
  cd "$RESTORE_DIR"
  sha256sum -c "$LAB_DIR/source.sha256"     > "$LAB_DIR/manifest-check.txt"
)

grep -q ': OK$' "$LAB_DIR/manifest-check.txt"
[[ "$(grep -c ': OK$' "$LAB_DIR/manifest-check.txt")" -eq 4 ]]

RESTORED_MANIFEST_SHA=$(sha256sum "$LAB_DIR/source.sha256" |
  awk '{print $1}')
printf 'manifest\tfiles_ok=4\tmanifest_sha256=%s\n'   "$RESTORED_MANIFEST_SHA" | tee -a "$RECEIPT"

The current Ubuntu sha256sum man page documents --check as reading recorded checksums and checking the named files. Four OK lines prove the four expected regular files exist with matching bytes. They do not prove mode, owner, ACL, xattr, sparse-hole layout or absence of unexpected files unless those properties are separately inventoried.

Sparse data needs a GNU tar sparse-file allocation check after extraction; byte equality alone can coexist with much higher allocated space. Databases need semantic checks too. A SQLite site using WAL mode should begin with SQLite WAL backup and restore verification because a perfect archive cannot repair an inconsistent capture.

Make a Damaged Duplicate Fail on Purpose

A negative control checks whether the gate can reject a defect that should be visible. Copy the received frame, truncate only the copy by 64 bytes and require nonzero status. Never perform this control on the sole backup.

cp --reflink=auto   "$LAB_DIR/received.tar.zst"   "$LAB_DIR/damaged.tar.zst"

ORIGINAL_SIZE=$(stat -c %s "$LAB_DIR/damaged.tar.zst")
[[ "$ORIGINAL_SIZE" -gt 128 ]]
truncate -s "$((ORIGINAL_SIZE - 64))"   "$LAB_DIR/damaged.tar.zst"

set +e
zstd --test "$LAB_DIR/damaged.tar.zst"   > "$LAB_DIR/damaged.stdout"   2> "$LAB_DIR/damaged.stderr"
DAMAGED_RC=$?
set -e

[[ "$DAMAGED_RC" -ne 0 ]]
grep -Eqi 'error|corrupt|checksum|premature|unexpected'   "$LAB_DIR/damaged.stderr"

[[ "$(sha256sum "$LAB_DIR/damaged.tar.zst" | awk '{print $1}')"    != "$EXPECTED_ZST_SHA" ]]
[[ "$(sha256sum "$LAB_DIR/received.tar.zst" | awk '{print $1}')"    == "$EXPECTED_ZST_SHA" ]]

printf 'negative_control\tdamage=truncated_copy\trc=%s\toriginal_preserved=yes\n'   "$DAMAGED_RC" | tee -a "$RECEIPT"

This control proves rejection of this truncation shape, not every possible corruption or malicious archive. Its operational value is narrower: the test command, exit-status check and evidence capture are wired to stop on a known bad object.

Turn the Receipt into a Backup Gate

The last input records the connected decisions, copies the receipt outside the lab, runs the guarded cleanup and confirms the disposable path is gone.

printf 'acceptance\ttransport_hash_match=yes\tframe_test=yes\ttar_bytes_match=yes\trestored_manifest_match=yes\tnegative_control_rejected=yes\tsource_manifest_sha256=%s\trestored_manifest_sha256=%s\n'   "$SOURCE_MANIFEST_SHA"   "$RESTORED_MANIFEST_SHA" |
  tee -a "$RECEIPT"

cp "$RECEIPT" "$RECEIPT_COPY"
cleanup_zstd_lab
trap - EXIT
[[ ! -e "$LAB_DIR" ]]

printf 'cleanup\tlab_absent=yes\treceipt=%s\n'   "$RECEIPT_COPY" | tee -a "$RECEIPT_COPY"

Representative output from the reproduced run:

environment  zstd=1.5.7  tar=1.35  sha256sum=9.7  kernel=6.12.96+deb13-amd64
fixture  files=4  bytes=143469  manifest_sha256=d2ed613c2c544270fd1ed609d98dd9dbd8e5a9830b671c5dcdf57af904af7e0b
compressed  tar_bytes=153600  zst_bytes=7945  zst_sha256=f89a5d0b67ba8d2489a92a4888ebce3367ea7dbe7e0c909774f43a809c5ea81b
transfer  expected_sha256=f89a5d0b67ba8d2489a92a4888ebce3367ea7dbe7e0c909774f43a809c5ea81b  received_sha256=f89a5d0b67ba8d2489a92a4888ebce3367ea7dbe7e0c909774f43a809c5ea81b
frame_test  status=decoded_without_error  checksum_flag=present
restore  members=8  unsafe_paths=0
manifest  files_ok=4  manifest_sha256=d2ed613c2c544270fd1ed609d98dd9dbd8e5a9830b671c5dcdf57af904af7e0b
negative_control  damage=truncated_copy  rc=1  original_preserved=yes
acceptance  transport_hash_match=yes  frame_test=yes  tar_bytes_match=yes  restored_manifest_match=yes  negative_control_rejected=yes
cleanup  lab_absent=yes  receipt=zstd-verification-184.txt

Accept the backup candidate only when expected and received SHA-256 match; frame metadata records the intended checksum state; the original passes complete zstd --test decoding; member-path preflight finds no absolute or parent-traversal name; extraction occurs in an empty isolated directory; all four expected files pass the retained manifest; the damaged duplicate exits nonzero; the original digest is unchanged; and marker-owned cleanup is recorded. Any failed condition blocks promotion.

Integrity is still not recovery time. Use a backup file-count restore-time test to measure how object count affects the window, then run workload-specific reads, database checks, service startup and authorization tests against the isolated candidate.

If any gate fails, preserve the original compressed object, sender digest, manifest and diagnostics; keep the existing workload and data authority unchanged; discard only the marker-owned candidate restore; and investigate transfer, producer or format compatibility without disabling checks. Never recompress, truncate or extract over the only backup as a repair attempt.