Check fstab Before Rebooting, Including Warnings

Before a remote Linux server reboots, an fstab change window needs four durable facts: the known-good table, the exact proposed row, a static-verification receipt and a tested backout route. None can be reconstructed reliably after the host has stopped reaching normal multi-user boot.

The working rule is procedural: preserve the return path first, review one changed row, define the exact live test, and only then use static verification to decide whether the window may advance. A clean parser result is one checkpoint inside that record, not the whole change.

This runbook is for a Linux operator working over SSH. Its six tested inputs use a private temporary directory, a 16 MiB filesystem image and a tmpfs row; they never edit /etc/fstab, mount storage or reload systemd. Every temporary lab path is marker-owned and removed, while one portable receipt is deliberately retained in the working directory for review.

Open the Change Window With a Known-Good Table

/etc/fstab is a declarative mount table. According to the current fstab manual, tools such as mount, umount and fsck iterate its records, and record order can matter. A bad line can therefore affect boot ordering, filesystem checks, optional storage, network shares or application paths before an operator gets a normal login.

findmnt --verify checks the table through util-linux’s libmount logic. The findmnt manual describes the default job as checking /etc/fstab parsability and usability. In practice, that includes target reachability, source tags, filesystem-type evidence, duplicate targets, parent/child ordering and filesystem-check metadata where the tool has enough information.

Before installing a candidate, preserve the live file with its owner, mode and timestamp; record its hash; identify the one changed target; and confirm that console, rescue or provider recovery access actually opens. The change record should also name the person allowed to approve a warning. A successful SSH session is not alternate access when the same reboot can remove its root or network dependency.

Two verifier result channels remain distinct:

• The exit status becomes nonzero when verifier errors exist.
• The printed warning count can be positive while the exit status remains zero.

The util-linux verification source keeps warning and error counters separately. That implementation detail matches our observation: warnings describe unresolved evidence, but they do not necessarily make the program fail. Sleepless Beastie’s worked fstab verification examples also show warning-only output and a distinct error case.

Warnings describe unresolved evidence even when they do not make the process fail. A warning about an intentionally absent removable device may later be accepted by policy, but the named exception belongs in the change record. It must not disappear merely because $? is zero.

Translate the Changed Row Into a Boot Contract

Each non-comment line carries source, target, filesystem type, mount options, dump frequency and filesystem-check pass number. Spaces inside a field must be escaped as \040; quotes do not remove that requirement. Device tags such as UUID= and LABEL= are normally more stable than /dev/sdX names, but a tag is useful only when it resolves to the intended filesystem.

Six fields answer different questions:

1. Source: Which block device, tag, remote export, image or pseudo-filesystem supplies the data?
2. Target: Which existing path will become the mount point?
3. Type: Which kernel or userspace filesystem handler should interpret the source?
4. Options: Is the mount automatic, optional, read-only, network-dependent or constrained by another policy?
5. Dump flag: Should the historical dump tool consider the filesystem?
6. Pass number: In what fsck order should a checkable local filesystem run?

These fields are not interchangeable safety knobs. Adding nofail because a UUID is wrong merely changes failure handling; it does not make the source correct. Likewise, _netdev changes boot dependency classification for a network mount, but it does not prove DNS, authentication or the remote export will work. When a boot waits on interface readiness, the separate systemd-networkd wait-online diagnosis helps identify the interface and policy owner instead of masking the delay inside fstab.

Define Exact-Target Acceptance Before the Rehearsal

Static verification reduces risk; it does not emulate a reboot. The current mount manual explicitly recommends findmnt --verify instead of using mount -a merely as an fstab syntax checker. mount -a can change live state, skip already mounted filesystems and interact with every automatic entry in the real table.

Reload generated mount units only after the file is accepted

On a systemd host, systemctl daemon-reload causes the manager to regenerate mount units from the saved table. Run it after installing the reviewed candidate, not before the static check. Then inspect the exact generated unit and dependency chain when the change involves _netdev, automounts, encrypted devices or parent mount points.

Do not reboot until console, rescue environment, provider recovery mode or a genuinely independent administrative path has been tested. The close criteria should state who observes the first boot, how long the workload is checked and what exact evidence triggers a backout.

Mount only the changed target inside its change window

When attaching the filesystem is safe, call mount with the exact target rather than broad mount -a, then confirm runtime identity with explicit columns:

findmnt --mountpoint /srv/data --output TARGET,SOURCE,FSTYPE,OPTIONS

Compare the resolved source, type and effective options with the reviewed candidate. For a read-write data path, create and remove one uniquely named application-approved probe. A status line alone does not prove the application user can traverse parent directories, write files, preserve ownership or read old data. If access still fails after a correct mount, Voxfor’s permission-denial layer guide helps distinguish mount flags, ACLs, MAC policy and service namespaces instead of reaching for chmod 777.

Audit the Verifier in a Disposable Rehearsal

Copying the proposed lines into a separate file gives the verifier a stable review artifact and keeps the known-good table unchanged. The first input creates that private scope, verifies required tools, builds one small ext4 image and records the actual environment.

set -Eeuo pipefail
export LC_ALL=C
lab_root=$(mktemp -d /tmp/voxfor-fstab-173.XXXXXX)
marker=$lab_root/.voxfor-owned
receipt_copy=$PWD/findmnt-fstab-receipt-173.txt
cleanup() {
  if [[ -n ${lab_root:-} && -d $lab_root && -f $marker ]] &&
     [[ $(<"$marker") == voxfor-fstab-173 ]] &&
     [[ $lab_root == /tmp/voxfor-fstab-173.* ]]; then
    find "$lab_root" -depth -mindepth 1 -delete
    rmdir "$lab_root"
  fi
}
trap cleanup EXIT
test ! -e "$receipt_copy"
printf '%s\n' voxfor-fstab-173 > "$marker"
for tool in findmnt mkfs.ext4 truncate grep sed awk sha256sum; do
  command -v "$tool" >/dev/null
done
mkdir -p "$lab_root/clean-target" "$lab_root/image-target"
truncate -s 16M "$lab_root/ext4.img"
mkfs.ext4 -q -F "$lab_root/ext4.img"
printf 'findmnt=%s\nkernel=%s\n' \
  "$(findmnt --version | sed -n '1s/^findmnt from util-linux //p')" "$(uname -r)"

Our receipt records util-linux 2.41 and Linux 6.12.96+deb13-amd64. Production evidence should also capture the candidate-table hash, host identity, UTC time, changed lines, console or rescue path and reviewer. Never publish real device UUIDs, remote-share credentials or private hostnames with the receipt.

Before testing a whole server file, create a protected backup with metadata preserved and compare the proposed copy. Do not paste the lab’s tmpfs or image paths into production; they exist only to make verifier behavior reproducible without a real disk.

Three fixtures expose why a single yes/no check is too weak. They share one shell so the marker and paths stay constant.

Establish the clean control

For a clean control, use a pseudo-filesystem with an existing target. After verification, a separate findmnt --fstab query proves how libmount parsed the row. No mount occurs.

printf 'tmpfs %s tmpfs defaults,nofail 0 0\n' \
  "$lab_root/clean-target" > "$lab_root/clean.fstab"
findmnt --verify --verbose --tab-file "$lab_root/clean.fstab" 2>&1 |
  tee "$lab_root/clean.out"
grep -Fqx 'Success, no errors or warnings detected' "$lab_root/clean.out"
findmnt --fstab --tab-file "$lab_root/clean.fstab" --noheadings \
  --output TARGET,SOURCE,FSTYPE,OPTIONS | tee "$lab_root/clean.parsed"
grep -Eq "^$lab_root/clean-target[[:space:]]+tmpfs[[:space:]]+tmpfs[[:space:]]+defaults,nofail$" \
  "$lab_root/clean.parsed"

Parsed output matters because a visually plausible line can still point at the wrong target or carry an unintended option. Scripts should request explicit output columns; findmnt’s default presentation is not a stable machine interface.

Prove that warnings can keep exit zero

Next, the warning fixture declares XFS over an ext4 image. The regular-file source also triggers a warning because the verifier sees a non-bind source that is not a block device. Both are real evidence failures for this candidate, yet warning_rc remains zero.

This is the reproduced util-linux 2.41 counterexample: the command returns 0 while printing two warnings, including xfs does not match with on-disk ext4. The observation belongs here, after the operational close criteria are already defined, because it tests one reason that a change window must remain on hold.

printf '%s %s xfs loop,nofail 0 2\n' \
  "$lab_root/ext4.img" "$lab_root/image-target" > "$lab_root/warning.fstab"
set +e
findmnt --verify --verbose --tab-file "$lab_root/warning.fstab" \
  > "$lab_root/warning.out" 2>&1
warning_rc=$?
set -e
cat "$lab_root/warning.out"
test "$warning_rc" -eq 0
grep -Fq '[W] non-bind mount source' "$lab_root/warning.out"
grep -Fq '[W] xfs does not match with on-disk ext4' "$lab_root/warning.out"
grep -Eq '0 parse errors, 0 errors, 2 warnings' "$lab_root/warning.out"
printf 'warning_case_exit=%s warnings=2 admitted_by_exit_only=yes\n' "$warning_rc"

Privilege can change some observations. A non-root process may be unable to read a block device and report that it cannot detect the on-disk type. Run the production check with the privilege needed to inspect the intended sources, then resolve the warning rather than assuming it is false. The Simplified Guide workflow usefully pairs full-table verification with a later exact-target mount check; our addition is making warning admission explicit.

Keep a required target error as a negative control

As a negative control, the third table names a target that does not exist and leaves the entry required. This time the verifier emits [E] and exits 1.

printf 'tmpfs %s tmpfs defaults 0 0\n' \
  "$lab_root/missing-target" > "$lab_root/error.fstab"
set +e
findmnt --verify --verbose --tab-file "$lab_root/error.fstab" \
  > "$lab_root/error.out" 2>&1
error_rc=$?
set -e
cat "$lab_root/error.out"
test "$error_rc" -eq 1
grep -Fq '[E] unreachable on boot required target' "$lab_root/error.out"
grep -Eq '0 parse errors, 1 error, 0 warnings' "$lab_root/error.out"
printf 'error_case_exit=%s errors=1 admitted_by_exit_only=no\n' "$error_rc"

Creating the directory may fix this fixture, but production repair requires more than making an error line disappear. Confirm target ownership, permissions, parent mounts and whether an application expects existing data beneath that path. Mounting over a nonempty directory hides its prior contents until unmount.

Apply the recorded admission rule

Convert the observations into the change decision only after all three fixtures have run. A clean fixture may advance to the already-defined exact-target test. Any error backs out the candidate; any warning holds the window until its named owner resolves it or records one narrow reviewed exception.

A useful automation rule rejects three states: nonzero verifier exit, any [W] or [E] detail, and absence of the exact clean summary. The summary check protects against output or behavior that does not match the tested contract.

verify_fstab_clean() {
  local table=$1 output rc
  output=$(mktemp "$lab_root/verify.XXXXXX")
  set +e
  findmnt --verify --verbose --tab-file "$table" > "$output" 2>&1
  rc=$?
  set -e
  if [[ $rc -ne 0 ]] || grep -Eq '^ +\[[EW]\] ' "$output" ||
     ! grep -Fqx 'Success, no errors or warnings detected' "$output"; then
    printf 'REJECT table=%s findmnt_exit=%s\n' "$(basename "$table")" "$rc"
    return 10
  fi
  printf 'ADMIT table=%s findmnt_exit=%s\n' "$(basename "$table")" "$rc"
}
verify_fstab_clean "$lab_root/clean.fstab" | tee "$lab_root/admit.out"
set +e
verify_fstab_clean "$lab_root/warning.fstab" | tee "$lab_root/reject-warning.out"
reject_warning_rc=${PIPESTATUS[0]}
verify_fstab_clean "$lab_root/error.fstab" | tee "$lab_root/reject-error.out"
reject_error_rc=${PIPESTATUS[0]}
set -e
test "$reject_warning_rc" -eq 10
test "$reject_error_rc" -eq 10
grep -Fqx 'ADMIT table=clean.fstab findmnt_exit=0' "$lab_root/admit.out"
grep -Fqx 'REJECT table=warning.fstab findmnt_exit=0' "$lab_root/reject-warning.out"
grep -Fqx 'REJECT table=error.fstab findmnt_exit=1' "$lab_root/reject-error.out"

This wrapper is deliberately stricter than findmnt’s exit semantics. If a team approves a specific warning, encode that exception narrowly against a known util-linux version, target and reason; do not change the general policy to ignore every warning.

Finally, record the decision, hash the portable receipt and remove only the marker-owned temporary scope. The receipt copy remains in the working directory so another operator can review it before the live change.

{
  printf 'findmnt=%s\n' "$(findmnt --version | sed -n '1s/^findmnt from util-linux //p')"
  printf 'clean=admit,exit:0,errors:0,warnings:0\n'
  printf 'warning_only=reject,exit:0,errors:0,warnings:2\n'
  printf 'required_target_error=reject,exit:1,errors:1,warnings:0\n'
  printf 'policy=exit_zero_and_clean_summary_required\n'
} | tee "$lab_root/receipt.txt" "$receipt_copy"
sha256sum "$receipt_copy"
grep -Fqx 'voxfor-fstab-173' "$marker"
find "$lab_root" -depth -mindepth 1 -delete
rmdir "$lab_root"
trap - EXIT
test ! -e "$lab_root"
printf 'cleanup_scope=%s absent=yes\n' "$lab_root"

Representative output from the complete sequence:

findmnt=2.41
kernel=6.12.96+deb13-amd64
Success, no errors or warnings detected
warning_case_exit=0 warnings=2 admitted_by_exit_only=yes
error_case_exit=1 errors=1 admitted_by_exit_only=no
ADMIT table=clean.fstab findmnt_exit=0
REJECT table=warning.fstab findmnt_exit=0
REJECT table=error.fstab findmnt_exit=1
clean=admit,exit:0,errors:0,warnings:0
warning_only=reject,exit:0,errors:0,warnings:2
required_target_error=reject,exit:1,errors:1,warnings:0
policy=exit_zero_and_clean_summary_required
cleanup_scope=/tmp/voxfor-fstab-173.rV6632 absent=yes

The reproduced admission is successful when the clean table has exit 0, no warning or error markers and the exact clean summary; the mismatched type is rejected despite exit 0; the missing required target is rejected with exit 1; the parsed clean row matches the intended source, target, type and options; and the marker-owned lab path is absent. A production receipt should add the candidate hash, reviewer, console path and an application-specific post-mount check.

Route Failures That the Table Cannot Repair

findmnt --verify is not a filesystem repair tool. Kernel I/O errors or an ext4 protective read-only remount require disk evidence and offline ext4 repair before writes are trusted. A VM image adds another container layer; qemu-img and guest-filesystem integrity must be evaluated independently.

Remote mounts add server identity and network behavior. An NFS line can parse cleanly and still return stale handles after an export or backing filesystem changes; use NFS export-identity recovery steps for that state. Likewise, a mounted filesystem whose free space appears inconsistent may involve a deleted file still held by a process, which belongs to deleted-open-file disk diagnosis rather than fstab editing.

Close the Window or Back Out the Exact Change

A successful close record contains the previous and current table hashes, the clean verifier receipt, the parsed row, the generated mount-unit check when systemd is involved, the live findmnt identity and the workload probe. Keep the window open through the first observed reboot; a clean static table is a prerequisite, not a promise of boot success.

If the candidate produces any unresolved warning, error, parsed-field mismatch or unsafe live-mount result, do not reboot. Preserve the verifier output and candidate hash, restore only the reviewed previous /etc/fstab copy with its original owner and mode, run the verifier again, reload systemd only if the live file changed, and detach only a mount created by this change after confirming no process or unsaved write still owns it. Never delete a device signature, run fsck on a mounted filesystem, add nofail blindly or reboot merely to see what happens.