Audit a systemd Journal File with journalctl --verify

A successful journalctl --verify run can establish that one systemd journal file is internally consistent at the bytes you tested. It cannot establish that every expected event was recorded, that the file came from the claimed machine, or that an online file stayed unchanged during the scan.

Use a workflow narrower than “verify the journal.” Preserve one file whose header says ARCHIVED, copy it into a private workspace, prove the copy hash, verify that copy, and make the same gate reject a deliberately damaged second copy. In a reproduced Debian 13 lab with systemd 257, the intact 16 MiB copy passed with exit 0, the disposable negative control failed with exit 1 at the modified object, and the source SHA-256 remained unchanged.

This guide is for a technical operator with shell access and permission to read persistent journals. The seven tested inputs run in one Bash session. They never stop systemd-journald, rotate or vacuum logs, write to the selected source, or delete a production journal.

Define the Claim Before You Touch the File

According to the current journalctl manual, --verify checks a journal file’s internal consistency. That is a structural claim: headers, objects, offsets, hashes, and references form a readable file according to the format understood by this systemd version.

Three different claims must stay separate:

• Internal consistency: the tested bytes satisfy journal structure checks. This is what ordinary --verify addresses.
• Authenticity: entries can be cryptographically tied to a Forward Secure Sealing history only when sealing was configured and the appropriate verification key is supplied with --verify-key=.
• Completeness: every event that should exist was actually accepted, persisted, retained, and transferred into this file.

A clean structure cannot restore an event dropped before journald received it. If audit records are part of the incident record, Linux Audit queue-loss evidence is a separate completeness check; a valid journal containing a loss message is still incomplete evidence.

Avoid a broad journalctl --verify as the first incident command. The concise Linux Audit journalctl reference shows that useful discovery form, but an audit decision needs a named, stable input and a saved return code. The first tested input creates a mode-private, marker-owned workspace and selects only a rotated journal whose own header reports ARCHIVED.

set -euo pipefail
umask 077
JOURNAL_LAB="$(mktemp -d /tmp/voxfor-journal-integrity.XXXXXX)"
JOURNAL_MARKER="$JOURNAL_LAB/.voxfor-journal-integrity-lab"
JOURNAL_RECEIPT="$(mktemp /tmp/voxfor-journal-integrity-receipt-181.XXXXXX)"
JOURNAL_SOURCE=''
install -m 0600 /dev/null "$JOURNAL_MARKER"

while IFS= read -r JOURNAL_CANDIDATE; do
  JOURNAL_CANDIDATE_STATE="$(
    journalctl --file="$JOURNAL_CANDIDATE" --header 2>/dev/null |
      awk -F': ' '/^State:/ {print $2; exit}'
  )"
  if test "$JOURNAL_CANDIDATE_STATE" = 'ARCHIVED'; then
    JOURNAL_SOURCE="$JOURNAL_CANDIDATE"
    break
  fi
done < <(
  find /var/log/journal /run/log/journal -type f -name 'system@*.journal' \
    ! -name '*.journal~' -print 2>/dev/null | sort
)

test -n "$JOURNAL_SOURCE"
test -r "$JOURNAL_SOURCE"

If no archived file is available, stop. Do not force a rotation, stop the journal service, or vacuum logs merely to complete this procedure. Acquire a stable copy under the incident’s approved preservation process instead.

Freeze an Archived Journal and Prove the Copy

Systemd’s journal file format defines file states including ONLINE, OFFLINE, and ARCHIVED. An archived file has been rotated out of active writing by journald. That makes it a better audit input than the active system.journal.

Changing inputs matter in practice. An upstream systemd live-verification report demonstrates failures that appeared while journals were being written and disappeared on a later scan. A result from a changing input cannot support the same decision as a result from a hash-pinned copy.

Input two records the source identity, creates a private intact copy, and requires byte-for-byte SHA-256 parity before analysis.

JOURNAL_INTACT="$JOURNAL_LAB/intact.journal"
JOURNAL_CORRUPT="$JOURNAL_LAB/corrupt-control.journal"
JOURNAL_SOURCE_NAME="$(basename "$JOURNAL_SOURCE")"
JOURNAL_SOURCE_BYTES="$(stat -c '%s' "$JOURNAL_SOURCE")"
JOURNAL_SOURCE_HASH_BEFORE="$(sha256sum "$JOURNAL_SOURCE" | awk '{print $1}')"

install -m 0600 -- "$JOURNAL_SOURCE" "$JOURNAL_INTACT"
JOURNAL_COPY_HASH="$(sha256sum "$JOURNAL_INTACT" | awk '{print $1}')"

test "$JOURNAL_COPY_HASH" = "$JOURNAL_SOURCE_HASH_BEFORE"
printf 'systemd_version=%s\n' "$(systemd-analyze --version | awk 'NR == 1 {print $2}')"
printf 'source_kind=archived source_bytes=%s\n' "$JOURNAL_SOURCE_BYTES"
printf 'source_sha256=%s\ncopy_sha256=%s\ncopy_matches_source=yes\n' \
  "$JOURNAL_SOURCE_HASH_BEFORE" "$JOURNAL_COPY_HASH"

Reproduced acquisition output:

systemd_version=257
source_kind=archived source_bytes=16777216
source_sha256=1a84b5ece4d1af32247b2aa49ebcada5110f3c7a7b72efa2cc0b6b76d9e478c2
copy_sha256=1a84b5ece4d1af32247b2aa49ebcada5110f3c7a7b72efa2cc0b6b76d9e478c2
copy_matches_source=yes

Published output omits the machine ID and source filename. Preserve those fields in the access-controlled incident receipt, not in a public ticket. A copied hash does not authenticate the source; it proves only that the lab began with the same bytes the operator selected.

Read the Header Before You Trust a PASS

A PASS without input context is weak evidence. Input three records the copy’s state, variable header size, object counts, time range, and independently queryable entry count. The header size will also define the negative-control offset later; do not hardcode the 272-byte value observed in this lab.

journalctl --file="$JOURNAL_INTACT" --header > "$JOURNAL_LAB/header.txt"
JOURNAL_STATE="$(awk -F': ' '/^State:/ {print $2; exit}' "$JOURNAL_LAB/header.txt")"
JOURNAL_HEADER_BYTES="$(
  awk -F': ' '/^Header size:/ {print $2; exit}' "$JOURNAL_LAB/header.txt"
)"
JOURNAL_ENTRY_OBJECTS="$(
  awk -F': ' '/^Entry objects:/ {print $2; exit}' "$JOURNAL_LAB/header.txt"
)"
JOURNAL_QUERYABLE_ENTRIES="$(
  journalctl --file="$JOURNAL_INTACT" --no-pager -o json | wc -l
)"

test "$JOURNAL_STATE" = 'ARCHIVED'
test "$JOURNAL_HEADER_BYTES" -gt 0
test "$JOURNAL_HEADER_BYTES" -lt "$JOURNAL_SOURCE_BYTES"
test "$JOURNAL_QUERYABLE_ENTRIES" = "$JOURNAL_ENTRY_OBJECTS"
grep -E '^(State|Header size|Objects|Entry objects|Head realtime timestamp|Tail realtime timestamp):' \
  "$JOURNAL_LAB/header.txt"
printf 'queryable_entry_count=%s\n' "$JOURNAL_QUERYABLE_ENTRIES"

Treat the first and last timestamps as scope, not proof of clock accuracy. If the range is implausible, inspect chrony clock-drift evidence before correlating this journal with records from another host.

Count equality is a useful lab assertion, not a universal promise for every damaged journal. If journalctl -o json exits nonzero or produces a different count, retain both results. Do not discard the file because one index path is unreadable.

Require Both an Intact PASS and a Damaged FAIL

Input four runs the actual integrity check on the hash-pinned copy, captures diagnostic output and return code, and stops the workflow unless the intact file passes.

set +e
JOURNAL_INTACT_VERIFY="$(
  SYSTEMD_LOG_LEVEL=debug journalctl --file="$JOURNAL_INTACT" --verify 2>&1
)"
JOURNAL_INTACT_RC=$?
set -e

printf '%s\n' "$JOURNAL_INTACT_VERIFY" |
  grep -E '^(PASS|FAIL|File corruption detected)' |
  sed "s|$JOURNAL_INTACT|intact.journal|g" > "$JOURNAL_LAB/intact-verify.txt"
printf 'intact_verify_rc=%s\n' "$JOURNAL_INTACT_RC" >> "$JOURNAL_LAB/intact-verify.txt"
test "$JOURNAL_INTACT_RC" -eq 0
cat "$JOURNAL_LAB/intact-verify.txt"

Representative intact result:

PASS: intact.journal
intact_verify_rc=0

A passing tool path is not enough. The gate must also recognize known-bad input. According to the format specification, the first object begins immediately after the variable-sized header. Input five copies the intact lab file again and changes one byte at that derived boundary. It never writes to either the source or the intact copy.

cp -- "$JOURNAL_INTACT" "$JOURNAL_CORRUPT"
printf '\377' | dd of="$JOURNAL_CORRUPT" bs=1 seek="$JOURNAL_HEADER_BYTES" \
  count=1 conv=notrunc status=none
printf 'negative_control_offset=%s negative_control_scope=disposable-copy-only\n' \
  "$JOURNAL_HEADER_BYTES"

set +e
JOURNAL_CORRUPT_VERIFY="$(
  SYSTEMD_LOG_LEVEL=debug journalctl --file="$JOURNAL_CORRUPT" --verify 2>&1
)"
JOURNAL_CORRUPT_RC=$?
set -e

printf '%s\n' "$JOURNAL_CORRUPT_VERIFY" |
  grep -E '^(Attempt to move|[0-9a-f]+: Invalid object|File corruption detected|FAIL:)' |
  sed "s|$JOURNAL_CORRUPT|corrupt-control.journal|g" \
  > "$JOURNAL_LAB/corrupt-verify.txt"
printf 'corrupt_verify_rc=%s\n' "$JOURNAL_CORRUPT_RC" >> "$JOURNAL_LAB/corrupt-verify.txt"
test "$JOURNAL_CORRUPT_RC" -ne 0
grep -Fq 'FAIL: corrupt-control.journal' "$JOURNAL_LAB/corrupt-verify.txt"
cat "$JOURNAL_LAB/corrupt-verify.txt"

Expected negative-control rejection:

negative_control_offset=272 negative_control_scope=disposable-copy-only
Attempt to move to object with invalid type (255): 272
000110: Invalid object: Bad message
File corruption detected at corrupt-control.journal:272 (of 16777216 bytes, 0%).
FAIL: corrupt-control.journal (Bad message)
corrupt_verify_rc=1

Exact offset and error detail depend on the file and systemd version. What matters is the decision pair: the unchanged copy returns 0 and the deliberately damaged copy returns nonzero with a corruption finding. A control that unexpectedly passes invalidates the gate; it does not rehabilitate the damaged file.

Accept the integrity receipt only when the selected source was readable and ARCHIVED, source and intact-copy SHA-256 values matched, the header and query path described the expected evidence scope, the intact copy returned 0 with PASS, the disposable changed copy returned nonzero with FAIL, and a final source hash still matched the pre-test value. Escalate any missing condition instead of converting it into a warning.

Turn Each Result Into a Bounded Operator Decision

Input six re-hashes the source after both checks and writes a compact receipt outside the disposable directory. It records claims and limitations together so PASS cannot later be quoted without its boundary.

JOURNAL_SOURCE_HASH_AFTER="$(sha256sum "$JOURNAL_SOURCE" | awk '{print $1}')"
test "$JOURNAL_SOURCE_HASH_AFTER" = "$JOURNAL_SOURCE_HASH_BEFORE"

{
  printf 'systemd_version=%s\n' "$(systemd-analyze --version | awk 'NR == 1 {print $2}')"
  printf 'source_name=%s\nsource_state=%s\nsource_bytes=%s\n' \
    "$JOURNAL_SOURCE_NAME" "$JOURNAL_STATE" "$JOURNAL_SOURCE_BYTES"
  printf 'source_sha256_before=%s\nsource_sha256_after=%s\nsource_unchanged=yes\n' \
    "$JOURNAL_SOURCE_HASH_BEFORE" "$JOURNAL_SOURCE_HASH_AFTER"
  printf 'copy_sha256=%s\nentry_objects=%s\nqueryable_entries=%s\n' \
    "$JOURNAL_COPY_HASH" "$JOURNAL_ENTRY_OBJECTS" "$JOURNAL_QUERYABLE_ENTRIES"
  printf 'intact_verify_rc=%s\ncorrupt_control_rc=%s\n' \
    "$JOURNAL_INTACT_RC" "$JOURNAL_CORRUPT_RC"
  printf 'claim=internal-consistency-at-tested-hash\n'
  printf 'not_proved=origin,completeness,ordinary-pass-authenticity\n'
} > "$JOURNAL_RECEIPT"

grep -Fq 'source_unchanged=yes' "$JOURNAL_RECEIPT"
grep -Fq 'intact_verify_rc=0' "$JOURNAL_RECEIPT"
grep -Fq 'corrupt_control_rc=1' "$JOURNAL_RECEIPT"

Use the result matrix as a response boundary:

A Bad message failure identifies a structural read problem, not its root cause. Disk pressure can contribute to interrupted writes; compare df, du, and deleted-open-file evidence with Linux disk-space discrepancy diagnosis rather than deleting logs to make room. Repeated journal damage alongside I/O or mount errors moves the decision toward ext4 read-only filesystem evidence and an offline storage plan.

Do not assume journalctl repairs the file. The tool provides verification and reading paths, not an in-place --repair operation. The public portion of a Red Hat journal corruption case shows the same PASS, FAIL, offset, and Bad message vocabulary on older systemd, but version-specific recovery still needs the operating system vendor’s current procedure and an evidence-preservation decision.

Remove Only the Lab and Retain the Receipt

Input seven deletes the two private copies and diagnostic files only after validating the ownership marker and expected path. It leaves the source and receipt in place.

case "$JOURNAL_LAB" in
  /tmp/voxfor-journal-integrity.*) ;;
  *) printf '%s\n' 'refusing cleanup outside the expected lab path' >&2; exit 1 ;;
esac
test -f "$JOURNAL_MARKER"
test "$(stat -c '%a' "$JOURNAL_MARKER")" = '600'

rm -f -- "$JOURNAL_INTACT" "$JOURNAL_CORRUPT" \
  "$JOURNAL_LAB/header.txt" "$JOURNAL_LAB/intact-verify.txt" \
  "$JOURNAL_LAB/corrupt-verify.txt" "$JOURNAL_MARKER"
rmdir -- "$JOURNAL_LAB"
test ! -e "$JOURNAL_LAB"
test -s "$JOURNAL_RECEIPT"
printf 'cleanup=marker-owned-lab-removed receipt_retained=yes\n'

If acquisition, verification, or the negative control fails, make no change to the source: retain the original path, before/after hashes, copy, raw output, return codes, systemd version, and incident owner; remove only a marker-owned disposable control after the evidence has been secured. Do not stop journald, rotate, vacuum, truncate, replace, or delete journals as an automatic rollback.

Keep the receipt with the preserved file or evidence manifest, not inside the temporary directory. Record the host and boot identity, source path, file size, SHA-256, header state and time range, systemd version, exact command, return codes, control result, operator, UTC timestamp, and storage location. If long-term monitoring is needed, use an AIDE trusted-baseline workflow for stable system files; an actively changing journal is not a sensible static baseline target.

That record supports one precise decision: these preserved bytes passed or failed systemd’s structural verification at a known time. Keeping the claim that narrow is what makes the result useful during the next storage, authenticity, or incident-timeline review.