Repair a Truncated Redis AOF and Prove What Survived

Redis refusing an append-only file is first a forensic boundary, not permission to run --fix against the only copy. Preserve the complete persistence set, identify whether the damage is an incomplete final command or corruption in the middle, and decide what loss is acceptable before changing a byte.

For a genuinely truncated tail, the recovery path is bounded: validate the incremental AOF, retain a checksummed incident copy, repair a working copy at the last valid RESP command, start Redis in isolation, and compare named keys and values with an application receipt. A successful PING proves availability; it does not prove that the right state survived.

This guide is for an operator of a self-managed Redis instance who can use a Linux shell and schedule a maintenance window. The reproduced lab used Debian 13.6, Redis and redis-check-aof 8.0.2, and a loopback-only disposable instance. Redis 7 and later use a multipart AOF: a manifest tracks one base file and one or more incremental files inside an AOF directory. That detail is why legacy advice pointing only to /var/lib/redis/appendonly.aof can select the wrong artifact.

Read the Failure Before You Touch the AOF

Redis’s current persistence documentation separates two cases. An incomplete command at the end can follow a crash, full volume, or interrupted write. Current Redis releases normally tolerate that condition when aof-load-truncated yes is active, discard the malformed tail, and log the offset. Setting the option to no instead makes startup stop for inspection, as the lab does deliberately.

Mid-file corruption has a different loss boundary. Repairing at the first invalid byte may discard every valid command after that point, not merely one in-flight tail. Stop when the checker reports damage away from the end, the manifest is inconsistent, the base file fails validation, or you cannot explain which acknowledged writes the removed range contains. Prefer a healthy replica or a known backup in those cases.

Other startup errors need their own branch. A checksum failure in dump.rdb, an out-of-memory abort, permissions, a missing mount, or a full filesystem is not proven AOF truncation. OneUptime’s broader corruption workflow provides useful log and storage branches, but the literal local log and checker result still own the classification. If the kernel killed Redis during load, trace Linux OOM ownership before editing persistence files.

Build a Current Multipart-AOF Rehearsal

Never manufacture corruption on production. The first input creates an exact marker-guarded lab, writes a Redis 8 configuration, and binds only to 127.0.0.1:6396. It disables snapshots, uses appendfsync always so the committed fixture has a clear lab boundary, and sets aof-load-truncated no to make the negative startup observable.

set -Eeuo pipefail
LAB=/tmp/voxfor-redis-aof-recovery-136
PORT=6396
MARKER="$LAB/.voxfor-redis-aof-recovery-136"
[[ ! -e "$LAB" ]]
[[ -z "$(ss -H -ltn "sport = :$PORT")" ]]
install -d -m 700 "$LAB/data"
: > "$MARKER"
cat > "$LAB/redis.conf" <<EOF
bind 127.0.0.1
protected-mode yes
port $PORT
daemonize yes
pidfile $LAB/redis.pid
logfile $LAB/redis.log
dir $LAB/data
dbfilename dump.rdb
save ""
appendonly yes
appenddirname appendonlydir
appendfilename appendonly.aof
appendfsync always
aof-use-rdb-preamble yes
aof-load-truncated no
auto-aof-rewrite-percentage 0
EOF
redis-server "$LAB/redis.conf"
READY=0
for _ in {1..50}; do
  [[ -s "$LAB/redis.pid" ]] && redis-cli -p "$PORT" PING 2>/dev/null | grep -qx PONG \
    && { READY=1; break; }
  sleep 0.1
done
[[ "$READY" == 1 ]]
REDIS_PID=$(<"$LAB/redis.pid")
kill -0 "$REDIS_PID"
readlink -f /proc/"$REDIS_PID"/fd/* 2>/dev/null | grep -Fq "$LAB/data/"
ss -ltnp "sport = :$PORT" | grep -q "pid=$REDIS_PID,"
redis-cli -p "$PORT" PING | grep -qx PONG
printf '%s\n' "$REDIS_PID" > "$LAB/initial.pid"

Baseline first: write two application-like strings and a recovery receipt. After BGREWRITEAOF, the block waits for the rewrite to finish successfully, stops Redis cleanly, and selects the current incremental file from the manifest by its type i field. File names are discovered rather than assumed.

set -Eeuo pipefail
LAB=/tmp/voxfor-redis-aof-recovery-136
PORT=6396
[[ -f "$LAB/.voxfor-redis-aof-recovery-136" ]]
REDIS_PID=$(<"$LAB/redis.pid")
kill -0 "$REDIS_PID"
readlink -f /proc/"$REDIS_PID"/fd/* 2>/dev/null | grep -Fq "$LAB/data/"
ss -ltnp "sport = :$PORT" | grep -q "pid=$REDIS_PID,"
redis-cli -p "$PORT" MSET order:1001 paid order:1002 queued >/dev/null
redis-cli -p "$PORT" HSET recovery:receipt expected_keys 3 fixture redis-aof-136 >/dev/null
redis-cli -p "$PORT" BGREWRITEAOF >/dev/null
REWRITE_OK=0
for _ in {1..100}; do
  INFO=$(redis-cli -p "$PORT" INFO persistence | tr -d '\r')
  grep -q '^aof_rewrite_in_progress:0$' <<<"$INFO" \
    && grep -q '^aof_last_bgrewrite_status:ok$' <<<"$INFO" && { REWRITE_OK=1; break; }
  sleep 0.1
done
[[ "$REWRITE_OK" == 1 ]]
redis-cli -p "$PORT" SHUTDOWN NOSAVE
STOPPED=0
for _ in {1..50}; do
  kill -0 "$REDIS_PID" 2>/dev/null || { STOPPED=1; break; }
  sleep 0.1
done
[[ "$STOPPED" == 1 ]]
[[ -z "$(ss -H -ltn "sport = :$PORT")" ]]
AOF_DIR="$LAB/data/appendonlydir"
MANIFEST="$AOF_DIR/appendonly.aof.manifest"
INCR=$(awk '$5=="type" && $6=="i" {print $2}' "$MANIFEST" | tail -n1)
[[ -n "$INCR" && -f "$AOF_DIR/$INCR" ]]
cp -a "$AOF_DIR" "$LAB/committed-aof"
printf 'manifest=%s incremental=%s\n' "$MANIFEST" "$INCR"

AOF rewrite health and crash recovery answer different questions. The rewrite can complete perfectly, then a later append can be torn. If production pauses coincide with rewrites, measure the separate Redis fork and copy-on-write window rather than treating a repair as a performance fix.

Preserve and Classify the Damaged Tail

To create the negative control, append one deliberately incomplete RESP SET command to the disposable incremental file. The third input then copies the complete multipart directory as incident evidence and records the exact damaged file’s SHA-256 before running the checker without --fix.

set -Eeuo pipefail
LAB=/tmp/voxfor-redis-aof-recovery-136
[[ -f "$LAB/.voxfor-redis-aof-recovery-136" ]]
AOF_DIR="$LAB/data/appendonlydir"
MANIFEST="$AOF_DIR/appendonly.aof.manifest"
INCR=$(awk '$5=="type" && $6=="i" {print $2}' "$MANIFEST" | tail -n1)
printf '*3\r\n$3\r\nSET\r\n$12\r\norder:broken\r\n$7\r\npartial' >> "$AOF_DIR/$INCR"
cp -a "$AOF_DIR" "$LAB/corrupt-evidence"
find "$LAB/corrupt-evidence" -type f -print0 | sort -z \
  | xargs -0 sha256sum > "$LAB/corrupt-evidence.sha256"
[[ "$(wc -l < "$LAB/corrupt-evidence.sha256")" -ge 3 ]]
set +e
redis-check-aof "$AOF_DIR/$INCR" > "$LAB/check-before.txt" 2>&1
CHECK_RC=$?
set -e
[[ "$CHECK_RC" -ne 0 ]]
grep -Eq 'Expected|not valid' "$LAB/check-before.txt"
printf 'checker_rc=%s damaged_bytes=%s\n' "$CHECK_RC" "$(stat -c %s "$AOF_DIR/$INCR")"

In a real incident, copy the entire AOF directory, not only the file named in the last log line. The base, incremental files, and manifest form one set. Store the copy on a volume with enough free space, make it read-only for reviewers, and record hashes. The evidence-preservation pattern in AIDE trusted-baseline change control likewise helps prove that later analysis still refers to the original bytes.

OneUptime’s repair walkthrough shows the familiar checker and backup sequence. For Redis 7+, add manifest-aware discovery and retain the complete directory so a plausible file name never substitutes for evidence.

Make the Negative Startup Observable

Daemonized redis-server may return zero to the launching shell before the child fails while loading persistence. Therefore, the command’s immediate exit code is not the acceptance test. The lab waits and proves that PING remains unavailable while the log contains the expected incomplete-AOF message.

set -Eeuo pipefail
LAB=/tmp/voxfor-redis-aof-recovery-136
PORT=6396
[[ -f "$LAB/.voxfor-redis-aof-recovery-136" ]]
[[ -z "$(ss -H -ltn "sport = :$PORT")" ]]
set +e
redis-server "$LAB/redis.conf" > "$LAB/start-before.txt" 2>&1
START_COMMAND_RC=$?
sleep 0.3
redis-cli -p "$PORT" PING >/dev/null 2>&1
PING_RC=$?
set -e
[[ "$PING_RC" -ne 0 ]]
grep -Eq 'Unexpected end of file|Bad file format|short read' "$LAB/redis.log"
printf 'daemon_command_rc=%s ping_rc=%s service=unavailable\n' "$START_COMMAND_RC" "$PING_RC"

That distinction matters in automation: systemd can report a failed service even though a wrapper command looked successful, while an orchestrator can loop endlessly unless the readiness check owns the verdict. Capture the log, the unit or container exit state, the resolved dir, appenddirname, manifest content, Redis version, filesystem free space, and the last known application checkpoint before deciding on repair.

Repair Only the Tail You Understand

In the reproduced run, the checker found a 43-byte incremental file with ok_up_to=0: the entire file was the deliberately partial command, while all three committed keys lived in the valid base AOF. Repair therefore removes exactly that invalid tail. A production result with a large diff, an unexpected offset, or valid commands following the defect requires review, not an automatic yes.

set -Eeuo pipefail
LAB=/tmp/voxfor-redis-aof-recovery-136
[[ -f "$LAB/.voxfor-redis-aof-recovery-136" ]]
AOF_DIR="$LAB/data/appendonlydir"
MANIFEST="$AOF_DIR/appendonly.aof.manifest"
INCR=$(awk '$5=="type" && $6=="i" {print $2}' "$MANIFEST" | tail -n1)
[[ -d "$LAB/corrupt-evidence" && -f "$LAB/corrupt-evidence.sha256" ]]
printf 'y\n' | redis-check-aof --fix "$AOF_DIR/$INCR" > "$LAB/fix.txt" 2>&1
redis-check-aof "$AOF_DIR/$INCR" > "$LAB/check-after.txt" 2>&1
grep -Eq 'Successfully truncated' "$LAB/fix.txt"
grep -Eq 'AOF .* is empty|AOF is valid|diff=0' "$LAB/check-after.txt"
sha256sum -c "$LAB/corrupt-evidence.sha256" >/dev/null
printf 'repair=validated corrupt_copy=unchanged\n'

A broader multipart AOF incident runbook correctly treats disk-full, OOM, RDB failure, and AOF syntax as separate branches. Follow that discipline. Do not disable AOF merely to make Redis start; that can load an older RDB and turn an explicit recovery choice into silent state regression.

Automatic tail truncation is an availability policy, not evidence that no acknowledged write was lost. Redis documents that appendfsync everysec can lose roughly the latest second during a disaster, while always has different latency and durability tradeoffs. Compare the recovered state with the application’s own durable source, queue, transaction log, or receipt instead of promising a universal byte-loss bound.

Accept State, Then Restart It Again

On the first recovery start, check availability, exact values, the recovery hash, the absence of the partial key, total key count, and AOF write status. Replace these fixtures with application-specific invariants: last processed order ID, ledger balance, queue ownership, session count, or a checksum of critical records.

set -Eeuo pipefail
LAB=/tmp/voxfor-redis-aof-recovery-136
PORT=6396
[[ -f "$LAB/.voxfor-redis-aof-recovery-136" ]]
[[ -z "$(ss -H -ltn "sport = :$PORT")" ]]
redis-server "$LAB/redis.conf"
READY=0
for _ in {1..50}; do
  [[ -s "$LAB/redis.pid" ]] && redis-cli -p "$PORT" PING 2>/dev/null | grep -qx PONG \
    && { READY=1; break; }
  sleep 0.1
done
[[ "$READY" == 1 ]]
REDIS_PID=$(<"$LAB/redis.pid")
[[ "$REDIS_PID" != "$(<"$LAB/initial.pid")" ]]
kill -0 "$REDIS_PID"
readlink -f /proc/"$REDIS_PID"/fd/* 2>/dev/null | grep -Fq "$LAB/data/"
ss -ltnp "sport = :$PORT" | grep -q "pid=$REDIS_PID,"
printf '%s\n' "$REDIS_PID" > "$LAB/recovery-first.pid"
[[ "$(redis-cli -p "$PORT" GET order:1001)" == paid ]]
[[ "$(redis-cli -p "$PORT" GET order:1002)" == queued ]]
[[ "$(redis-cli -p "$PORT" HGET recovery:receipt fixture)" == redis-aof-136 ]]
[[ "$(redis-cli -p "$PORT" EXISTS order:broken)" == 0 ]]
[[ "$(redis-cli -p "$PORT" DBSIZE)" == 3 ]]
redis-cli -p "$PORT" INFO persistence | tr -d '\r' | grep -q '^aof_last_write_status:ok$'
printf 'state=accepted dbsize=3 partial_key=absent\n'

DBSIZE=3 alone would be a weak green signal: three wrong keys also satisfy it. The guide on verifying restored data uses the same acceptance principle—validate identities and contents, not just file presence or process status.

One more restart proves that the repaired set remains loadable after Redis opens and closes the current incremental file. It also catches a repair procedure that works only in a temporary process state.

set -Eeuo pipefail
LAB=/tmp/voxfor-redis-aof-recovery-136
PORT=6396
[[ -f "$LAB/.voxfor-redis-aof-recovery-136" ]]
FIRST_RECOVERY_PID=$(<"$LAB/recovery-first.pid")
kill -0 "$FIRST_RECOVERY_PID"
readlink -f /proc/"$FIRST_RECOVERY_PID"/fd/* 2>/dev/null | grep -Fq "$LAB/data/"
ss -ltnp "sport = :$PORT" | grep -q "pid=$FIRST_RECOVERY_PID,"
redis-cli -p "$PORT" SHUTDOWN NOSAVE
STOPPED=0
for _ in {1..50}; do
  kill -0 "$FIRST_RECOVERY_PID" 2>/dev/null || { STOPPED=1; break; }
  sleep 0.1
done
[[ "$STOPPED" == 1 ]]
[[ -z "$(ss -H -ltn "sport = :$PORT")" ]]
redis-server "$LAB/redis.conf"
READY=0
for _ in {1..50}; do
  [[ -s "$LAB/redis.pid" ]] && redis-cli -p "$PORT" PING 2>/dev/null | grep -qx PONG \
    && { READY=1; break; }
  sleep 0.1
done
[[ "$READY" == 1 ]]
SECOND_RECOVERY_PID=$(<"$LAB/redis.pid")
[[ "$SECOND_RECOVERY_PID" != "$FIRST_RECOVERY_PID" ]]
kill -0 "$SECOND_RECOVERY_PID"
readlink -f /proc/"$SECOND_RECOVERY_PID"/fd/* 2>/dev/null | grep -Fq "$LAB/data/"
ss -ltnp "sport = :$PORT" | grep -q "pid=$SECOND_RECOVERY_PID,"
printf '%s\n' "$SECOND_RECOVERY_PID" > "$LAB/recovery-second.pid"
[[ "$(redis-cli -p "$PORT" MGET order:1001 order:1002 | paste -sd, -)" == paid,queued ]]
[[ "$(redis-cli -p "$PORT" DBSIZE)" == 3 ]]
printf 'second_restart=accepted values=paid,queued dbsize=3\n'

Here is the representative receipt from the complete lab:

Redis: 8.0.2
redis-check-aof before: rc=1, invalid truncated tail detected
fail-closed restart: daemon command rc=0, ping rc=1, aof-load-truncated=no refused service availability
repair: Successfully truncated the 43-byte invalid incremental AOF tail
recovered keys: order:1001=paid, order:1002=queued, recovery:receipt.fixture=redis-aof-136
discarded partial key: order:broken exists=0
dbsize after first and second restart: 3
corrupt multipart evidence: manifest, base, and incremental file checksums preserved and verified
cleanup: exact marked lab removed; port 6396 is no longer listening

Recovery is ready for a controlled production return only when the original multipart AOF set remains preserved and checksummed, the checker identifies an understood tail boundary, the repaired copy validates, Redis becomes ready in isolation, every named application invariant matches its external receipt, the discarded partial write is reconciled, persistence status is healthy, and a second restart reproduces the same state.

Move From the Lab to a Production Decision

Stop client writes before copying a production persistence set, or obtain a crash-consistent volume snapshot. Record ownership and permissions, then rehearse on another port, container, VM, or isolated host. The Prequel recovery reference likewise puts backup before repair and isolated validation before service return. Do not let the recovery instance join Sentinel, Cluster, a load balancer, or a client service-discovery path until acceptance passes.

If a replica is known healthy and its replication offset covers the incident, promoting or rebuilding from it can preserve more data than truncating a damaged primary. Likewise, a recent verified backup can be safer than accepting a large checker diff. Engine migration is a different change; keep incident recovery separate from Redis-to-Valkey cutover planning unless the rollback and single-writer boundary have been designed separately.

Disk-full symptoms can persist after files are moved when a deleted file remains open. If free space does not return, inspect deleted open files on Linux before forcing another persistence rewrite.

Cleanup remains scoped: stop the exact lab instance, verify the marker and path, remove only the disposable tree, and confirm the port is no longer listening.

set -Eeuo pipefail
LAB=/tmp/voxfor-redis-aof-recovery-136
PORT=6396
[[ "$LAB" == /tmp/voxfor-redis-aof-recovery-136 ]]
[[ -f "$LAB/.voxfor-redis-aof-recovery-136" ]]
REDIS_PID=$(<"$LAB/recovery-second.pid")
kill -0 "$REDIS_PID"
readlink -f /proc/"$REDIS_PID"/fd/* 2>/dev/null | grep -Fq "$LAB/data/"
ss -ltnp "sport = :$PORT" | grep -q "pid=$REDIS_PID,"
redis-cli -p "$PORT" SHUTDOWN NOSAVE
STOPPED=0
for _ in {1..50}; do
  kill -0 "$REDIS_PID" 2>/dev/null || { STOPPED=1; break; }
  sleep 0.1
done
[[ "$STOPPED" == 1 ]]
[[ -z "$(ss -H -ltn "sport = :$PORT")" ]]
rm -rf -- "$LAB"
[[ ! -e "$LAB" ]]
[[ -z "$(ss -H -ltn "sport = :$PORT")" ]]
printf 'cleanup=verified path_absent=%s port=%s_closed\n' "$LAB" "$PORT"

If validation, repair, startup, or state acceptance fails, keep the original service offline and leave the checksummed incident copy unchanged. Stop the isolated recovery process, discard only the marked working copy, and return to a healthy replica or verified backup. Never repeat --fix, disable AOF, or promote the partial state merely to obtain a green health check.