Which sysctl.d File Wins? Check Before You Apply It

A sysctl.d override is not decided by directory priority alone. The loader first chooses one file when the same filename exists in several supported directories. It then sorts every surviving .conf file lexically and applies assignments in that order. The last surviving assignment for a key wins.

That two-stage rule produced three unambiguous results in the reproduced Debian 13.6 lab: vm.swappiness=10 came from 90-local.conf, kernel.pid_max=200000 came from the /etc copy of 50-shared.conf, and fs.file-max=100000 remained from 40-vendor.conf. An intentionally conflicting /etc/sysctl.conf value did not enter the systemd source set. All three running kernel values stayed unchanged.

This article does not recommend those fixture values. It is to answer a safer operational question: which exact source line would win before anything writes to the live kernel? The workflow below builds a marker-owned alternate root, asks systemd to expose its merged source set, reduces that set to final values, checks parser acceptance with procps dry-run mode, and deletes only the owned lab path.

Two Rules Decide Which Assignment Wins

Current systemd sysctl.d documentation describes both rules, but they are easy to collapse into the inaccurate slogan “/etc wins.” Directory priority applies only when files have the same filename. In this lab, /etc/sysctl.d/50-shared.conf suppresses /usr/lib/sysctl.d/50-shared.conf as a whole. The lower-priority file contributes neither kernel.pid_max=100000 nor its unrelated vm.overcommit_memory=0 line.

Filename order is the next decision. The surviving 40-vendor.conf, 50-shared.conf, 70-runtime.conf, and 90-local.conf files are sorted together, regardless of their directories. A later assignment to the same key replaces an earlier assignment. Thus 90-local.conf supplies the final swappiness value even though 70-runtime.conf lives in /run.

Upstream discussion in systemd issue 12791 isolates this exact source of confusion: same-name files compete by directory priority, while different filenames compete through lexical application order. Treat those as separate questions in every review:

1. Which file paths survive same-name shadowing?
2. In what lexical order are the survivors parsed?
3. Which surviving line last assigns each key?

This is analogous to, but not interchangeable with, SSH effective-configuration preflight: both workflows inspect the resolved result instead of trusting the file an operator remembers editing. Each loader still has its own search and precedence rules.

Build an Alternate Root, Not a Live Change

Run the tested inputs in one Bash shell on a disposable Linux host. They create only /tmp/voxfor-sysctl-precedence-193, never write to live /etc, /run, or /usr/lib, and never invoke sysctl without --dry-run. The cleanup function refuses to remove the directory unless both its literal path and ownership marker match.

Begin with an input that admits the environment, prepares the alternate root, records tool versions, and snapshots the three live values that later must remain identical.

set -Eeuo pipefail

LAB=/tmp/voxfor-sysctl-precedence-193
ROOT="$LAB/root"
MARKER="$LAB/.marker"
MARKER_VALUE=voxfor-sysctl-precedence-193

fail() { printf 'ERROR: %s\n' "$*" >&2; exit 1; }
owned_cleanup() {
  if [[ -d "$LAB" ]]; then
    [[ "$LAB" == /tmp/voxfor-sysctl-precedence-193 ]] ||
      fail "unexpected lab path"
    [[ -f "$MARKER" ]] ||
      fail "refusing cleanup without marker"
    [[ "$(<"$MARKER")" == "$MARKER_VALUE" ]] ||
      fail "marker mismatch"
    rm -rf -- "$LAB"
  fi
}
trap 'rc=$?; owned_cleanup; exit $rc' EXIT

for tool in systemd-analyze sysctl awk grep sha256sum \
            find sort xargs sed cut paste tr; do
  command -v "$tool" >/dev/null || fail "missing $tool"
done
[[ ! -e "$LAB" ]] || fail "lab path already exists"

install -d -m 0700 "$LAB"
install -d -m 0755 \
  "$ROOT/usr/lib/sysctl.d" \
  "$ROOT/run/sysctl.d" \
  "$ROOT/etc/sysctl.d"
printf '%s\n' "$MARKER_VALUE" > "$MARKER"

SYSTEMD_VERSION=$(systemd-analyze --version | head -n 1)
SYSCTL_VERSION=$(sysctl --version | head -n 1)
printf 'environment=os=%s kernel=%s systemd=%s procps=%s\n' \
  "$(. /etc/os-release; printf '%s_%s' "$ID" "$VERSION_ID")" \
  "$(uname -r)" \
  "${SYSTEMD_VERSION// /_}" \
  "${SYSCTL_VERSION// /_}"

BEFORE_SWAPPINESS=$(sysctl -n vm.swappiness)
BEFORE_PID_MAX=$(sysctl -n kernel.pid_max)
BEFORE_FILE_MAX=$(sysctl -n fs.file-max)

This containment is an inspection technique, not a production loader simulation. It proves how the installed systemd tooling resolves files inside the supplied root. It does not prove that a future boot will load a module in time, that another service will not write the same key later, or that these values are suitable for the workload.

ArchWiki’s current Sysctl operations guide explains the practical move from one-off values to persistent configuration. The alternate-root step adds a narrower safety gate: inspect exactly which persistent sources survive before choosing any real load command.

Persistent-configuration reviews need the same stop/go discipline as fstab pre-reboot verification workflow: retain the known sources, verify the proposed resolution, and define the backout boundary before a disruptive transition.

Prove Whole-File Shadowing Before Value Order

Create seven deliberately conflicting fixtures. The /usr/lib and /etc copies of 50-shared.conf test same-name selection. The 40, 70, and 90 filenames test lexical order. 99-ignored.txt checks that a tempting later value does not enter the .conf set. /etc/sysctl.conf establishes the loader-scope control used later.

printf '%s\n' \
  'vm.swappiness = 40' \
  'fs.file-max = 100000' \
  > "$ROOT/usr/lib/sysctl.d/40-vendor.conf"

printf '%s\n' \
  'kernel.pid_max = 100000' \
  'vm.overcommit_memory = 0' \
  > "$ROOT/usr/lib/sysctl.d/50-shared.conf"

printf '%s\n' 'vm.swappiness = 25' \
  > "$ROOT/run/sysctl.d/70-runtime.conf"
printf '%s\n' 'kernel.pid_max = 200000' \
  > "$ROOT/etc/sysctl.d/50-shared.conf"
printf '%s\n' 'vm.swappiness = 10' \
  > "$ROOT/etc/sysctl.d/90-local.conf"
printf '%s\n' 'vm.swappiness = 1' \
  > "$ROOT/etc/sysctl.d/99-ignored.txt"
printf '%s\n' 'vm.swappiness = 7' \
  > "$ROOT/etc/sysctl.conf"

find "$ROOT" -type f -print0 |
  sort -z |
  xargs -0 sha256sum > "$LAB/fixture-hashes.txt"
[[ $(wc -l < "$LAB/fixture-hashes.txt") -eq 7 ]] ||
  fail "unexpected fixture count"
FIXTURE_MANIFEST=$(sha256sum "$LAB/fixture-hashes.txt" | awk '{print $1}')
printf 'fixture_count=%s fixture_manifest_sha256=%s\n' \
  "$(wc -l < "$LAB/fixture-hashes.txt")" \
  "$FIXTURE_MANIFEST"

A .txt file is not merely a weak override; it is outside the sysctl.d filename contract. The same distinction appears in Linux ACL effective-permissions proof: a stored entry and an effective result are different facts. Here, existence on disk and membership in the loader’s effective source set are different facts.

Ask systemd to enumerate that set against the alternate root. --tldr keeps comments and empty lines out of the evidence, while the saved full path allows assertions against exact sources.

systemd-analyze \
  --root="$ROOT" \
  --tldr \
  cat-config sysctl.d \
  > "$LAB/systemd-cat-config.txt"

sed "s#$ROOT#\$ROOT#g" \
  "$LAB/systemd-cat-config.txt"

Source tracing is useful only if failure becomes executable. Assert every decision instead of visually scanning the output: the local same-name file must be present, the vendor same-name file must be absent, the wrong extension and /etc/sysctl.conf must be absent, and the four selected filenames must appear in the expected order.

grep -Fq "$ROOT/etc/sysctl.d/50-shared.conf" \
  "$LAB/systemd-cat-config.txt" ||
  fail "local same-name file missing"

! grep -Fq "$ROOT/usr/lib/sysctl.d/50-shared.conf" \
  "$LAB/systemd-cat-config.txt" ||
  fail "lower-priority same-name file survived"

! grep -Fq '99-ignored.txt' "$LAB/systemd-cat-config.txt" ||
  fail "non-conf file entered source set"
! grep -Fq "$ROOT/etc/sysctl.conf" "$LAB/systemd-cat-config.txt" ||
  fail "/etc/sysctl.conf entered systemd source set"

LINE_40=$(grep -nF '40-vendor.conf' "$LAB/systemd-cat-config.txt" | cut -d: -f1)
LINE_50=$(grep -nF '50-shared.conf' "$LAB/systemd-cat-config.txt" | cut -d: -f1)
LINE_70=$(grep -nF '70-runtime.conf' "$LAB/systemd-cat-config.txt" | cut -d: -f1)
LINE_90=$(grep -nF '90-local.conf' "$LAB/systemd-cat-config.txt" | cut -d: -f1)

[[ $LINE_40 -lt $LINE_50 &&
   $LINE_50 -lt $LINE_70 &&
   $LINE_70 -lt $LINE_90 ]] ||
  fail "lexical order assertion failed"
printf 'selection_assertion=same_name_etc_shadows_usr ignored_extension=yes sysctl_conf_in_systemd_set=no lexical_order=40,50,70,90\n'

Operators sometimes continue reading the shadowed vendor file line by line. That is wrong: same-name selection removes the entire lower-priority file before assignment order is evaluated. In this fixture, vm.overcommit_memory=0 disappears with the vendor copy even though the local 50-shared.conf does not mention that key.

Reduce the Surviving Assignments to Final Values

Source order is not yet an answer for a specific key. Reduce the merged text in order and retain the last assignment. The helper below ignores comments and non-assignment lines, trims whitespace, and returns the final matching value. It does not interpret every sysctl syntax edge case; the later procps dry-run remains the parser admission gate.

effective_value() {
  local wanted=$1
  awk -F= -v wanted="$wanted" '
    /^[[:space:]]*#/ || !/=/{next}
    {
      key=$1
      value=substr($0,index($0,"=")+1)
      gsub(/^[[:space:]]+|[[:space:]]+$/,"",key)
      gsub(/^[[:space:]]+|[[:space:]]+$/,"",value)
      if(key==wanted) result=value
    }
    END{
      if(result=="") exit 1
      print result
    }
  ' "$LAB/systemd-cat-config.txt"
}

SWAPPINESS=$(effective_value vm.swappiness)
PID_MAX=$(effective_value kernel.pid_max)
FILE_MAX=$(effective_value fs.file-max)

[[ "$SWAPPINESS" == 10 &&
   "$PID_MAX" == 200000 &&
   "$FILE_MAX" == 100000 ]] ||
  fail "effective value assertion failed"

printf '%s\n' \
  "vm.swappiness = $SWAPPINESS" \
  "kernel.pid_max = $PID_MAX" \
  "fs.file-max = $FILE_MAX" \
  > "$LAB/effective.conf"
printf 'effective_values=vm.swappiness:%s,kernel.pid_max:%s,fs.file-max:%s\n' \
  "$SWAPPINESS" "$PID_MAX" "$FILE_MAX"

Each of the three outcomes tests a different case. Swappiness is assigned in three surviving files, so 90-local.conf wins. PID maximum appears in both same-named files, so only the /etc value survives. File maximum is assigned once and remains unchanged by later files. Together they test more than one repeated override.

Now let procps parse the calculated file without applying it. Snapshot live values again after the command and require exact equality with the admission snapshot.

sysctl \
  --dry-run \
  --load="$LAB/effective.conf" \
  > "$LAB/dry-run.txt"

AFTER_SWAPPINESS=$(sysctl -n vm.swappiness)
AFTER_PID_MAX=$(sysctl -n kernel.pid_max)
AFTER_FILE_MAX=$(sysctl -n fs.file-max)

[[ "$BEFORE_SWAPPINESS" == "$AFTER_SWAPPINESS" ]] ||
  fail "dry-run changed vm.swappiness"
[[ "$BEFORE_PID_MAX" == "$AFTER_PID_MAX" ]] ||
  fail "dry-run changed kernel.pid_max"
[[ "$BEFORE_FILE_MAX" == "$AFTER_FILE_MAX" ]] ||
  fail "dry-run changed fs.file-max"
printf 'dry_run_parser=accepted values=%s current_unchanged=vm.swappiness:%s,kernel.pid_max:%s,fs.file-max:%s\n' \
  "$(paste -sd, "$LAB/dry-run.txt")" \
  "$AFTER_SWAPPINESS" "$AFTER_PID_MAX" "$AFTER_FILE_MAX"

Successful dry-run establishes two narrow facts: procps accepted these three assignment lines, and the observed live values did not change. It does not confirm that the kernel would accept every value during a real write, because --dry-run prints without writing. Apply only in a separate change window with a workload-specific reason, baseline metrics and a tested rollback.

Keep systemd and procps Loader Scope Separate

Here, the alternate systemd source trace excluded /etc/sysctl.conf, but that does not make the legacy file irrelevant to every command. Debian’s current procps sysctl.conf manual documents that sysctl --system reads configuration directories and then /etc/sysctl.conf. systemd-sysctl consumes the sysctl.d set; procps sysctl --system has its own aggregate scope.

Current Arch systemd manual packaging corroborates the systemd directory and lexical semantics, but distribution version and invocation still belong in the receipt. Record both loaders by name. “Linux reads this value” is too vague for an audit.

Finish with an input that parses the alternate /etc/sysctl.conf explicitly, proves the live swappiness value still matches its original snapshot, prints the cleanup receipt, and releases only the marker-owned path.

sysctl \
  --dry-run \
  --load="$ROOT/etc/sysctl.conf" \
  > "$LAB/sysctl-conf-dry-run.txt"

SYSCTLCONF_AFTER=$(sysctl -n vm.swappiness)
[[ "$BEFORE_SWAPPINESS" == "$SYSCTLCONF_AFTER" ]] ||
  fail "explicit sysctl.conf dry-run changed state"
printf 'loader_boundary=systemd_cat_config_excludes_etc_sysctl_conf explicit_procps_dry_run=%s current_unchanged=%s\n' \
  "$(tr -d '\n' < "$LAB/sysctl-conf-dry-run.txt")" \
  "$SYSCTLCONF_AFTER"

printf 'verification=alternate_root_only host_values_unchanged=yes final_assignment_source=90-local.conf same_name_source=etc/50-shared.conf\n'
printf 'rollback=delete_path:%s only_after_marker:%s fixture_hash_manifest:%s\n' \
  "$LAB" \
  "$MARKER_VALUE" \
  "$(sha256sum "$LAB/fixture-hashes.txt" | awk '{print $1}')"

owned_cleanup
trap - EXIT
[[ ! -e "$LAB" ]] ||
  fail "owned lab path remained after cleanup"

If source selection matters after an incident, preserve the command, version, source trace, fixture manifest and assertions alongside systemd journal integrity audit rather than saving only a screenshot or remembered filename. A receipt is useful because another operator can reproduce its decision boundary.

Here is the representative output from the exact lab run:

environment=os=debian_13 kernel=6.12.101+deb13-amd64 systemd=systemd_257_(257.13-1~deb13u1) procps=sysctl_from_procps-ng_4.0.4
fixture_count=7 fixture_manifest_sha256=511ff3264fa058e45100ee7a9642ccc01f6a53e51449547bc93476ad47b2bfec
# $ROOT/usr/lib/sysctl.d/40-vendor.conf
vm.swappiness = 40
fs.file-max = 100000
# $ROOT/etc/sysctl.d/50-shared.conf
kernel.pid_max = 200000
# $ROOT/run/sysctl.d/70-runtime.conf
vm.swappiness = 25
# $ROOT/etc/sysctl.d/90-local.conf
vm.swappiness = 10
selection_assertion=same_name_etc_shadows_usr ignored_extension=yes sysctl_conf_in_systemd_set=no lexical_order=40,50,70,90
effective_values=vm.swappiness:10,kernel.pid_max:200000,fs.file-max:100000
dry_run_parser=accepted values=vm.swappiness = 10,kernel.pid_max = 200000,fs.file-max = 100000 current_unchanged=vm.swappiness:60,kernel.pid_max:4194304,fs.file-max:9223372036854775807
loader_boundary=systemd_cat_config_excludes_etc_sysctl_conf explicit_procps_dry_run=vm.swappiness = 7 current_unchanged=60
verification=alternate_root_only host_values_unchanged=yes final_assignment_source=90-local.conf same_name_source=etc/50-shared.conf
rollback=delete_path:/tmp/voxfor-sysctl-precedence-193 only_after_marker:voxfor-sysctl-precedence-193 fixture_hash_manifest:511ff3264fa058e45100ee7a9642ccc01f6a53e51449547bc93476ad47b2bfec

This hash identifies the fixture manifest, not a universal approved configuration. Recompute it for every review. The equality that matters is the before/after live-value comparison within the same run.

Turn the Trace Into a Change Receipt

Real changes may proceed only when the source trace and the operational reason agree. Keep the source set, last assignment per key, parser result, current live value, proposed value, owner, application method, monitoring window and rollback command in one change record.

Stop when any one of these conditions is true:

1. A same-name file suppresses unrelated assignments you still need.
2. The last assignment comes from a source your team does not own.
3. systemd-sysctl and sysctl --system would consume different final inputs, but the actual boot or automation path is unknown.
4. A dry-run diagnostic is nonzero or the key is unavailable on the target kernel.
5. Another service, boot hook, container manager or configuration agent can rewrite the key after the chosen loader.
6. The proposed value lacks a workload-specific acceptance metric and rollback threshold.

For networking symptoms, source precedence is only one branch. An effective nf_conntrack_max value can be exactly what the files specify and still be operationally insufficient. Use Linux conntrack capacity diagnosis to measure the live table, allocation failures and recovery boundary instead of treating configuration resolution as capacity proof.

Admit the configuration to a real change window only when the operating-system, systemd and procps versions are recorded; the alternate root is collision-free and marker-owned; the seven fixture hashes are preserved; the systemd source set contains exactly the intended four .conf files in 40,50,70,90 order; the /etc same-name file suppresses the /usr/lib copy; the .txt file and /etc/sysctl.conf remain outside that systemd set; the reduced values and their last sources match the planned configuration; procps accepts the calculated file in dry-run mode; all three live kernel values remain byte-for-byte identical; loader scope is named explicitly; and cleanup leaves no owned path.

If any assertion fails, stop before a real sysctl write. Preserve the source trace, fixture hashes, diagnostics and live-value snapshot, then remove only /tmp/voxfor-sysctl-precedence-193 after validating marker value voxfor-sysctl-precedence-193. A production rollback is a separate change: restore the previously approved source file, run the correct loader for that system, verify the prior live value and confirm the workload metric has returned. Never delete a broader /tmp, /etc/sysctl.d, /run/sysctl.d or /usr/lib/sysctl.d path.

One safe answer to “which sysctl.d file wins?” is a reproducible source trace, not a remembered directory rule. Select same-name files first, sort the survivors second, reduce each key to its last assignment third, and keep loader scope explicit. Only then does a real kernel change become a deliberate decision instead of a precedence experiment.