Test Ansible Idempotency With a Fail-Closed Second Run

Run the same Ansible playbook twice against unchanged state. The first normal run may make every necessary change; the second should finish with changed=0, unreachable=0, and failed=0 for every targeted host. Then introduce one safe, known drift, prove the playbook corrects it once, and require the immediate repeat to return to zero.

That sequence is stronger than a successful first run. It also answers a different question from --check: check mode predicts what supporting modules would change, while a second normal run measures how the real playbook reports already-converged state.

This guide is for developers and automation operators who can read YAML and run a playbook. The reproduced fixture needs Python 3 with virtual-environment support, network access to install a pinned ansible-core, and a disposable Linux account that can create files under /tmp. It does not touch system packages, services, users, /etc, or a production inventory.

Idempotency Is a Measured Second Run

An operation is idempotent when repeating it without an intervening state change leaves the same result. Ansible’s playbook documentation says most modules first check whether the desired state already exists, but it also warns that not every module and playbook behaves that way.

Five related checks therefore need separate names:

Provisioning completion belongs outside this table. A host may exist before its application is ready; Cloud-Init readiness evidence shows why a finished provisioning stage and a usable service are separate boundaries. Ansible convergence should likewise be paired with an application-specific check rather than treated as universal health proof.

Official check and diff mode documentation calls check mode a simulation. Modules without check support may do nothing, and conditionals that depend on registered results from earlier tasks may not have usable data. Use the preview to inspect a proposed change, then use two normal runs in an authorized sandbox to measure convergence.

Build and Converge a Guarded Fixture

Keep one shell open while working through the lab. The bootstrap refuses a pre-existing path, writes a marker before later cleanup, pins ansible-core 2.21.3, and uses only a localhost inventory. Pinning is for reproduction, not a recommendation to ignore later supported releases; update the pin deliberately and rerun the same acceptance sequence when your project upgrades.

set -Eeuo pipefail
LAB_ROOT="${TMPDIR:-/tmp}/voxfor-ansible-idempotency-lab"
MARKER="$LAB_ROOT/.voxfor-ansible-idempotency-lab"

if [[ -e "$LAB_ROOT" ]]; then
  printf 'Refusing existing path: %s\n' "$LAB_ROOT" >&2
  exit 1
fi

mkdir -p "$LAB_ROOT/logs"
printf 'VOXFOR_ANSIBLE_IDEMPOTENCY_LAB\n' > "$MARKER"
python3 -m venv "$LAB_ROOT/venv"
"$LAB_ROOT/venv/bin/python" -m pip install \
  --quiet --disable-pip-version-check 'ansible-core==2.21.3'

cat > "$LAB_ROOT/inventory.ini" <<'EOF'
[lab]
local ansible_connection=local ansible_python_interpreter=/usr/bin/python3

[masking_pair]
clean_peer ansible_connection=local ansible_python_interpreter=/usr/bin/python3
changing_peer ansible_connection=local ansible_python_interpreter=/usr/bin/python3
EOF

cat > "$LAB_ROOT/ansible.cfg" <<'EOF'
[defaults]
inventory = inventory.ini
host_key_checking = False
retry_files_enabled = False
stdout_callback = default
interpreter_python = auto_silent
display_skipped_hosts = False
force_color = False
EOF

export LAB_ROOT ANSIBLE_CONFIG="$LAB_ROOT/ansible.cfg" ANSIBLE_NOCOLOR=1
export PATH="$LAB_ROOT/venv/bin:$PATH"
cd "$LAB_ROOT"
ansible --version | sed -n '1,5p'

Below, the playbook declares a directory, exact configuration bytes, a release identity, and a read-only digest check. ansible.builtin.copy owns file content and mode. By contrast, the SHA-256 command is explicitly changed_when: false because it only observes state; its assertion proves the command succeeded and returned the expected output shape.

cat > "$LAB_ROOT/converge.yml" <<'EOF'
---
- name: Converge the disposable application fixture
  hosts: lab
  gather_facts: false
  vars:
    target_root: "{{ lookup('ansible.builtin.env', 'LAB_ROOT') }}/target"
  tasks:
    - name: Ensure the target directory exists
      ansible.builtin.file:
        path: "{{ target_root }}"
        state: directory
        mode: '0750'

    - name: Declare the application configuration
      ansible.builtin.copy:
        dest: "{{ target_root }}/app.conf"
        mode: '0640'
        content: |
          port=8080
          mode=production

    - name: Declare the release identity
      ansible.builtin.copy:
        dest: "{{ target_root }}/release.txt"
        mode: '0640'
        content: "release=2026.08\n"

    - name: Read the deployed configuration without reporting a change
      ansible.builtin.command:
        argv:
          - /usr/bin/sha256sum
          - "{{ target_root }}/app.conf"
      register: config_digest
      changed_when: false

    - name: Verify the deployed configuration digest is present
      ansible.builtin.assert:
        that:
          - config_digest.rc == 0
          - config_digest.stdout is match('^[0-9a-f]{64}  .+/app\\.conf$')
        quiet: true
EOF

Now create a gate that records both runs and inspects every recap row, not just the final line or one favored host. An empty recap is rejected. Any host with non-zero changed, unreachable, or failed is rejected as well.

cat > "$LAB_ROOT/second-run-gate.sh" <<'EOF'
#!/usr/bin/env bash
set -Eeuo pipefail
playbook="${1:?playbook path required}"
label="${2:?label required}"

ansible-playbook "$playbook" | tee "logs/${label}-first.txt"
ansible-playbook "$playbook" | tee "logs/${label}-second.txt"
recap="$(awk '/^[^[:space:]][^:]*[[:space:]]+:/ { print }' \
  "logs/${label}-second.txt")"

if ! awk '
  /^[^[:space:]][^:]*[[:space:]]+:/ {
    seen += 1
    changed = unreachable = failed = -1
    for (field = 1; field <= NF; field += 1) {
      split($field, pair, "=")
      if (pair[1] == "changed") changed = pair[2] + 0
      if (pair[1] == "unreachable") unreachable = pair[2] + 0
      if (pair[1] == "failed") failed = pair[2] + 0
    }
    if (changed != 0 || unreachable != 0 || failed != 0) bad = 1
  }
  END { exit !(seen > 0 && bad == 0) }
' "logs/${label}-second.txt"; then
  printf 'idempotence_gate=rejected label=%s recap=%s\n' \
    "$label" "$recap" >&2
  exit 1
fi

printf 'idempotence_gate=accepted label=%s recap=%s\n' "$label" "$recap"
EOF

chmod 0750 "$LAB_ROOT/second-run-gate.sh"
ansible-playbook "$LAB_ROOT/converge.yml" --syntax-check
"$LAB_ROOT/second-run-gate.sh" "$LAB_ROOT/converge.yml" stable \
  | tee "$LAB_ROOT/logs/stable-gate.txt"

In the reproduced fixture, first convergence changed three tasks; the second changed none. That is the expected shape: idempotence does not mean the first run must be green or unchanged. It means repetition stops producing further state changes once the declared result exists.

Make the Gate Reject a Successful False Change

A zero exit code is not a convergence result. The next playbook targets two logical inventory hosts through the localhost connection: clean_peer deliberately skips the operation, while changing_peer touches the same file on every run. This isolates recap parsing rather than network behavior. /usr/bin/touch succeeds both times for the changing peer, but ansible.builtin.command executes an operation rather than comparing a desired file state, so that recap row remains changed=1 and the whole gate must fail.

cat > "$LAB_ROOT/always-changed.yml" <<'EOF'
---
- name: Demonstrate a task that cannot describe desired state
  hosts: masking_pair
  gather_facts: false
  vars:
    target_root: "{{ lookup('ansible.builtin.env', 'LAB_ROOT') }}/target"
  tasks:
    - name: Touch a receipt on every run
      ansible.builtin.command:
        argv:
          - /usr/bin/touch
          - "{{ target_root }}/receipt.txt"
      when: inventory_hostname == 'changing_peer'
EOF

set +e
"$LAB_ROOT/second-run-gate.sh" "$LAB_ROOT/always-changed.yml" false-change \
  > "$LAB_ROOT/logs/false-change-gate.txt" 2>&1
false_change_rc=$?
set -e

if [[ "$false_change_rc" -eq 0 ]]; then
  printf 'The negative control was accepted unexpectedly.\n' >&2
  exit 1
fi
grep 'idempotence_gate=rejected' "$LAB_ROOT/logs/false-change-gate.txt"
grep -E '^clean_peer +: .*changed=0 .*unreachable=0 .*failed=0' \
  "$LAB_ROOT/logs/false-change-second.txt"
grep -E '^changing_peer +: .*changed=1 .*unreachable=0 .*failed=0' \
  "$LAB_ROOT/logs/false-change-second.txt"

The mixed recap is the masking test: clean_peer reports changed=0, while changing_peer reports changed=1, and the parser rejects the combined result. Both aliases use the same local machine, so this proves all-row evaluation only; it does not claim to reproduce SSH reachability or distributed-host failure modes.

Ansible’s command module reference provides creates and removes when file existence is the real condition that decides whether a command should run. A sentinel is safe only when its presence reliably means the entire operation completed and its result remains valid. Otherwise, prefer a purpose-built module or query the authoritative state before deciding what to do.

Do not “fix” the example by adding changed_when: false blindly. Ansible’s changed_when guidance explains that the condition controls both reported change status and whether handlers are notified. Suppressing a real mutation would make the recap look clean while preserving the harmful behavior.

Replace the Operation With Declared State

Because the receipt is a file with known bytes and permissions, ansible.builtin.copy can own the result directly. The first repaired run replaces the empty touched file; the second sees the same content and mode and reports no change.

cat > "$LAB_ROOT/repaired.yml" <<'EOF'
---
- name: Replace the operation with declared file state
  hosts: lab
  gather_facts: false
  vars:
    target_root: "{{ lookup('ansible.builtin.env', 'LAB_ROOT') }}/target"
  tasks:
    - name: Declare the receipt content and permissions
      ansible.builtin.copy:
        dest: "{{ target_root }}/receipt.txt"
        mode: '0640'
        content: "created_by=ansible\n"
EOF

"$LAB_ROOT/second-run-gate.sh" "$LAB_ROOT/repaired.yml" repaired \
  | tee "$LAB_ROOT/logs/repaired-gate.txt"

State-aware modules reduce custom logic, but task authors still own the declared value. A template containing a fresh timestamp, random token, unordered input, or environment-dependent path can change on every run even though the module is working correctly. Trace the diff to the unstable input instead of blaming the module.

Inject Drift, Correct It Once, Then Stop

A playbook that always does nothing could pass a second-run parser while failing its real purpose. The stronger test changes exactly one disposable input outside Ansible, reruns convergence, and expects exactly one corrective change. The next run must stop again.

printf 'port=9090\nmode=unsafe\n' > "$LAB_ROOT/target/app.conf"

ansible-playbook "$LAB_ROOT/converge.yml" \
  | tee "$LAB_ROOT/logs/drift-repair.txt"
ansible-playbook "$LAB_ROOT/converge.yml" \
  | tee "$LAB_ROOT/logs/post-drift-second.txt"

post_drift_recap="$(grep -E '^local +: ' \
  "$LAB_ROOT/logs/post-drift-second.txt" | tail -n 1)"
[[ "$post_drift_recap" =~ changed=0 ]]
[[ "$post_drift_recap" =~ unreachable=0 ]]
[[ "$post_drift_recap" =~ failed=0 ]]
grep -qx 'port=8080' "$LAB_ROOT/target/app.conf"
grep -qx 'mode=production' "$LAB_ROOT/target/app.conf"
[[ "$(wc -l < "$LAB_ROOT/target/app.conf")" -eq 2 ]]

sha256sum \
  "$LAB_ROOT/target/app.conf" \
  "$LAB_ROOT/target/release.txt" \
  "$LAB_ROOT/target/receipt.txt"

Captured on Debian 13 with Python 3.13.5 and ansible-core 2.21.3, the representative output preserves the meaningful transitions rather than every task banner.

stable first:   local : ok=5 changed=3 unreachable=0 failed=0
stable second:  local : ok=5 changed=0 unreachable=0 failed=0
idempotence_gate=accepted label=stable

false second:   clean_peer    : ok=0 changed=0 unreachable=0 failed=0 skipped=1
                changing_peer : ok=1 changed=1 unreachable=0 failed=0 skipped=0
idempotence_gate=rejected label=false-change

repaired first: local : ok=1 changed=1 unreachable=0 failed=0
repaired second:local : ok=1 changed=0 unreachable=0 failed=0
idempotence_gate=accepted label=repaired

drift repair:   local : ok=5 changed=1 unreachable=0 failed=0
drift repeat:   local : ok=5 changed=0 unreachable=0 failed=0
83f5a0594544d01fef4abdd60de64ddb52f185a4c224836ffd0c2403861ae657  app.conf

The fixture is accepted when the stable and repaired second runs have zero changes, unreachable hosts, and failures; the negative control is rejected with changed=1; deliberate configuration drift produces one corrective change; the immediate repeat returns to zero; and app.conf contains exactly port=8080 plus mode=production with SHA-256 83f5a0594544d01fef4abdd60de64ddb52f185a4c224836ffd0c2403861ae657.

Ansible can correct only state it owns. Outside a deployment, AIDE trusted-baseline monitoring can reveal unauthorized file drift across a broader integrity policy; it does not replace Ansible’s convergence test, and Ansible does not replace intrusion or change detection.

Carry the Evidence Into CI Without Overclaiming

Run the gate on an isolated, resettable target whose initial state is known. The guidance on self-hosted runner baselines helps separate runner persistence, credentials, permissions and workspace cleanup from the playbook’s own state. Ephemeral containers or VMs may be better when the role needs packages, services, reboots or kernel features that localhost cannot model.

For reusable roles, current Molecule workflow documentation supports an idempotence action inside a scenario test sequence. Molecule can automate environment creation and whole-role reruns; the acceptance meaning remains the same. Preserve the first output, second output, target image/version, inventory identity and any state-specific verification rather than saving only a green job badge.

Place convergence evidence before deployment and rollback handoff, but do not confuse the tools’ state domains. Before a Terraform-to-OpenTofu cutover, consult OpenTofu state-readiness evidence because infrastructure state compatibility is not proven by an Ansible recap.

Handlers deserve explicit review. An unnecessary changed result can trigger a restart or reload; a dishonest changed_when: false can prevent a required handler. Test the notifying task, the handler’s actual effect and a second stable run. Where rolling changes are intended—certificate renewal, package update, database migration, secret refresh—the desired version or authoritative state must still tell Ansible when the operation is complete.

Finally, changed=0 says nothing about exactly-once business effects. A command may charge a card, create an account or send a message before its response is lost. For those boundaries, preserve durable action-reconciliation evidence at the system that owns the effect. Ansible reporting cannot safely deduplicate an external transaction it cannot query.

Preserve the Receipt, Then Remove the Lab

Keep stable-first.txt, stable-second.txt, the rejected negative-control output, repaired outputs, drift outputs, final hashes and the ansible-core version with the automation change. Those artifacts explain what was tested if a later module, variable, collection or target image changes behavior.

The cleanup below refuses an unexpected path and removes nothing unless the exact marker matches. It is appropriate for this disposable fixture only. Do not translate it into deleting a production inventory, project directory or managed target. Production rollback must restore the application-specific prior state that your change plan identified.

EXPECTED_ROOT="${TMPDIR:-/tmp}/voxfor-ansible-idempotency-lab"
if [[ "$LAB_ROOT" != "$EXPECTED_ROOT" ]]; then
  printf 'Refusing unexpected LAB_ROOT=%s\n' "$LAB_ROOT" >&2
  exit 1
fi
if [[ ! -f "$MARKER" ]] || \
   ! grep -qx 'VOXFOR_ANSIBLE_IDEMPOTENCY_LAB' "$MARKER"; then
  printf 'Refusing cleanup without the exact lab marker.\n' >&2
  exit 1
fi
rm -rf -- "$LAB_ROOT"
test ! -e "$LAB_ROOT"
printf 'cleanup=complete path=%s\n' "$LAB_ROOT"

Rollback for this lab means retaining the evidence, validating the exact /tmp/voxfor-ansible-idempotency-lab path and marker, then deleting only that disposable fixture. If the same technique exposes a production playbook defect, revert the playbook change through version control and restore only the application state named in its own tested recovery plan; never use the lab’s deletion step on a real host.

Keep the Second-Run Receipt

A useful release record is compact: pinned automation version, exact inventory or fixture identity, first and second recaps, the deliberately rejected negative control, one bounded drift correction, application-specific acceptance, final hashes where relevant, and cleanup or rollback evidence. When that record is reproducible, changed=0 becomes an engineering result instead of an optimistic glance at green output.