cloud-init Uses Instance ID to Decide User Data Replay

Changing user-data bytes is not a new-instance event. By default, cloud-init decides whether per-instance modules may run from instance identity and cached state, not by hashing the latest user data and comparing it with the previous copy.

A reproduced NoCloud lab below makes that contract measurable. The first payload runs under iid-a; a changed payload under the same ID stays suppressed; changing the ID to iid-b makes the per-instance module eligible and executes the new payload. The run counts are 1 → 1 → 2.

That distinction matters after an operator edits provider user data, reboots a server, clones a disk or captures a golden image. It separates three different questions: Is the configuration valid? Is the module eligible in this instance? Did the intended effect occur? Treating those as one question is how a green schema check becomes a missed bootstrap action—or how an overly broad cache reset repeats destructive first-boot work.

Changed User Data Is Not a New Instance

An instance ID is the datasource’s identity for one launched machine. On boot, cloud-init compares the current datasource ID with the ID it cached during the previous run. If they match, cloud-init treats the machine as the same instance. If they differ, it normally takes the first-boot path and creates a new instance state directory.

Official cloud-init guidance on first-boot determination documents this cached-ID comparison. It also explains the separate manual_cache_clean mode: when enabled, cloud-init trusts that an image pipeline removed cache before capture instead of checking identity automatically. That is an image-build trust decision, not a convenient replay switch.

Modules add another boundary. scripts-user, which executes user scripts late in boot, normally has per-instance frequency. Its semaphore under the active instance directory records that the module already completed for that identity. Rebooting or replacing user-data content does not, by itself, remove that record. The official module run-frequency reference distinguishes per-always, per-instance and per-once behavior.

This is narrower than a failed first boot. If status, logs or a missing effect show that initial provisioning broke, use the cloud-init first-boot failure diagnosis workflow. Here, the first run succeeds; the question is why later input is not automatically replayed.

Read the Identity cloud-init Is Using

Start with a read-only audit on the affected server. The queryable instance-data view, cached ID file, active instance symlink and semaphore answer related but different questions. Keep all of them in the ticket because datasource integrations sometimes expose a normalized query value while the filesystem points to the concrete cache directory.

This block creates only an article-owned report directory in /tmp; every cloud-init path is read-only.

set -Eeuo pipefail
article_root=/tmp/voxfor-cloud-init-iid-170
receipt_path=$article_root/receipt.json
test ! -e "$article_root"
install -d -m 0700 "$article_root"
printf '%s\n' voxfor-cloud-init-iid-170 >"$article_root/.owner-marker"

{
  cloud-init --version
  cloud-init status --long || true
  printf 'query_instance_id='; cloud-init query instance_id 2>/dev/null || printf 'unavailable\n'
  printf 'cached_instance_id='; tr -d '\n' </var/lib/cloud/data/instance-id 2>/dev/null || printf 'unavailable'
  printf '\nactive_instance='; readlink -f /var/lib/cloud/instance 2>/dev/null || printf 'unavailable'
  printf '\nscripts_user_semaphores:\n'
  find /var/lib/cloud/instances -maxdepth 3 -type f \
    -path '*/sem/config_scripts_user' -printf '%p\n' 2>/dev/null | sort
} | tee "$article_root/readonly-audit.txt"

A same-instance diagnosis needs four facts: the datasource is available; the cached ID still names the current instance; the instance symlink targets that cache; and the relevant module’s semaphore exists. Absence of a semaphore is not proof that a script succeeded or failed—it only changes the next question. Check /var/log/cloud-init.log, /var/log/cloud-init-output.log and the module’s real effect before deciding.

Cloud-init’s instance-data query documentation describes supported keys and warns that sensitive values may require root. Do not paste the complete instance-data JSON into a public ticket; record only the fields needed for the decision.

A Valid File Can Still Be Ineligible to Run

Schema validation answers whether cloud-init understands a cloud-config document. It does not prove that a per-instance module will become eligible on the next reboot. The following optional policy changes scripts-user to every-boot behavior, and the installed cloud-init 25.1.4 validator accepted it.

cat >"$article_root/99-scripts-user-always.cfg" <<'YAML'
#cloud-config
cloud_final_modules:
  - [scripts-user, always]
YAML

cloud-init schema -c "$article_root/99-scripts-user-always.cfg" --annotate \
  | tee "$article_root/schema.txt"
grep -q 'Valid schema' "$article_root/schema.txt"

Do not install that override merely to repair one missed run. Every-boot semantics are appropriate only when the script is intentionally idempotent, has bounded failure behavior and is safe to repeat after an ordinary reboot. A configuration-management system may be the better long-lived owner; a fail-closed Ansible second-run test shows how to prove repeatability instead of assuming it.

For one approved recovery, the official rerun cloud-init guidance documents module-level frequency override and full clean/reboot methods. Prefer the smallest module scope that owns the missing effect. A full clean can revisit network, SSH, package and filesystem initialization; it is not a generic “retry user data” button.

Build the Counterexample Inside a Private Namespace

Our lab uses a NoCloud seed because it makes instance identity explicit. It enters a private mount namespace, bind-mounts disposable directories over cloud-init state, runtime, configuration and logs, disables network configuration and hides the host seed device inside the namespace. The host’s real /var/lib/cloud is never changed.

Install the complete runner exactly as shown. It refuses non-root execution, unexpected temporary paths and receipt overwrite; its traps unmount the private views and delete only a marker-owned random directory.

cat >"$article_root/run_lab.sh" <<'BASH'
#!/usr/bin/env bash
set -Eeuo pipefail
readonly LAB_MARKER='.voxfor-cloud-init-iid-lab'
fail(){ printf 'ERROR: %s\n' "$*" >&2; exit 1; }
require_command(){ command -v "$1" >/dev/null 2>&1 || fail "required command not found: $1"; }

inside_namespace(){
  local lab_root=$1 receipt_path=$2
  local data_root="$lab_root/data" run_root="$lab_root/run" cfg_root="$lab_root/cfgd"
  local log_file="$lab_root/cloud-init.log" output_log="$lab_root/cloud-init-output.log"
  local runs_file="$lab_root/runs" phase_log="$lab_root/phases.tsv"
  [[ -f "$lab_root/$LAB_MARKER" ]] || fail 'lab marker is missing'
  [[ "$lab_root" == /tmp/voxfor-cloud-init-iid.* ]] || fail 'unexpected lab root'
  mkdir -p "$data_root/seed/nocloud" "$run_root" "$cfg_root"
  : >"$log_file"; : >"$output_log"; : >"$runs_file"; : >"$phase_log"
  cat >"$cfg_root/99-voxfor-lab.cfg" <<'YAML'
datasource_list: [NoCloud]
preserve_hostname: true
network: {config: disabled}
manual_cache_clean: false
YAML

  mount --bind "$data_root" /var/lib/cloud
  mount --bind "$run_root" /run/cloud-init
  mount --bind "$log_file" /var/log/cloud-init.log
  mount --bind "$output_log" /var/log/cloud-init-output.log
  mount --bind "$cfg_root" /etc/cloud/cloud.cfg.d
  if [[ -e /dev/sr0 ]]; then mount --bind /dev/null /dev/sr0; fi
  namespace_cleanup(){
    if [[ -e /dev/sr0 ]]; then umount /dev/sr0 2>/dev/null || true; fi
    umount /var/log/cloud-init-output.log 2>/dev/null || true
    umount /var/log/cloud-init.log 2>/dev/null || true
    umount /run/cloud-init 2>/dev/null || true
    umount /var/lib/cloud 2>/dev/null || true
    umount /etc/cloud/cloud.cfg.d 2>/dev/null || true
  }
  trap namespace_cleanup EXIT

  seed_nocloud(){
    local instance_id=$1
    cat >"$data_root/seed/nocloud/meta-data" <<EOF
instance-id: $instance_id
local-hostname: voxfor-iid-lab
EOF
    cat >"$data_root/seed/nocloud/user-data" <<'YAML'
#cloud-config
preserve_hostname: true
network: {config: disabled}
YAML
  }
  simulate_boot(){
    find "$run_root" -mindepth 1 -delete
    cloud-init --force init --local >/dev/null 2>&1
  }
  run_scripts_user_phase(){
    local phase=$1 payload=$2 instance_id run_count semaphore
    mkdir -p /var/lib/cloud/instance/scripts
    cat >/var/lib/cloud/instance/scripts/part-001 <<EOF
#!/bin/sh
echo '$payload' >> '$runs_file'
EOF
    chmod 700 /var/lib/cloud/instance/scripts/part-001
    cloud-init --force single --name scripts-user --frequency instance >/dev/null 2>&1
    instance_id=$(cloud-init query instance_id 2>/dev/null)
    run_count=$(wc -l <"$runs_file")
    [[ -e /var/lib/cloud/instance/sem/config_scripts_user ]] && semaphore=present || semaphore=missing
    printf '%s\t%s\t%s\t%s\n' "$phase" "$instance_id" "$run_count" "$semaphore" >>"$phase_log"
  }

  seed_nocloud iid-a
  simulate_boot
  run_scripts_user_phase first_user_data payload-a
  run_scripts_user_phase changed_user_data_same_instance payload-b
  seed_nocloud iid-b
  simulate_boot
  run_scripts_user_phase new_instance_id payload-b

  mapfile -t phase_rows <"$phase_log"
  [[ ${phase_rows[0]} == $'first_user_data\tiid-a\t1\tpresent' ]] || fail 'first phase mismatch'
  [[ ${phase_rows[1]} == $'changed_user_data_same_instance\tiid-a\t1\tpresent' ]] || fail 'same-ID phase mismatch'
  [[ ${phase_rows[2]} == $'new_instance_id\tiid-b\t2\tpresent' ]] || fail 'new-ID phase mismatch'
  [[ $(cat "$runs_file") == $'payload-a\npayload-b' ]] || fail 'payload sequence mismatch'
  mapfile -t semaphore_paths < <(
    find "$data_root/instances" -path '*/sem/config_scripts_user' -printf '%P\n' | sort
  )
  [[ ${semaphore_paths[*]} == 'iid-a/sem/config_scripts_user iid-b/sem/config_scripts_user' ]] \
    || fail 'semaphore set mismatch'
  local cloud_init_version cache_after_clean=present
  cloud_init_version=$(cloud-init --version 2>&1 | awk '{print $NF}')
  cloud-init clean >/dev/null 2>&1
  if [[ ! -e /var/lib/cloud/instance && ! -d /var/lib/cloud/instances ]]; then
    cache_after_clean=absent
  fi
  [[ "$cache_after_clean" == absent ]] || fail 'isolated cache remained after clean'

  jq -n \
    --arg tested_at_utc "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
    --arg cloud_init_version "$cloud_init_version" \
    --arg cache_after_clean "$cache_after_clean" \
    --argjson phases "$(awk -F '\t' '{printf "%s{\"phase\":\"%s\",\"instance_id\":\"%s\",\"run_count\":%s,\"semaphore\":\"%s\"}", (NR>1?",":""), $1,$2,$3,$4}' "$phase_log" | awk '{print "["$0"]"}')" \
    --argjson executed_payloads "$(jq -R -s 'split("\n") | map(select(length > 0))' "$runs_file")" \
    --argjson semaphore_paths "$(printf '%s\n' "${semaphore_paths[@]}" | jq -R -s 'split("\n") | map(select(length > 0))')" \
    '{tested_at_utc:$tested_at_utc,environment:{cloud_init_version:$cloud_init_version,datasource:"NoCloud",isolation:"private mount namespace",network_configuration:"disabled",host_cloud_state_touched:false},phases:$phases,executed_payloads:$executed_payloads,semaphore_paths_before_clean:$semaphore_paths,cache_after_isolated_clean:$cache_after_clean}' \
    >"$receipt_path"
}

if [[ ${1:-} == --inside ]]; then
  [[ $# -eq 3 ]] || fail 'internal invocation requires lab root and receipt path'
  inside_namespace "$2" "$3"; exit 0
fi
[[ ${EUID} -eq 0 ]] || fail 'run this private mount-namespace lab as root'
for command_name in cloud-init find jq mount sha256sum unshare umount; do require_command "$command_name"; done
readonly OUTPUT_PATH=${1:-"$PWD/cloud-init-instance-id-receipt.json"}
[[ "$OUTPUT_PATH" == /* ]] || fail 'receipt path must be absolute'
[[ ! -e "$OUTPUT_PATH" ]] || fail "refusing to overwrite receipt: $OUTPUT_PATH"
LAB_ROOT=$(mktemp -d /tmp/voxfor-cloud-init-iid.XXXXXX); readonly LAB_ROOT
touch "$LAB_ROOT/$LAB_MARKER"
outer_cleanup(){
  [[ -f "$LAB_ROOT/$LAB_MARKER" ]] || return 0
  [[ "$LAB_ROOT" == /tmp/voxfor-cloud-init-iid.* ]] || return 0
  rm -rf -- "$LAB_ROOT"
}
trap outer_cleanup EXIT
unshare --mount --propagation private -- "$0" --inside "$LAB_ROOT" "$LAB_ROOT/receipt.json"
jq -e '[.phases[].run_count]==[1,1,2] and [.phases[].instance_id]==["iid-a","iid-a","iid-b"] and .executed_payloads==["payload-a","payload-b"] and .semaphore_paths_before_clean==["iid-a/sem/config_scripts_user","iid-b/sem/config_scripts_user"] and .cache_after_isolated_clean=="absent" and .environment.host_cloud_state_touched==false' "$LAB_ROOT/receipt.json" >/dev/null
install -m 0600 "$LAB_ROOT/receipt.json" "$OUTPUT_PATH"
jq -r '.environment.cloud_init_version as $v | "cloud_init_version=\($v)",(.phases[]|"phase=\(.phase) instance_id=\(.instance_id) run_count=\(.run_count) semaphore=\(.semaphore)"),"executed_payloads=\(.executed_payloads|join(","))","semaphores_before_clean=\(.semaphore_paths_before_clean|join(","))","cache_after_isolated_clean=\(.cache_after_isolated_clean)"' "$OUTPUT_PATH"
printf 'receipt_sha256=%s\n' "$(sha256sum "$OUTPUT_PATH" | awk '{print $1}')"
BASH
chmod 0700 "$article_root/run_lab.sh"

Root is needed only because Linux mount namespaces and bind mounts require it. Run the runner on a disposable host you own, not in a restricted shared shell. No network request occurs, and the generated user script writes only into the random lab root.

Read the 1 → 1 → 2 Receipt

Execute the runner once to a new absolute receipt path. It refuses to overwrite an earlier receipt, so repeated tests cannot silently replace evidence.

"$article_root/run_lab.sh" "$receipt_path" \
  | tee "$article_root/lab-output.txt"
test -s "$receipt_path"
test "$(stat -c '%a' "$receipt_path")" = 600

This is the representative output from the reproduced cloud-init 25.1.4 run:

cloud_init_version=25.1.4
phase=first_user_data instance_id=iid-a run_count=1 semaphore=present
phase=changed_user_data_same_instance instance_id=iid-a run_count=1 semaphore=present
phase=new_instance_id instance_id=iid-b run_count=2 semaphore=present
executed_payloads=payload-a,payload-b
semaphores_before_clean=iid-a/sem/config_scripts_user,iid-b/sem/config_scripts_user
cache_after_isolated_clean=absent

phase=changed_user_data_same_instance is the negative control: payload B exists, but the iid-a semaphore keeps scripts-user at one execution. The next line changes the datasource identity, not the module command; cloud-init creates the iid-b state, the per-instance module becomes eligible and the count advances to two. Both semaphore paths remain in the disposable state until the isolated cloud-init clean proves that cleanup is also confined there.

Verify the machine receipt rather than relying on the human-readable summary:

jq -e '
  [.phases[].run_count] == [1,1,2] and
  [.phases[].instance_id] == ["iid-a","iid-a","iid-b"] and
  .executed_payloads == ["payload-a","payload-b"] and
  .semaphore_paths_before_clean == [
    "iid-a/sem/config_scripts_user",
    "iid-b/sem/config_scripts_user"
  ] and
  .cache_after_isolated_clean == "absent" and
  .environment.host_cloud_state_touched == false
' "$receipt_path"
sha256sum "$receipt_path" | tee "$article_root/receipt.sha256"

The evidence passes when the IDs are exactly iid-a, iid-a, iid-b; counts are exactly 1, 1, 2; only payload-a and then payload-b executed; each identity owns its own config_scripts_user semaphore; isolated cache is absent after clean; and the receipt says host cloud state was untouched. Any other sequence rejects the mechanism claim and the image decision.

This receipt also marks the boundary of what was proved. It shows default per-instance suppression and new-ID eligibility in NoCloud. It does not claim every provider exposes editable instance IDs, that every module has the same frequency, or that a full production reboot will succeed.

Choose Replay Scope Before You Remove State

There are three legitimate actions, and the evidence should select among them.

Recover one missing effect. Identify the owning module, back up the state it may change, and use the documented module-level replay only in an approved window. Verify the actual effect, not just cloud-init exit status. If the script installs packages, edits access controls or creates users, define a rollback for those resources first.

Make an action repeat on every boot. Use an explicit frequency policy only for a genuinely idempotent script. The validated scripts-user: always example proves syntax; a second-run acceptance test must still prove behavior. Keep locks, timeouts and failure reporting so a slow dependency cannot hold every boot indefinitely.

Prepare a reusable image. Let the image pipeline own cloud-init cleanup immediately before capture, then stop the builder and create a clone in a separate acceptance environment. Confirm that the clone receives a new provider identity, does not inherit sensitive datasource cache and performs exactly one intended first-boot run. Disk-container checks are not enough: qemu-img versus guest-filesystem verification explains why a healthy image structure does not prove the guest state inside it. If the image uses a QCOW2 backing chain, preserve the separate rebase and commit file-effect boundary before changing layers.

Do not manually edit /var/lib/cloud/data/instance-id to impersonate a new provider instance. The cached ID, datasource data, instance symlink, semaphores, SSH host keys and network state form a lifecycle contract. Partial edits can create a hybrid state that neither the provider nor cloud-init intended.

Clean the article-owned lab only after retaining the receipt hash:

test "$(cat "$article_root/.owner-marker")" = voxfor-cloud-init-iid-170
test -s "$article_root/receipt.sha256"
find "$article_root" -mindepth 1 -maxdepth 1 -type f -delete
rmdir "$article_root"
test ! -e "$article_root"
printf 'cleanup_scope=%s absent=yes\n' "$article_root"

If the lab fails, its traps unmount the private cloud-init views and remove only the random directory carrying .voxfor-cloud-init-iid-lab; the final block removes only /tmp/voxfor-cloud-init-iid-170 after verifying its separate owner marker. On a real server, do not delete cache as an improvised rollback. Preserve console access and a snapshot, restore the backed-up module-owned resources if a replay changes them, and return to the original boot state before retrying a narrower action.

Capture the Image Only After the Boundary Is Deliberate

A defensible image pipeline records the builder’s datasource mode, cloud-init version, cleanup action, shutdown point, image digest and clone acceptance receipt. The clone must show a provider-issued identity different from the builder, exactly one intended first-boot execution and no inherited secret or machine-specific cache.

For broader provisioning and virtualization work, the VPS hosting article hub links the adjacent network, storage and guest-execution checks. The decision here remains precise: changed input is not changed identity. Replay one module for one recovery, declare every-boot semantics for truly repeatable work, or make image cleanup and clone acceptance an explicit pipeline contract.