Test Alertmanager Routing Before the Wrong Team Gets Paged

Before editing a route, write down who must own each representative alert. The candidate configuration is acceptable only if native amtool evaluation returns those exact receiver names; a parseable YAML file is merely an input to that decision.

This lab starts with an ownership table, turns each row into a fail-closed assertion, and then uses one deliberately shadowed route to prove the assertions can reject a valid file. That contract-first order keeps the operator focused on routing outcomes instead of treating a green syntax check as the goal.

Define the Ownership Contract First

Choose label sets that change a human action, escalation path or fallback. For this deliberately small tree, three rows cover the narrow critical branch, the broad critical branch and the root receiver.

Each row is a release decision, not a sample for visual inspection. A production matrix should add every distinct branch whose receiver changes operational ownership. Repeating labels that traverse the same route adds noise rather than coverage.

Route order belongs in the contract. The detailed Alertmanager routing examples show why a broad sibling above a narrow sibling can make the latter unreachable. Regex anchoring, missing labels and continue add more branches, so tests should use label values copied from real rule outputs rather than invented ideal values.

Label governance also matters upstream. If one label starts multiplying series unexpectedly, use Prometheus cardinality diagnosis before adding routing complexity around accidental dimensions.

Translate Alertmanager Traversal Into Testable Rules

Alertmanager reads an ordered routing tree, not a flat list of independent filters. The current Prometheus configuration reference says every alert starts at the root, examines child routes, and stops after the first matching child unless that route sets continue: true. When no child matches, the current node’s receiver handles the alert.

amtool check-config answers whether Alertmanager can parse and load the configuration. It catches malformed YAML, invalid fields and references to receivers that do not exist. It cannot infer that the payments team’s critical alert should bypass a broader platform route. Operational intent lives outside the syntax unless the contract encodes it.

Use promtool alert timing tests to prove an upstream rule emits expected labels at the expected time. Receiver admission begins at the next boundary, where Alertmanager maps those labels to notification owners.

Current Alertmanager documentation exposes the native route command in the official project README, while the Debian amtool manual documents local-file precedence and receiver verification. Neither command sends a page, so the contract can run safely before a reload or inside CI.

Build Two Fixtures Around One Ordering Hypothesis

For this secret-free lab, Alertmanager 0.33.1, released on July 4, 2026, supplies the tested amtool build. The workflow downloads the official archive and checksum file, works only inside a fresh marker-owned directory and does not start Alertmanager or contact any receiver.

Run all tested blocks in one Bash session. The examples require curl, tar, grep and sha256sum on Linux amd64.

Acquire the exact amtool build

First, refuse stale state instead of deleting it, verify the release archive and record the binary revision. Review the current Alertmanager release page before changing the pinned version.

set -Eeuo pipefail
lab_root=/tmp/voxfor-alertmanager-route-168
version=0.33.1
archive="alertmanager-${version}.linux-amd64.tar.gz"
base_url="https://github.com/prometheus/alertmanager/releases/download/v${version}"

test ! -e "$lab_root"
for tool in curl tar grep sha256sum; do command -v "$tool" >/dev/null; done
install -d -m 0700 "$lab_root"
printf '%s\n' 'voxfor-alertmanager-route-168' >"$lab_root/.owner-marker"
cd "$lab_root"
curl -fsSLo "$archive" "$base_url/$archive"
curl -fsSLo SHA256SUMS "$base_url/sha256sums.txt"
grep "  $archive\$" SHA256SUMS >SHA256SUMS.selected
sha256sum -c SHA256SUMS.selected
tar -xzf "$archive"
amtool="$lab_root/alertmanager-${version}.linux-amd64/amtool"
"$amtool" --version 2>&1 | sed -n '1p'

Create the shadowed tree intentionally

Both child routes match a payments-critical alert. Since the broad severity="critical" route is first and does not continue, the second sibling is never evaluated.

cat >"$lab_root/alertmanager-bad.yml" <<'YAML'
route:
  receiver: fallback
  routes:
    - receiver: platform-pager
      matchers:
        - severity = "critical"
    - receiver: payments-pager
      matchers:
        - team = "payments"
        - severity = "critical"
receivers:
  - name: fallback
  - name: platform-pager
  - name: payments-pager
YAML

Receiver names are deliberately inert: none contains email, webhook, PagerDuty or chat configuration. routes test evaluates the tree locally, so this negative control cannot wake anyone.

Separate syntax success from route outcome

Run the configuration check first, then ask where the exact payments label set resolves. A green first command and wrong second result are the core failure this article protects against.

"$amtool" check-config "$lab_root/alertmanager-bad.yml" \
  >"$lab_root/bad-check.txt"
"$amtool" config routes test \
  --config.file="$lab_root/alertmanager-bad.yml" \
  team=payments severity=critical \
  | tee "$lab_root/bad-route.txt"
test "$(tail -n 1 "$lab_root/bad-route.txt")" = platform-pager
printf 'bad_config_syntax=valid bad_actual_receiver=%s\n' \
  "$(tail -n 1 "$lab_root/bad-route.txt")"

That final test assertion prevents a future tool or fixture change from quietly turning this intended negative control into a different result. Evidence is useful only when the lab itself fails if its premise stops being true.

Interrogate the Candidate With a Failing Assertion

Seeing the resolved receiver is informative; verifying the expected receiver is a release gate. --verify.receivers returns nonzero when its expected set differs from the resolved set, which CI can enforce without parsing decorative tree output.

Retain the expected mismatch

Do not hide the failing command behind || true. Capture its status, prove it failed for the expected reason and keep the text until the repaired matrix passes.

set +e
"$amtool" config routes test \
  --config.file="$lab_root/alertmanager-bad.yml" \
  --verify.receivers=payments-pager \
  team=payments severity=critical \
  >"$lab_root/bad-verify.txt" 2>&1
bad_rc=$?
set -e
test "$bad_rc" -ne 0
grep -q 'Expected receivers did not match resolved receivers' \
  "$lab_root/bad-verify.txt"
grep -q '^platform-pager$' "$lab_root/bad-verify.txt"
printf 'negative_control=retained rc=%s\n' "$bad_rc"

Failure retention distinguishes a working assertion from a test that always returns success. The community article Unit Testing Alertmanager Routing and Inhibition Rules makes the same broader point: default, named-route and inhibition cases need explicit expected outcomes rather than a visual inspection of configuration.

Put the narrow decision before the broad one

For this one-destination policy, moving the payments-specific route above the general critical route is the smallest repair. Adding continue: true would change the contract to multiple receivers; it is not a substitute for ordering when only one team should be paged.

cat >"$lab_root/alertmanager-fixed.yml" <<'YAML'
route:
  receiver: fallback
  routes:
    - receiver: payments-pager
      matchers:
        - team = "payments"
        - severity = "critical"
    - receiver: platform-pager
      matchers:
        - severity = "critical"
receivers:
  - name: fallback
  - name: platform-pager
  - name: payments-pager
YAML
"$amtool" check-config "$lab_root/alertmanager-fixed.yml" \
  >"$lab_root/fixed-check.txt"

No receiver definition, timer or grouping key changed. A minimal diff keeps the cause visible: sibling order, not a new notifier, repaired ownership.

Convert Three Outcomes Into a Decision Receipt

Passing the repaired payments case alone does not prove the broad branch and fallback remained intact. Execute all three contracts after every route edit.

Verify each expected receiver fail-closed

"$amtool" config routes test \
  --config.file="$lab_root/alertmanager-fixed.yml" \
  --verify.receivers=payments-pager \
  team=payments severity=critical >"$lab_root/payments.txt"

"$amtool" config routes test \
  --config.file="$lab_root/alertmanager-fixed.yml" \
  --verify.receivers=platform-pager \
  team=platform severity=critical >"$lab_root/platform.txt"

"$amtool" config routes test \
  --config.file="$lab_root/alertmanager-fixed.yml" \
  --verify.receivers=fallback \
  team=payments severity=warning >"$lab_root/fallback.txt"

printf 'fixed_contract=payments:%s platform:%s warning:%s\n' \
  "$(tail -n 1 "$lab_root/payments.txt")" \
  "$(tail -n 1 "$lab_root/platform.txt")" \
  "$(tail -n 1 "$lab_root/fallback.txt")"

The following block is the byte-faithful standard output from the complete reproduced sequence. The strict command’s detailed mismatch text was written to bad-verify.txt; the visible receipt records its asserted nonzero status without inventing a normalized message.

alertmanager-0.33.1.linux-amd64.tar.gz: OK
amtool, version 0.33.1 (branch: HEAD, revision: 2c8da51e03f3dbbed24f9711ca2d76aab4eef9c5)
platform-pager
bad_config_syntax=valid bad_actual_receiver=platform-pager
negative_control=retained rc=1
fixed_contract=payments:payments-pager platform:platform-pager warning:fallback
cleanup_scope=/tmp/voxfor-alertmanager-route-168 absent=yes

Use the Matrix as a Source-Control Contract

Keep representative label contracts in source control beside the Alertmanager file. A pull request should run check-config and every routes test --verify.receivers assertion against the exact candidate file that deployment will load. Pin the tool version or container digest, print it in the job and preserve failed output as an artifact.

Route tests should come from rule output, not memory. Grafana-managed alerts require their own state policy; Grafana No Data and Error handling helps decide which upstream states should emit labels before those labels reach Alertmanager.

amtool route evaluation does not prove receiver credentials, network delivery, templates, grouping timers, inhibition, mute or active time intervals, notification retries or human acknowledgement. After static admission passes, test safe nonpaging integrations in staging. Check external endpoint reachability with Blackbox Exporter protocol probes, while customer-visible recovery should still be measured with journey-level uptime tests.

Accept the reproduced route change when the pinned amtool archive passes its published checksum; the intentionally broad-first file passes syntax yet resolves payments-critical to platform-pager; strict expectation of payments-pager returns nonzero and retains the mismatch; the repaired file passes syntax; payments-critical, platform-critical and payments-warning resolve exactly to payments-pager, platform-pager and fallback; and marker-scoped cleanup leaves no lab directory. Apply the same release rule to every production branch whose receiver changes an operational owner.

Keep the Candidate Reversible

Cleanup has no daemon to stop; it verifies the exact marker and deletes only files inside the dedicated directory. Do not reuse this block against /etc/alertmanager or any shared release cache.

test "$(cat "$lab_root/.owner-marker")" = voxfor-alertmanager-route-168
find "$lab_root" -mindepth 1 -maxdepth 1 -type f -delete
find "$lab_root" -mindepth 1 -maxdepth 1 -type d -exec rm -rf -- {} +
rmdir "$lab_root"
test ! -e "$lab_root"
printf 'cleanup_scope=%s absent=yes\n' "$lab_root"

When any production contract fails, keep the currently loaded Alertmanager configuration unchanged, preserve the candidate file, exact label set, resolved receiver, expected receiver, tool version and job output, then correct the smallest owning route and rerun the complete matrix. If a bad file was already reloaded, restore the last reviewed configuration through the normal deployment mechanism, confirm that reload succeeded, and execute both static route tests and a safe end-to-end staging notification before closing the change.

End at the Static-to-Live Boundary

A route test can prove that a candidate tree maps labels to the intended receiver names without disturbing a live system. It cannot prove that the receiver accepts a payload or that a human sees and acknowledges it.

Use the matrix as the first fail-closed gate, then reload through controlled change management and exercise a safe staging notification. Syntax answers whether Alertmanager can read the file; receiver tests answer where representative alerts will go; end-to-end notification tests answer whether the destination actually works.