A Green Kubeconform Run Can Hide Skipped Resources

Kubeconform exited 0 in the reproduced v0.8.0 lab while its JSON summary reported skipped=1. Nothing was invalid and no schema download failed; one custom resource simply had no matching schema, and -ignore-missing-schemas converted that coverage gap into a skipped result.

A reliable CI gate must inspect all four summary counts—not only the process exit code. Require zero invalid, zero errors and zero skipped resources for the exact rendered manifests. Even that is an offline schema admission result, not proof that an API server, admission policy, controller or workload will accept and run the objects.

Treat Exit Zero as One Signal, Not the Decision

Kubeconform validates Kubernetes YAML and JSON against schemas derived from Kubernetes OpenAPI definitions. It is useful before cluster credentials enter a pipeline because a developer can catch field names, types and duplicate YAML keys on an ordinary runner. The maintained Kubeconform project also supports pinned Kubernetes versions, additional schema locations and machine-readable output.

Exit status still answers a narrower question than many pipelines assume. With normal settings, an invalid resource or missing schema makes the command nonzero. Once -ignore-missing-schemas is added, an unknown kind becomes skipped and the command can return zero. A green job then means “no checked resource failed,” not necessarily “every submitted resource was checked.”

That distinction matters most in repositories containing operators, GitOps controllers or platform-specific custom resources. A newly added kind can bypass offline validation until its schema catalog catches up. Marking the job successful without reading summary.skipped silently changes the admission surface.

Runner ownership matters too. A validation job consumes untrusted branch content, downloads schemas and may execute renderers before Kubeconform sees their output. Scope credentials and persistence with self-hosted GitHub Actions runner controls rather than treating a reusable shell account as an isolated build boundary.

Read Four Result States Before Writing the Gate

Kubeconform’s JSON summary separates valid, invalid, errors and skipped. Those fields represent different owners and should trigger different CI actions.

The current usage reference defines -strict as rejecting additional properties and duplicated keys. It defines -ignore-missing-schemas separately: missing-schema files are skipped instead of causing failure. Combining both flags therefore does not make coverage strict; it makes known schemas strict while permitting unknown schema identities to become skipped.

Here is the representative receipt produced by the complete lab. The first line pins the tool and target schema version; the remaining lines preserve the negative controls and final gate decision.

kubeconform=v0.8.0 kubernetes_schema=1.33.0
strict_unknown_field=rejected strict_duplicate_key=rejected
unknown_cr_without_schema=error ignore_missing_exit=0 skipped=1
fail_closed_gate_before=42 custom_schema_gate_after=0 valid=3
missing_image_schema_valid=yes api_server_and_policy_validation=still_required
cleanup=complete path_absent=yes

Pin One Validator and One Kubernetes Schema Version

Run every tested block below in one fresh Bash session. The first input refuses an existing path, creates an owner marker, downloads the v0.8.0 Linux archive plus release checksums, verifies the archive and extracts only into a mode-0700 workspace. The v0.8.0 release was published on June 4, 2026; a later version should receive its own reviewed checksum and behavior receipt instead of inheriting this result.

set -Eeuo pipefail
lab_root=/tmp/voxfor-kubeconform-166
owner_marker="$lab_root/.voxfor-kubeconform-166"
owner_token=voxfor-kubeconform-166-v1
version=v0.8.0
asset=kubeconform-linux-amd64.tar.gz
kubeconform_bin="$lab_root/bin/kubeconform"

test ! -e "$lab_root"
for tool in curl sha256sum tar python3; do command -v "$tool" >/dev/null; done
install -d -m 0700 "$lab_root/bin" "$lab_root/downloads" \
  "$lab_root/manifests" "$lab_root/schemas" "$lab_root/evidence"
printf '%s\n' "$owner_token" >"$owner_marker"
trap 'rc=$?; if (( rc != 0 )); then printf "failure_evidence_retained=%s\n" "$lab_root" >&2; fi' EXIT

curl -fsSLo "$lab_root/downloads/$asset" \
  "https://github.com/yannh/kubeconform/releases/download/$version/$asset"
curl -fsSLo "$lab_root/downloads/CHECKSUMS" \
  "https://github.com/yannh/kubeconform/releases/download/$version/CHECKSUMS"
(
  cd "$lab_root/downloads"
  grep "  $asset\$" CHECKSUMS | sha256sum -c -
)
tar -xzf "$lab_root/downloads/$asset" -C "$lab_root/bin" kubeconform
test "$($kubeconform_bin -v)" = "$version"
printf 'tool=%s asset_sha256=%s\n' "$($kubeconform_bin -v)" \
  "$(sha256sum "$lab_root/downloads/$asset" | awk '{print $1}')"

Pin -kubernetes-version to the API version family that receives the artifact, not automatically to master. A repository targeting more than one supported cluster version should run an explicit matrix and retain a receipt for each target. Schema caches can reduce external dependencies, but cached bytes need ownership, review and invalidation just like the validator binary.

Workspace isolation is deliberately modest here. A private directory prevents accidental file reuse; it does not create a kernel or tenant security boundary. Use container and VM isolation boundaries when choosing where untrusted render steps may execute.

Make Strict Mode Prove Two Separate Failures

Start with one ordinary Deployment, then add the misspelled field imagePullPolciy. Default validation accepts that additional property in this reproduced path. Strict mode rejects it and returns 1. The test stores both outputs, so a future tool or schema change cannot quietly alter the expected difference.

cat >"$lab_root/manifests/valid.yaml" <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: receipt-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: receipt-api
  template:
    metadata:
      labels:
        app: receipt-api
    spec:
      containers:
        - name: app
          image: nginx:1.29.1-alpine
          ports:
            - name: http
              containerPort: 80
YAML
cp "$lab_root/manifests/valid.yaml" "$lab_root/manifests/unknown-field.yaml"
python3 - "$lab_root/manifests/unknown-field.yaml" <<'PY'
from pathlib import Path
import sys
p = Path(sys.argv[1])
s = p.read_text()
p.write_text(s.replace(
    '          image: nginx:1.29.1-alpine\n',
    '          image: nginx:1.29.1-alpine\n          imagePullPolciy: Always\n'
))
PY
"$kubeconform_bin" -kubernetes-version 1.33.0 -summary \
  "$lab_root/manifests/unknown-field.yaml" >"$lab_root/evidence/unknown-default.txt"
set +e
"$kubeconform_bin" -kubernetes-version 1.33.0 -strict -summary \
  "$lab_root/manifests/unknown-field.yaml" >"$lab_root/evidence/unknown-strict.txt" 2>&1
unknown_strict_rc=$?
set -e
test "$unknown_strict_rc" -eq 1
grep -q 'imagePullPolciy' "$lab_root/evidence/unknown-strict.txt"
printf 'unknown_field default=accepted strict=rejected strict_rc=%s\n' "$unknown_strict_rc"

Strictness catches misspelled additional properties only when the selected schema knows where properties are allowed. It does not make every semantic requirement appear in OpenAPI, and it cannot validate an unknown kind without a schema.

Reject duplicate keys before the last value wins

YAML permits text that some loaders interpret with “last key wins” behavior. Kubeconform strict mode rejects the duplicated replicas key during YAML-to-JSON conversion. That is a different control from the misspelled field: one prevents ambiguous source representation, while the other enforces a schema’s property surface.

python3 - "$lab_root/manifests/valid.yaml" "$lab_root/manifests/duplicate-key.yaml" <<'PY'
from pathlib import Path
import sys
s = Path(sys.argv[1]).read_text()
Path(sys.argv[2]).write_text(s.replace(
    '  replicas: 1\n',
    '  replicas: 1\n  replicas: 2\n'
))
PY
set +e
"$kubeconform_bin" -kubernetes-version 1.33.0 -strict -summary \
  "$lab_root/manifests/duplicate-key.yaml" >"$lab_root/evidence/duplicate-strict.txt" 2>&1
duplicate_rc=$?
set -e
test "$duplicate_rc" -eq 1
grep -Eqi 'duplicat|replicas' "$lab_root/evidence/duplicate-strict.txt"
printf 'duplicate_key strict=rejected strict_rc=%s\n' "$duplicate_rc"

Validate the artifact that will actually be submitted. For Helm or Kustomize, render with reviewed environment values first and pipe or stage that output for Kubeconform. Validating only source fragments can miss defects introduced by overlays, substitutions or conditional templates. Xentoo’s rendered-manifest CI example makes the same important distinction between a rendered stream and every file in a directory.

Mark What Offline OpenAPI Cannot Prove

The next control removes the container image. Kubeconform v0.8.0 reports the Deployment as schema-valid against Kubernetes 1.33.0. That result is useful precisely because it prevents an overclaim: offline JSON-schema acceptance does not cover every validation performed later by Kubernetes.

python3 - "$lab_root/manifests/valid.yaml" "$lab_root/manifests/missing-image.yaml" <<'PY'
from pathlib import Path
import sys
s = Path(sys.argv[1]).read_text()
Path(sys.argv[2]).write_text(s.replace('          image: nginx:1.29.1-alpine\n', ''))
PY
"$kubeconform_bin" -kubernetes-version 1.33.0 -strict -summary -output json \
  "$lab_root/manifests/missing-image.yaml" >"$lab_root/evidence/missing-image.json"
python3 - "$lab_root/evidence/missing-image.json" <<'PY'
import json, sys
j = json.load(open(sys.argv[1]))
assert j['summary'] == {'valid': 1, 'invalid': 0, 'errors': 0, 'skipped': 0}, j
print('missing_image schema=accepted valid=1 limitation=server_semantics_unproved')
PY

Kubeconform’s validation overview explicitly says controllers and server-side checks perform validations outside the OpenAPI specifications it consumes. Kubernetes documents kubectl apply --dry-run=server as submitting the request without persisting the resource; the current kubectl apply reference also distinguishes strict, warn and ignore field validation.

Runtime acceptance extends even further. A schema cannot prove a Service selector finds a ready Pod, a named port resolves to a listener or a request returns the expected body. The reproduced Kubernetes Service backend path is one example of controller and workload evidence that begins after source-schema admission.

Turn a Missing Custom Schema Into Work, Not Success

Custom resources extend the Kubernetes API with kinds outside built-in schemas. Kubernetes requires structural OpenAPI schemas for apiextensions.k8s.io/v1 CRDs, but an offline validator still needs a matching converted schema location. The official CRD documentation explains how the API server uses that schema for validation and field pruning.

This synthetic Widget belongs to a reserved invalid domain and never reaches a cluster. Without a matching schema, default behavior is an error and exit 1. Adding -ignore-missing-schemas changes the same object to skipped=1 with exit zero.

cat >"$lab_root/manifests/widget.yaml" <<'YAML'
apiVersion: example.voxfor.invalid/v1
kind: Widget
metadata:
  name: receipt-widget
spec:
  message: ready
YAML
set +e
"$kubeconform_bin" -strict -summary -output json \
  "$lab_root/manifests/widget.yaml" >"$lab_root/evidence/widget-error.json"
widget_error_rc=$?
set -e
test "$widget_error_rc" -eq 1
"$kubeconform_bin" -strict -ignore-missing-schemas -summary -output json \
  "$lab_root/manifests/widget.yaml" >"$lab_root/evidence/widget-skipped.json"
python3 - "$lab_root/evidence/widget-error.json" "$lab_root/evidence/widget-skipped.json" <<'PY'
import json, sys
err, skipped = (json.load(open(p)) for p in sys.argv[1:])
assert err['summary']['errors'] == 1 and err['summary']['skipped'] == 0, err
assert skipped['summary']['errors'] == 0 and skipped['summary']['skipped'] == 1, skipped
print('unknown_cr default=error error_rc=1 ignore_missing=exit_0 skipped=1')
PY

Skipping can be intentional for generated list wrappers or a reviewed kind validated elsewhere. If so, record the exact GVK, owner, alternative check and expiration. A blanket ignore flag without an inspected skipped count makes later schema additions and removals invisible.

Add the schema, then aggregate every count

The final admission input creates a small local Draft 4 JSON schema for this one synthetic kind. Production teams should derive and review schemas from the owning CRD version, store them with change control, and bind a schema revision to the controller release. A public CRD catalog can be a discovery source; it should not silently redefine an internal API.

Three manifests enter the gate: a valid Deployment, the missing-image semantic-boundary Deployment and the Widget. Before the additional schema location is supplied, Kubeconform exits zero but the Python aggregator exits 42 because one resource was skipped. After the Widget schema is supplied, the summary becomes valid=3 invalid=0 errors=0 skipped=0 and the same aggregator accepts it. The missing-image object remains schema-valid, preserving the separate server-side limitation instead of hiding it.

cat >"$lab_root/schemas/widget_v1.json" <<'JSON'
{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "additionalProperties": false,
  "required": ["apiVersion", "kind", "metadata", "spec"],
  "properties": {
    "apiVersion": {"type": "string", "enum": ["example.voxfor.invalid/v1"]},
    "kind": {"type": "string", "enum": ["Widget"]},
    "metadata": {
      "type": "object",
      "additionalProperties": false,
      "required": ["name"],
      "properties": {"name": {"type": "string", "minLength": 1}}
    },
    "spec": {
      "type": "object",
      "additionalProperties": false,
      "required": ["message"],
      "properties": {"message": {"type": "string", "minLength": 1}}
    }
  }
}
JSON
install -d -m 0700 "$lab_root/admission"
cp "$lab_root/manifests/valid.yaml" "$lab_root/manifests/missing-image.yaml" \
  "$lab_root/manifests/widget.yaml" "$lab_root/admission/"

run_gate() {
  local report=$1
  shift
  local validator_rc=0
  "$kubeconform_bin" -kubernetes-version 1.33.0 -strict \
    -ignore-missing-schemas -summary -output json "$@" \
    "$lab_root/admission" >"$report" || validator_rc=$?
  test "$validator_rc" -eq 0
  python3 - "$report" <<'PY'
import json, sys
j = json.load(open(sys.argv[1]))
s = j['summary']
print(f"summary valid={s['valid']} invalid={s['invalid']} errors={s['errors']} skipped={s['skipped']}")
if s['invalid'] or s['errors'] or s['skipped']:
    raise SystemExit(42)
PY
}

set +e
run_gate "$lab_root/evidence/gate-before.json" >"$lab_root/evidence/gate-before.txt" 2>&1
gate_before_rc=$?
set -e
test "$gate_before_rc" -eq 42
run_gate "$lab_root/evidence/gate-after.json" \
  -schema-location default \
  -schema-location "$lab_root/schemas/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json" \
  >"$lab_root/evidence/gate-after.txt"
grep -q 'summary valid=3 invalid=0 errors=0 skipped=0' "$lab_root/evidence/gate-after.txt"
printf 'gate before=reject_skipped(rc=%s) after=accept_all_schemas limitation=missing_image_still_schema_valid\n' \
  "$gate_before_rc"

JSON output is the stable decision surface here; pretty output remains useful to humans but should not be parsed with color codes or sentence fragments. Keep the original reports as build artifacts so reviewers can distinguish a manifest defect from a schema-source outage.

Image identity needs another gate. A syntactically valid image: value does not prove that a registry serves the intended digest, that credentials work or that retention preserves rollback bytes. Continue with a Harbor registry recovery workflow when the release decision depends on private image availability.

Use the Receipt in CI Without Overclaiming

Accept offline manifest admission only when the checksum-verified Kubeconform version and target Kubernetes schema version match the reviewed policy; strict mode rejects both negative controls; the custom-resource run records an error without its schema and a skipped result under ignore-missing; the fail-closed parser rejects any nonzero invalid, error or skipped count; the complete schema set reports valid=3 invalid=0 errors=0 skipped=0; and cleanup proves the marker-owned fixture is absent. This receipt says every submitted resource matched an available offline schema, not that Kubernetes will admit or run it.

Pipeline order should follow evidence ownership:

1. Render the exact environment artifact without production credentials.
2. Run pinned Kubeconform against the deployment target version and reviewed custom schemas.
3. Reject invalid, errors and skipped counts; retain JSON reports.
4. Run policy checks that express organization requirements not present in OpenAPI.
5. Use --dry-run=server against an authorized representative cluster when admission behavior matters.
6. Deploy through a change boundary, then prove controller and workload readiness.

For traffic-layer changes, place this source admission before Gateway API parallel cutover rather than treating valid YAML as proof that routes, certificates and backends work together.

The final tested block records the receipt and deletes only the exact owned path after its marker matches. It neither uninstalls a global tool nor changes any kubeconfig.

printf '%s\n' \
  "kubeconform=$version kubernetes_schema=1.33.0" \
  'strict_unknown_field=rejected strict_duplicate_key=rejected' \
  'unknown_cr_without_schema=error ignore_missing_exit=0 skipped=1' \
  'fail_closed_gate_before=42 custom_schema_gate_after=0 valid=3' \
  'missing_image_schema_valid=yes api_server_and_policy_validation=still_required' \
  | tee "$lab_root/evidence/receipt.txt"
grep -qx "$owner_token" "$owner_marker"
find "$lab_root" -depth -delete
trap - EXIT
test ! -e "$lab_root"
printf 'cleanup=complete path_absent=yes\n'

When any assertion fails, keep the last admitted manifest bundle unchanged and do not deploy the new artifact. The exit trap retains only /tmp/voxfor-kubeconform-166 for diagnosis; review its JSON reports, fix the owning manifest, schema source or pinned tool version, and rerun from a fresh marker-owned path. Remove that retained directory only after its exact owner token matches. Production rollback belongs to the deployment system’s last known-good artifact and change record—not to disabling strict mode, ignoring more kinds, deleting a shared schema cache or bypassing API-server admission.

Keep API-Server Admission and Runtime Proof Separate

A complete schema receipt removes one class of uncertainty: every rendered resource was parsed and checked against an available pinned schema. It does not prove image availability, admission webhooks, namespace policy, controller defaults, scheduling, readiness or a user request.

Layer the decision instead of weakening it. Offline schema coverage should be fail-closed, server-side admission should use authorized representative state, and runtime acceptance should test the actual workload path. A green result at one layer is evidence for that layer—not permission to rename skipped, untested or later-stage behavior as success.