When etcd Raises NOSPACE, Compact Before You Defragment

An etcd NOSPACE alarm means at least one member’s backend crossed its configured quota, so the cluster entered limited-operation maintenance mode. Reads and deletes may still work, but normal writes are rejected. Safe recovery is not “clear the alarm and hope”: preserve quorum and a snapshot, compact old MVCC history, defragment members one at a time, prove backend headroom, then disarm and test a real write.

That order comes from etcd’s current maintenance guidance. Compaction and defragmentation are different operations. Compaction makes obsolete revisions reusable inside the backend; defragmentation rewrites one member’s bbolt file so unused pages return to the host filesystem. Skipping either distinction can leave the cluster alarmed or produce a short-lived recovery that fills again.

A self-managed cluster assembled on VPS infrastructure with full root control gives operators access to the control plane and storage, but it also leaves etcd maintenance with the operator. Managed Kubernetes products may hide etcd entirely; follow the provider’s recovery process instead of trying to reach an internal datastore.

Read NOSPACE as maintenance mode, not ordinary disk full

The alarm is tied to etcd backend quota, not simply to df showing a full filesystem. A member can have free host disk and still exceed --quota-backend-bytes. Conversely, a full host filesystem can break WAL or snapshot writes before the quota is the first visible limit. Check both layers.

From the client side, Kubernetes may look partly alive: cached reads or API reads can succeed while creates, updates, lease renewals and controller progress fail. That asymmetry is the clue. The etcd v3.6 maintenance guide says quota maintenance mode accepts key reads and deletes while rejecting normal updates until enough space is freed, the backend is defragmented and the alarm is cleared.

Operational ownership should already be explicit. Teams deciding who maintains control-plane certificates, snapshots, member health and storage can use managed-versus-unmanaged VPS responsibilities to document the boundary before an incident.

Freeze the incident boundary before touching space

Do not delete member/snap/db, remove WAL files, restart every member, or raise quota blindly. First capture which endpoints are reachable, which member is leader, whether a learner exists, what alarm each member reports, and whether quorum is healthy enough for maintenance.

Establish authenticated endpoint identity

Use the client certificate and endpoint paths from the running service or static-pod manifest. The placeholders below deliberately avoid assuming kubeadm, RKE2, Talos or OpenShift paths:

export ETCDCTL_API=3
export ETCDCTL_ENDPOINTS="https://cp1.example.net:2379,https://cp2.example.net:2379,https://cp3.example.net:2379"
export ETCDCTL_CACERT="/secure/path/etcd-ca.crt"
export ETCDCTL_CERT="/secure/path/etcd-client.crt"
export ETCDCTL_KEY="/secure/path/etcd-client.key"

Keep the key file root-readable and run this work from a protected control-plane session. Then save the first receipt:

etcdctl member list -w table
etcdctl endpoint status --cluster -w table
etcdctl endpoint health --cluster
etcdctl alarm list

Stop and escalate if quorum is already lost, endpoint identity is uncertain, members disagree materially on applied state, or more than one member is unavailable. NOSPACE maintenance is not a substitute for disaster recovery.

Failure domains also matter. KVM hypervisor and host-isolation guidance helps explain why three etcd VMs placed on one physical host do not provide three independent failure domains.

Take and inspect a live snapshot

Choose one healthy endpoint, save a current snapshot to protected storage with enough free space, and verify its metadata. etcd’s disaster-recovery documentation notes that etcdctl snapshot save captures a live member, while etcdutl snapshot status reports the revision, hash, key count and size.

SNAPSHOT="/secure-backups/etcd-pre-nospace-CHANGE-ID.db"
etcdctl --endpoints="https://cp1.example.net:2379" snapshot save "$SNAPSHOT"
etcdutl snapshot status "$SNAPSHOT" -w table

Do not continue with an empty, unreadable or unverified snapshot. Also confirm the backup path does not share the nearly exhausted filesystem. A snapshot is a safety artifact; restoring it is a separate quorum-loss workflow, especially for Kubernetes where informer caches need revision-aware handling.

Compaction and defragmentation solve different geometry

Three measurements answer different questions. Modern etcdctl endpoint status output can expose database size and in-use size; current metrics expose etcd_mvcc_db_total_size_in_bytes and etcd_mvcc_db_total_size_in_use_in_bytes. Compare them with the configured quota and host free space.

According to etcd’s persistent-storage reference, the bbolt file never shrinks on its own. During defragmentation a replacement db.tmp... file is created and later replaces the original. Filesystem headroom is therefore a prerequisite, even when the in-use number suggests plenty of reclaimable space.

Recover one cluster in a controlled sequence

Compaction is a cluster-wide logical operation. Defragmentation is member-specific and can block that member while its backend is rewritten. Run the compact step once, then defragment one endpoint at a time with a health gate between members.

Record a revision and compact once

Read a current revision from one healthy endpoint, print it into the change record, and compact at that exact value. The jq expression is shown so the chosen revision is visible rather than hidden inside a long pipeline:

REVISION="$(etcdctl --endpoints="https://cp1.example.net:2379" endpoint status -w json | jq -r '.[0].Status.header.revision')"
echo "$REVISION"
etcdctl --endpoints="https://cp1.example.net:2379" compact "$REVISION"

Compaction makes revisions before the selected point unavailable to old reads and watchers. Use a revision confirmed from the current cluster and preserve the output. Re-running compaction at arbitrary values does not create more filesystem space by itself.

Defragment members sequentially

Start with one healthy follower when topology allows, wait for completion, then confirm cluster health before moving on. Repeat for the remaining follower and leader. Do not paste a --cluster defrag into a stressed production control plane merely because it is shorter.

etcdctl --endpoints="https://cp2.example.net:2379" defrag
etcdctl endpoint health --cluster
etcdctl --endpoints="https://cp3.example.net:2379" defrag
etcdctl endpoint health --cluster
etcdctl --endpoints="https://cp1.example.net:2379" defrag
etcdctl endpoint health --cluster

etcd documents defrag as a per-member operation so cluster-wide latency spikes can be avoided. For planned work on Kubernetes nodes around this procedure, Kubernetes node-maintenance readiness checks provide a separate application-mobility gate; draining a control-plane node is not automatically required for online etcd defrag.

Prove headroom before disarming

Re-run status and alarm commands. The backend must be below its quota with enough operating margin for the expected write rate, and every member must return healthy before the alarm is cleared.

etcdctl endpoint status --cluster -w table
etcdctl endpoint health --cluster
etcdctl alarm list
etcdctl alarm disarm
etcdctl alarm list

If total and in-use size remain close to quota, disarming is not recovery; the next burst can raise NOSPACE again. Investigate actual object churn, oversized values, retention requirements or a compaction policy before changing quota.

Prove normal writes without hiding a quorum problem

Health output alone is not the workload acceptance test. Create one uniquely named, harmless verification key, read it back through the cluster endpoints and remove exactly that key. Record the change ID so another operator cannot confuse it with application state.

CHECK_KEY="/maintenance/etcd-nospace-verify-CHANGE-ID"
etcdctl put "$CHECK_KEY" "ok"
etcdctl get "$CHECK_KEY"
etcdctl del "$CHECK_KEY"

For Kubernetes, follow with a representative API write that your change procedure permits, then verify the responsible controller observes it. Avoid creating production workloads merely for proof. Acceptance requires alarm clear, all endpoints healthy, member agreement, one successful write/read/delete cycle and resumed controller progress.

Rollback has a hard boundary here. Compaction cannot restore old MVCC history, and defrag rewrites member storage; the snapshot protects against a later recovery decision but does not make these actions reversible in place. If a member becomes unhealthy, stop the sequence and recover cluster health before touching another member.

Build headroom around churn, not a larger quota

Prevention starts with automatic compaction matched to watcher and retention needs. etcd v3.6 supports periodic and revision-based auto-compaction through --auto-compaction-mode and --auto-compaction-retention. No universal duration is safe: short windows reduce history but can break slow watchers; long windows preserve history at greater storage cost.

Monitor every member’s total backend size, in-use size, configured quota, leader changes, proposal failures, WAL fsync latency and backend commit latency. etcd metrics reset when a member restarts, so dashboards and alerts need a scraper that retains history outside the process. Metric hygiene matters too; Prometheus label-cardinality diagnosis helps prevent the monitoring system from becoming another capacity incident.

Quota increases belong after root-cause and capacity review. A larger backend can lengthen snapshots, defrag, recovery and disk I/O work. Raising the limit without correcting uncontrolled Kubernetes object churn, short-lived leases, large values or missing compaction merely moves the alarm.

Snapshots also need off-host retention and deletion separation. MinIO Object Lock backup design explains the repository-side boundary when an etcd recovery artifact must survive source-host loss or compromised uploader credentials.

Close with an incident receipt, not an empty alarm list

The durable output is a short record containing the alarmed member, endpoint status, snapshot path/hash/revision, compact revision, defrag order and duration, size before and after each member, disarm result, write acceptance, Kubernetes/controller proof, retained headroom and prevention owner.

That receipt turns a risky storage intervention into repeatable DevOps work. Recovery is complete only when writes work, quorum is healthy and recurrence controls have an owner—not when alarm list happens to print nothing.