Before You Drain a Kubernetes Node, Prove the Workload Can Move

A Kubernetes node drain is safe only when the workload can complete four separate moves: the Eviction API may remove the Pod, another eligible node can place it, the replacement becomes Ready, and the real service still works. A PodDisruptionBudget (PDB) addresses the first condition. It does not create replicas, storage access, scheduler capacity, topology spread, a correct readiness probe, or an application-level acceptance test.

That distinction matters before kernel updates, hardware work, node-image replacement, autoscaler consolidation, or any other planned worker interruption. A green cluster can still hide two replicas on one node, a replacement that cannot mount its volume, or a single-replica Deployment whose PDB either blocks maintenance forever or permits a complete outage through percentage rounding.

Treat kubectl drain as the execution step, not the readiness test. Maintenance begins with evidence about every affected workload, continues one node at a time, and ends only after an outside client proves the service path.

A PDB answers only the eviction question

Kubernetes separates voluntary disruptions, such as a node drain, from involuntary failures such as a kernel panic, a network partition, or a lost virtual machine. A PDB limits concurrent voluntary evictions for selected Pods when the operator or automation uses the Eviction API. Direct Pod or Deployment deletion can bypass it, and an involuntary loss can still take availability below the budget. The current Kubernetes disruption model states both boundaries explicitly.

Budget status is live arithmetic rather than a static promise. currentHealthy counts selected Pods with Ready=True; desiredHealthy is the availability floor; expectedPods comes from the owning controller; and disruptionsAllowed is the number of additional voluntary evictions currently permitted. A non-zero value proves budget headroom at that moment, not workload mobility.

Start with the target node and the PDB inventory:

kubectl get nodes -o wide
kubectl get pods --all-namespaces --field-selector spec.nodeName=NODE_NAME -o wide
kubectl get poddisruptionbudgets --all-namespaces
kubectl get poddisruptionbudget -n NAMESPACE PDB_NAME -o custom-columns=NAME:.metadata.name,EXPECTED:.status.expectedPods,HEALTHY:.status.currentHealthy,DESIRED:.status.desiredHealthy,ALLOWED:.status.disruptionsAllowed

Read zero as a stop signal that needs an owner. It may mean the budget intentionally permits no downtime, a selected Pod is already unready, the controller has fewer healthy replicas than expected, or a replacement cannot complete startup. Repeated eviction rejection is correct behavior when the request would breach the declared floor.

Percentage values deserve special attention because Kubernetes rounds up. The official PDB configuration guide notes that maxUnavailable: 30% on one desired replica permits disruption of that only Pod: 30 percent rounds up to one. By contrast, minAvailable: 1 on the same workload permits no eviction.

For a three-replica stateless service that can tolerate one unavailable Pod, a budget may look like this:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: production
spec:
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: api

Copying that object without the workload contract would be unsafe. maxUnavailable: 1 fits only when two remaining Ready replicas can carry the service, while AlwaysAllow lets an unhealthy running Pod leave even if the normal budget is already consumed. Current Kubernetes documentation recommends considering AlwaysAllow for drain progress, but the application owner must accept that an unhealthy Pod loses its chance to recover on the original node.

Build a workload-mobility proof before the window

The useful unit of review is not the node alone. Build one small record for every Deployment, StatefulSet, controller-managed Pod group, and special unmanaged Pod found on the target. Record the application owner, selector, desired replicas, PDB, current topology, storage dependency, termination behavior, acceptance probe, and person allowed to stop the window.

Match each budget to its real controller

Selector mistakes produce two dangerous outcomes: a PDB may protect nothing, or it may select more Pods than the operator intended. In policy/v1, an empty selector matches every Pod in the namespace, so copied manifests deserve direct inspection. Overlapping PDBs can also block an eviction because every applicable budget must allow it.

Compare the workload selector with the PDB and the actual Pod labels instead of trusting names:

kubectl get deployment -n NAMESPACE WORKLOAD -o yaml
kubectl get poddisruptionbudget -n NAMESPACE PDB_NAME -o yaml
kubectl get pods -n NAMESPACE -l 'app=WORKLOAD' --show-labels

Names do not establish ownership. Check ownerReferences when a selected Pod does not obviously belong to the expected Deployment or StatefulSet. For operator-managed resources, verify that the controller exposes the scale subresource when using maxUnavailable; arbitrary workloads have stricter PDB rules.

Prove replicas occupy different failure domains

Three replicas do not provide node-maintenance tolerance when all three sit on the worker being drained. Inspect placement and the Ready condition together:

kubectl get pods -n NAMESPACE -l 'app=WORKLOAD' -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,READY:'.status.conditions[?(@.type=="Ready")].status'
kubectl get nodes -L topology.kubernetes.io/zone,kubernetes.io/hostname

Use anti-affinity or topology spread constraints when availability requires replicas across nodes or zones. Placement policy and a PDB solve different problems: the first creates failure-domain diversity; the second meters voluntary removal. Neither substitutes for adequate replica count.

Reserve capacity for the replacement, not only normal traffic

A PDB can approve an eviction while the replacement remains Pending. Resource requests may exceed free CPU or memory; node affinity, taints, topology rules, host ports, or quotas may leave no eligible destination. Local persistent volumes and some storage classes can bind the workload to one node or zone.

Before maintenance, compare requests with allocatable resources and investigate any existing Pending Pod. Scheduler events provide the clearest refusal reason:

kubectl describe node NODE_NAME
kubectl get resourcequota --all-namespaces
kubectl describe pod -n NAMESPACE PENDING_POD
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp

Spare capacity must cover the largest affected scheduling unit, not an average percentage across the cluster. If remaining workers cannot admit it, add or resize capacity before the window. When the cluster needs a larger independent worker pool, compare cloud VPS sizing options against workload requests and the failure-domain plan rather than forcing eviction into an already full scheduler.

Include storage, ports, identity, and termination time

State follows its own mobility rules. A Pod using emptyDir loses that data when evicted; a hostPath or local volume may not follow it; a ReadWriteOnce volume may need detach and attach time; and a StatefulSet replacement keeps identity while waiting for the previous Pod to terminate. Host ports can further restrict placement.

Write those constraints into the maintenance record. Never add --delete-emptydir-data merely to make a command finish. That flag is a data-loss decision and belongs to the application owner. Likewise, --force addresses Pods without a controller; it does not prove that their work will be recreated safely. --disable-eviction is categorically different: the current kubectl drain reference says it forces deletion and bypasses PDB checks, so it must never be used as a shortcut around an availability decision.

Graceful termination needs the same scrutiny. The node-drain procedure honors Pod termination grace periods during an Eviction API drain. Confirm that the application stops accepting new work, drains connections or queues, and exits within the declared period. A long timeout can be correct for a stateful process; an unexplained hang is a reason to stop and diagnose.

Make readiness and service acceptance two different gates

Kubernetes considers a Pod healthy for PDB arithmetic when its Ready condition is true. That signal is necessary, but its quality depends on the readiness probe. A shallow probe may return success before caches are warm, migrations are complete, upstream connections exist, or the Pod can process the real transaction.

Capture a baseline from inside Kubernetes and from a client outside the cluster. The internal side should show selected Pods, EndpointSlices and recent events; the external side should perform a safe read, login, API request, queue publish, or controlled write appropriate to the workload.

kubectl get pods -n NAMESPACE -l 'app=WORKLOAD' -w
kubectl get endpointslices.discovery.k8s.io -n NAMESPACE -l kubernetes.io/service-name=SERVICE_NAME -o wide
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
curl --fail --silent --show-error https://SERVICE_HOST/HEALTH_PATH

Replace the example HTTP check when a health endpoint does not represent the user path. Database workloads may need a read plus a rollback-safe write; workers may need a test job to complete; message systems may need publish and consume evidence. Drain completion and application acceptance answer different questions.

Set an abort condition before starting. Useful boundaries include replacement Pending beyond a measured scheduler interval, Ready replicas below the service floor, elevated external errors, an attachment timeout, or a latency threshold breached for more than a short observation window. One named operator should have authority to pause without negotiating during the incident.

Rehearse the movement without calling it an upgrade

A maintenance rehearsal should test the same ownership and acceptance path without combining it with kernel, runtime, CNI, CSI, or application changes. Choose one representative low-risk worker, schedule a window, preserve the baseline, and move only the workloads that the team can observe end to end.

kubectl drain supports dry-run behavior in current kubectl releases, but a dry run cannot prove that a replacement Pod will actually schedule, attach storage, warm up, receive traffic, and complete a transaction. Use it only as an admission preview after checking the client/server version and command help:

kubectl version
kubectl drain --help
kubectl drain NODE_NAME --ignore-daemonsets --dry-run=server

Real mobility evidence comes from a controlled Eviction API path plus application observation. Do not rehearse by directly deleting a Deployment or Pod and assume the PDB was tested; the Kubernetes disruption documentation warns that direct deletion can bypass the budget.

Single-replica workloads require an explicit choice. Accept a maintenance outage, temporarily build and validate another replica when the application supports concurrency, move the service through an application-specific failover, or set a zero-eviction PDB and require owner coordination. A PDB cannot turn a non-replicated service into highly available software.

Run one bounded maintenance sequence

Once every workload has passed the four proofs, cordon the target so normal scheduling stops, confirm the inventory has not changed, then drain one node at a time. The Kubernetes documentation allows parallel drain commands, but sequential work preserves a clearer failure boundary and avoids consuming multiple disruption budgets or capacity margins at once.

kubectl cordon NODE_NAME
kubectl get pods --all-namespaces --field-selector spec.nodeName=NODE_NAME -o wide
kubectl drain NODE_NAME --ignore-daemonsets --timeout=15m
kubectl get nodes

Do not improvise flags in response to a timeout. Capture the blocked eviction, PDB status, Pending events, volume events and external service result. If the window must stop before maintenance begins, the cordon boundary is reversible:

kubectl uncordon NODE_NAME
kubectl get node NODE_NAME

After the drain returns successfully, perform the planned host work. Keep the node unschedulable until kubelet, runtime, networking, storage and required DaemonSets are healthy. Uncordon only when the node can accept normal work, then observe placement and service behavior through at least one representative workload cycle.

Close with a mobility record the next operator can reuse

A completed ticket should name the node, affected workloads, PDB generation and status, replica topology, spare-capacity evidence, storage constraints, graceful-termination result, external acceptance result, maintenance timestamps, abort owner, and any manual exception. That record turns the next node from a fresh experiment into a controlled comparison.

Capacity or PDB changes made only for the window need a separate keep-or-revert decision. Preserve extra replicas when they close a real availability gap; remove them only after confirming the original service objective still makes sense. Keep AlwaysAllow, topology rules, or probe changes only when their behavior was reviewed outside the pressure of maintenance.

The reliable rule is simple: a node is ready to leave service when its workloads have already proved they can leave it. Continue with Voxfor’s DevOps operations library for related deployment and infrastructure runbooks, while keeping this four-proof record attached to every future drain.