Ceph Nearfull Can Begin on One OSD

A Ceph cluster can report OSD_NEARFULL while its aggregate raw-use percentage still looks comfortable. Fullness belongs to the individual OSD boundary, not to the cluster average. One device above the configured nearfull ratio is enough to raise the health check, even when most peers retain substantial free space.

That difference changes the capacity question. Instead of asking only “How full is Ceph?”, ask which eligible OSD is fullest, how much room its failure domain retains, and whether a replacement destination can accept backfill. A percentage that looks safe before an OSD failure may be too high after Ceph needs to place another replica.

One maximum can invalidate a comfortable average

Consider ten equal-size OSDs. Nine are 69% used and one is 86% used. Their simple average is 70.7%, yet the 86% OSD can cross a live 85% nearfull boundary. Aggregate RAW USE communicates fleet consumption; it cannot describe this local maximum.

The same arithmetic becomes more consequential inside CRUSH. If the fuller OSD belongs to a device class or rack with no alternative destination below backfillfull, space elsewhere in the cluster may be ineligible for the affected pool. Usable recovery capacity is constrained by placement rules and failure domains, not merely by raw bytes on every disk.

Current Ceph health-check documentation separates three controls. OSD_NEARFULL warns, OSD_BACKFILLFULL prevents new backfill from using an overfilled destination, and OSD_FULL applies the full boundary. Placement Group state definitions identify backfill_toofull when a selected destination is already beyond its backfillfull ratio.

Five measurements define the real headroom

Capacity review needs five related measurements rather than one dashboard percentage:

Read them from one UTC snapshot so the OSD map does not change between observations:

incident_dir="/root/ceph-capacity-$(date -u +%Y%m%dT%H%M%SZ)"
install -d -m 0700 "$incident_dir"
date -u --iso-8601=seconds | tee "$incident_dir/time.txt"
ceph -s > "$incident_dir/status.txt"
ceph health detail > "$incident_dir/health-detail.txt"
ceph osd df tree > "$incident_dir/osd-df-tree.txt"
ceph osd tree > "$incident_dir/osd-tree.txt"
ceph df detail > "$incident_dir/df-detail.txt"
ceph pg stat > "$incident_dir/pg-stat.txt"
ceph osd dump > "$incident_dir/osd-dump.txt"
ceph balancer status > "$incident_dir/balancer-status.txt"

From osd-dump.txt, record the live ratios and flags. noout, nobackfill, norecover, norebalance, or pause can intentionally suppress movement. A flag with an active maintenance owner is different from forgotten state, so do not clear it merely because an alert is visible.

Compare OSDs only inside a meaningful peer group: same device class, intended CRUSH role, similar capacity, and eligibility for the same pools. A small SSD database/WAL device and a large data OSD do not become peers just because both appear in one tree.

Each fullness state removes a different option

Nearfull is not a write outage by itself. It is a warning that a specific OSD has crossed the monitor’s nearfull ratio. The cluster can still be active+clean, but that state proves current placement—not spare capacity for a future failure.

Backfillfull changes what Ceph can do next. An OSD above this ratio is not accepted as a backfill destination. When a PG reports backfill_toofull, data movement is waiting on the selected destination’s capacity boundary. Calling that condition “slow recovery” misses the constraint.

Full is the later protection boundary. Treating these ratios as three cosmetic alert levels is unsafe because the middle threshold protects room for redistribution. Raising it spends space that Ceph may need precisely when an OSD or host disappears.

Exact values are configuration and distribution decisions. Read the live cluster instead of copying defaults from another Ceph release or vendor-integrated platform. Managed products may also require their own supported command path for ratio changes.

Match the response to the capacity shape

Skew among healthy comparable OSDs

When one OSD is materially fuller than comparable peers, review PG placement, CRUSH capacity weight, override reweight, device class, and recent topology changes. The cluster may have enough eligible bytes but an uneven map.

Use non-mutating evaluation before moving data:

ceph balancer status
ceph balancer eval
ceph osd test-reweight-by-utilization --no-increasing
ceph progress

The Ceph balancer module can optimize PG placement automatically or under supervision. Ceph control-command guidance distinguishes temporary override reweights from CRUSH weights that express device capacity. Do not combine OSD override reweights with an active balancer: current Ceph guidance requires override values to stay at 1.0000, recommends the default upmap mode when no clients predate Luminous, and notes that the balancer optimizes PG shard counts rather than actual byte utilization. Review proposed OSD and PG movement before applying a reweight, and preserve every before-value needed for rollback.

One class, host, or rack lacks eligible room

Similar utilization across an entire CRUSH subtree points away from one-device skew. Verify the pool’s CRUSH rule, failure-domain type, device-class filter, and whether expected peers are both up and in. Space in another class or outside that rule cannot receive the affected replicas.

Ceph OSD troubleshooting guidance places monitor quorum and network checks before a narrow OSD diagnosis. During movement, apply workload discipline from Voxfor’s RAID recovery latency budget. A capacity correction that drives foreground storage latency outside its service objective needs a lower movement rate or a better maintenance window.

Comparable OSDs are uniformly close to the boundary

Even placement means balancing cannot manufacture free bytes. Data deletion must use the owning application or a supported Ceph interface with snapshot, retention, and replica consequences understood. Never delete files directly from an OSD data directory.

New capacity must join the correct CRUSH rule and failure domain before it helps affected pools. When the shortage is real bytes rather than mapping skew, capacity planning must compare additional storage nodes by disk layout and network throughput, as well as CRUSH failure-domain placement. Budget replicas or erasure-code overhead, BlueStore metadata, expected growth, and failure recovery.

Voxfor’s optimizer working-space model illustrates the same wider principle: live data can fit while required maintenance movement does not. Ceph needs eligible destinations with room to accept backfill after a fault.

Protect the recovery contract while space moves

Three shortcuts change risk without correcting the underlying capacity shape:

• Raising ratios first delays or suppresses a boundary but adds no bytes and fixes no placement.
• Lowering pool size or min_size changes durability or write availability rather than capacity ownership.
• Marking a healthy full OSD out casually can evacuate data toward peers with even less safe margin.

Before risky movement, ensure recent recovery artifacts match the workloads using Ceph. Proxmox operators should preserve a verifiable guest-consistency backup contract instead of assuming every completed VM archive is application-consistent.

Local storage faults also change the decision. If a nearfull OSD reports filesystem or block errors, preserve kernel and device evidence, then follow Voxfor’s ext4 fail-closed repair workflow where that filesystem owns the failing path. Rebalancing around a failing device and repairing an overfilled but healthy map are different incidents.

A temporary ratio change is defensible only when affected PGs, eligible destination bytes, projected movement, workload impact, rollback values, and the next durable capacity action are documented. The threshold change itself is never the acceptance result.

Turn recovery into a capacity policy

After any correction, recalculate the same five measurements: maximum OSD use, peer spread, live ratios, eligible bytes, and failure-domain reserve. PGs should leave backfill_toofull or recovery_toofull, no new OSD should cross a higher boundary, client latency should remain inside budget, and temporary flags or overrides need owners plus removal times.

Capacity policy should define two triggers. An imbalance trigger starts placement review when comparable OSD spread becomes abnormal. An expansion trigger starts procurement or data-lifecycle work while every relevant failure domain still has room for the next expected recovery. Keeping those triggers separate prevents balancing tools from being used against true exhaustion.

Ceph nearfull is resolved when every affected placement rule retains measurable recovery space—not when a warning disappears from one ceph -s screen.