Low Host Utilization Does Not Rule Out Container CPU Throttling

Low host CPU utilization does not prove that a container has CPU time available. A cgroup v2 limit in cpu.max is enforced at that cgroup boundary even when other cores look quiet. Prove the local ceiling first by locating the process’s exact cgroup, sampling cpu.stat over the slow interval, and matching new throttling events to latency or incomplete work.

That evidence separates three problems that need different fixes: a hard CPU bandwidth quota, competition between runnable workloads, and hypervisor contention outside the guest. Raising a limit helps only the first. More host capacity may help the second or third, while changing cpu.weight cannot override a finite cpu.max ceiling.

A CPU limit is a local time budget

Linux cgroup v2 organizes processes in a hierarchy and distributes resources from parent to child. A lower cgroup cannot escape a stricter ancestor limit. According to the kernel cgroup v2 interface, cpu.max contains two values:

MAX PERIOD

Both values are microseconds. 200000 100000 allows the cgroup to consume 200,000 microseconds of CPU time during each 100,000-microsecond period, equivalent to an average ceiling of two CPUs. max 100000 means that this cgroup has no local maximum, although a parent may still impose one.

Parallel threads can spend the budget quickly. Four runnable threads may consume a two-CPU allowance during the first half of a period and then wait for replenishment. Host-wide monitoring can average that burst away, especially on a machine with many cores. Quota is about CPU time inside the cgroup, not the host’s aggregate idle percentage.

Do not infer an incident from one lifetime counter. cpu.stat accumulates usage, elapsed periods, throttled periods and throttled microseconds since the cgroup was created. A service that was throttled during yesterday’s deployment can retain those totals after today’s symptom has disappeared. The useful evidence is the change during a defined workload window.

Capture the leaf cgroup without restarting it

Work from a host shell with enough permission to inspect the target process. Avoid writing directly to /sys/fs/cgroup during diagnosis; Docker, systemd or Kubernetes may own the desired state and overwrite an ad hoc kernel change.

First confirm the unified hierarchy and resolve one representative process. Replace the PID with the actual service or container init PID.

stat -fc %T /sys/fs/cgroup
PID=12345
CGROUP_REL=$(awk -F: '$1 == "0" {print $3}' "/proc/$PID/cgroup")
CGROUP="/sys/fs/cgroup$CGROUP_REL"
printf 'pid=%s\ncgroup=%s\n' "$PID" "$CGROUP"

cgroup2fs confirms cgroup v2. The kernel documents the unified membership line as 0::$PATH. If the command returns no path, stop and identify whether the host uses cgroup v1 or a hybrid layout; the filenames and units differ, so the v2 procedure does not apply unchanged.

Read the leaf and its ancestors. A container leaf may show max, while its pod, slice or parent service carries the actual ceiling.

date -u +'%Y-%m-%dT%H:%M:%SZ'
printf '%s\n' "$CGROUP"
cat "$CGROUP/cpu.max"
cat "$CGROUP/cpu.stat"
cat "$CGROUP/cpu.pressure"
cat "$CGROUP/cpu.weight"
cat "$CGROUP/cpuset.cpus.effective" 2>/dev/null || true

Repeat the snapshot across a representative slow interval without restarting the workload. Record the deltas for usage_usec, nr_periods, nr_throttled and throttled_usec, plus request latency, queue age or completed jobs. The kernel PSI interface adds CPU stall evidence through cpu.pressure, but pressure alone does not name a quota; it shows that runnable work lacked CPU progress.

What the counters can and cannot prove

An increasing nr_throttled means at least one fair-scheduler bandwidth period hit the cgroup limit. A rising throttled_usec quantifies accumulated throttled duration as reported by the kernel. Interpret both as interval evidence beside workload behavior.

Avoid presenting nr_throttled / nr_periods as the percentage of CPU capacity lost. That ratio describes how often periods experienced throttling, not how much useful work was delayed. Multithreaded workloads and nested cgroups make a universal conversion misleading. A few throttled periods can still hurt tail latency, while frequent events may have little user impact in a background batch job.

Ancestor inspection matters too. Walk upward from the leaf until /sys/fs/cgroup, recording cpu.max and cpu.stat at every existing level. Treat every finite leaf or ancestor budget as a candidate constraint. Identify the limiting level from the applicable quota, interval counter changes and sibling demand under the observed workload; a parent budget may be stricter than the leaf and may be shared by several children.

Three comparisons identify the owner

Finite quota plus new throttle events

A finite cpu.max, increasing throttle counters and matching service stalls form the strongest quota case. Confirm that the slow phase actually demands more CPU than the average budget. Short parallel bursts are a common pattern: average host CPU stays low, yet the cgroup exhausts its allowance early in repeated periods.

Look for useful completion, not process activity. An application can consume its full budget in retries, garbage collection or spin without improving throughput. CPU usage rising beside flat completed-work rate points to application inefficiency as well as a quota boundary.

Unlimited quota plus CPU pressure

When the leaf and every relevant ancestor show max, bandwidth throttling is not the cause. Rising CPU pressure, run queue and latency instead point toward runnable-task competition, affinity constraints or too few effective CPUs. cpu.weight changes relative distribution only when workloads compete; it does not reserve CPU and does not create a hard ceiling.

Check cpuset.cpus.effective before assuming the process can use every host core. Affinity, cpuset policy or orchestrator CPU management can legitimately narrow the available set even with unlimited cpu.max.

Guest looks constrained but the cgroup does not

Virtual machines add another boundary. If cpu.max is unlimited, throttle counters stay flat and the guest still loses runnable time, compare the same interval with CPU steal diagnosis. Steal time means the hypervisor scheduled other work while the guest wanted CPU; cgroup throttling is enforced inside the guest. Do not raise an application quota to treat host-level steal.

Workload-specific evidence remains necessary after the CPU boundary is identified. For containerized WordPress, continue with PHP-FPM worker saturation evidence because a busy pool, slow dependency or undersized queue can persist after throttling is removed. In event processing, Kafka partition-lag diagnosis shows whether CPU pressure belongs to the assigned consumer or whether one producer key, rebalance or broker path owns the delay.

Trace the kernel value back to its owner

The safe change belongs in the system that declared the limit. A direct write may disappear on restart, conflict with reconciliation, or bypass policy review.

Docker’s resource constraint reference equates --cpus=1.5 with a 150,000-microsecond quota in a 100,000-microsecond period. It also describes CPU shares as a soft, contention-dependent priority. Record the existing inspect output and deployment source before changing either.

For systemd services, resource-control settings map CPUQuota= to a maximum and CPUWeight= to relative allocation. Inspect both the service and its slice. A temporary systemctl set-property --runtime test can isolate causality, but the rollback value and persistent unit/drop-in remain the real configuration contract.

Kubernetes makes the scheduling/enforcement split explicit: CPU requests influence placement while CPU limits are throttled by the kernel. Inspect the workload template, LimitRange or policy that supplied the value before editing a live Pod. Controller-managed Pods will be recreated from the template, and cluster policy may reapply a default.

Change one contract and keep isolation visible

Change one resource contract at a time: raise a hard limit, remove it only when policy and host reserve allow, reduce unnecessary parallelism, or use relative weight when the actual requirement is priority during contention. Keep requests, placement and downstream capacity in view; extra CPU can move the bottleneck to a database, queue or rate-limited API.

Before release, preserve the old quota/period, weight, cpuset, replica count and workload configuration. Define an explicit rollback trigger such as worse tail latency, host pressure above the agreed bound, neighbor degradation, error growth or no increase in useful completions. A greener throttle counter is not enough if service output stays flat.

Sustained demand may expose a capacity decision rather than a bad limit. Dedicated CPU topology and hardware choices matter when the workload needs predictable parallel execution; consult dedicated CPU hardware guidance to compare cores, cache, NUMA and storage after measuring the real demand. Measured demand can then guide VPS hosting capacity without treating one burst or one lifetime counter as a sizing forecast.

Finish with a mixed acceptance window

Re-run the exact baseline capture under a representative burst, not an idle health check. Compare quota and period, throttle deltas, CPU pressure, effective CPUs, host run queue and steal time, then pair them with the application’s p95/p99 latency, queue age, successful work and dependency health.

Pass only when the intended service outcome improves without moving risk to the host or a neighbor. If throttle counters fall but latency does not, restore the prior contract and continue at the application or dependency boundary. If useful work improves while host reserve remains inside policy, persist the change in the owning platform and record the old value, new value, evidence window and rollback trigger.