vLLM Is Preempting Requests: Budget the KV Cache Before Raising Concurrency

Sequence group ... is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. That warning does not mean the vLLM process has crashed or the GPU is out of all memory. It means the scheduler could not keep every active sequence’s key-value state in the cache, so vLLM released one request’s cache blocks and will recompute them later.

Occasional preemption can be a deliberate recovery mechanism. Frequent preemption is different: the server remains available while latency and GPU work quietly increase. Diagnose that mismatch by measuring cache capacity, live request demand, and scheduler admission in the same load window. Raise no limit until those three budgets agree.

Preemption is recovery, not a CUDA OOM

Autoregressive generation retains attention state for tokens already processed. vLLM stores that state in a paged KV cache so many sequences can share GPU memory efficiently. When active sequences require more blocks than the cache can hold, the scheduler may preempt a sequence and later recompute its missing state. Current vLLM optimization guidance explicitly warns that repeated recomputation can hurt end-to-end latency.

A CUDA out-of-memory failure crosses another boundary. An allocation fails and the worker may terminate, reject startup, or abort a request. Preemption instead keeps the engine moving by spending extra compute. Separate the two before tuning:

• Preemption evidence: vLLM warning or metric increases while the server continues responding.
• Allocation failure: CUDA OOM text, worker exit, failed health check, or restart.
• Host pressure: system RAM, cgroup, or container OOM evidence rather than accelerator-cache exhaustion.

Model placement still matters. If GPU utilization looks wrong before traffic even arrives, compare device assignment and offload evidence with Ollama GPU offload diagnosis; do not assume an inference process on a GPU is using the expected accelerator path.

Three budgets determine the real concurrency ceiling

No single max_num_seqs value defines safe throughput. A stable vLLM server balances static GPU allocation, variable KV demand, and admitted work.

Static allocation leaves the cache remainder

Model weights, CUDA graphs, kernels, temporary activations, and other runtime allocations consume memory before KV blocks are available. --gpu-memory-utilization tells vLLM what fraction of GPU memory the model executor may use; it is not a target for observed application traffic and should not be pushed to 1.0 without considering co-resident processes and runtime variation.

Current vLLM releases can infer cache size from that fraction or accept an explicit --kv-cache-memory-bytes CLI byte budget. The CacheConfig reference exposes the corresponding configuration field as kv_cache_memory_bytes and treats the explicit byte setting as the stronger cache-size control. A value copied from another GPU, model, quantization, or boot state is not portable.

Capacity planning begins with VPS sizing guidance for local LLMs when CPU, RAM, storage, context length, and accelerator availability are still undecided. For an existing vLLM node, use its startup log and actual model configuration instead of a generic memory calculator.

Live KV demand grows with tokens in flight

Prompt tokens and generated tokens held by active sequences consume cache. Two requests with the same concurrency can have radically different demand when one workload sends 512-token prompts and another approaches the model’s context ceiling. Prefix caching may reuse computed blocks for matching prefixes, but it does not make arbitrary long conversations free.

--max-model-len limits the maximum context vLLM will accept. It does not reserve that many tokens for every request, yet a very high ceiling allows individual requests to occupy far more cache. Admission tests must therefore use the real prompt and output-length distribution rather than only average concurrency.

Scheduler limits decide how much work may compete

--max-num-seqs caps scheduled sequences in a batch, while --max-num-batched-tokens bounds tokens processed together. Lowering either can reduce simultaneous cache pressure. The cost is potential throughput loss or longer waiting time, so the correct value is the one that meets the service objective—not the smallest value that removes a warning.

Capture one comparable pressure window

First save the exact launch configuration, vLLM version, model identifier, tensor/pipeline parallel sizes, cache dtype, model-length ceiling, and free GPU memory before startup. Avoid recording API keys or request text.

vllm --version
nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free \
  --format=csv,noheader,nounits
ps -eo pid,etimes,args | grep '[v]llm serve'

Next run one representative load stage while sampling vLLM’s metrics endpoint. Metric names can change across releases, so preserve the raw output and discover current names rather than hard-coding a dashboard from an old blog post.

curl -fsS http://127.0.0.1:8000/metrics > /tmp/vllm-metrics.before
# Run a fixed prompt/output/concurrency stage here.
curl -fsS http://127.0.0.1:8000/metrics > /tmp/vllm-metrics.after
grep -Ei 'preempt|kv_cache|num_requests|queue|time_to_first_token|e2e' \
  /tmp/vllm-metrics.after

Correlate cumulative preemption, KV-cache usage, running and waiting requests, queue time, time to first token, inter-token latency, end-to-end latency, throughput, GPU utilization, and errors. The vLLM metrics design reference defines the maintained exporter surface and its lifecycle. When forwarding those series into Prometheus, use high-cardinality label diagnosis before attaching request IDs, prompts, tenants, or unbounded model attributes.

One peak gauge cannot prove the cause. A valid pressure window uses synchronized timestamps and a repeatable workload, because cache use near 100 percent may be efficient when preemption and tail latency remain controlled.

Let the evidence choose one lever

Change only one variable per experiment and keep the previous launch command as rollback. Otherwise an apparent improvement cannot be attributed, and a later regression becomes harder to reverse.

More cache memory: preserve headroom

Increasing --gpu-memory-utilization can make more memory available to vLLM and therefore to KV cache. Use that lever only when nvidia-smi and process evidence show durable free headroom, no co-tenant needs it, and startup/runtime allocation remains stable. A higher fraction can convert recomputation into a fatal allocation failure if the margin disappears.

An explicit --kv-cache-memory-bytes CLI budget is useful when repeated boots on identical hardware need deterministic cache sizing. Current vLLM engine-argument documentation notes that an explicit cache byte value overrides automatic inference. In Python or config contexts, the same setting appears as kv_cache_memory_bytes; keep that spelling separate from the command-line flag. Re-profile after changing GPU, model, quantization, runtime, parallelism, or co-resident workload.

Lower admission: trade throughput for latency stability

If cache demand rises with active long requests, test a lower --max-num-seqs first. If large prefills dominate, test a lower --max-num-batched-tokens. Use separate benchmark stages so you can see whether preemption falls at the cost of queue time or total tokens per second.

Current vLLM v0.26.0 also exposes --watermark, which keeps a configured fraction of KV-cache blocks free when admitting waiting or preempted requests; 0.0 disables that headroom. If repeated eviction and readmission is the symptom, test this version-gated control as its own experiment, confirm it exists with vllm serve --help, and measure the queue, latency, preemption, and throughput tradeoff.

vllm serve MODEL \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192

Those numbers are an experiment example, not universal recommendations. Replace MODEL and values with the previously captured baseline, validate the command against the installed version’s vllm serve --help, and preserve the old unit or container definition for rollback.

Parallelism buys memory with communication

Tensor parallelism can shard model weights across GPUs and leave more per-GPU room for cache. Pipeline parallelism can also change placement. Neither is free: communication, synchronization, topology, and load balance can reduce throughput or increase latency. Review GPU infrastructure tradeoffs before treating another accelerator as an automatic repair.

CPU offload or lower-precision cache can conserve GPU memory, but each introduces performance, compatibility, or accuracy considerations. Follow current vLLM memory-conservation guidance and benchmark the exact model/hardware pair rather than stacking every memory-saving flag.

Maximum context and safe concurrency answer different questions

--max-model-len answers, “What is the longest request this endpoint may accept?” Admission controls answer, “How many token-heavy requests may compete now?” Reducing maximum context can protect a service whose product does not need extremely long prompts, but it is a contract change and may reject existing clients.

Conversely, leaving a large context ceiling does not guarantee that several maximum-length requests can coexist. Enforce request-token limits at the API boundary, document them, and return a clear client error before expensive work begins. Place LiteLLM provider-key isolation in front of model backends when clients should not hold upstream credentials; that gateway still needs explicit token and concurrency policy.

Automatic prefix caching improves work reuse for shared prefixes. It may increase effective throughput on suitable traffic, but it does not replace an admission contract and should not be counted as guaranteed capacity for unrelated prompts.

Publish a measured operating envelope

Run a stepped benchmark across representative prompt sizes, output limits, and concurrency levels. Hold every other server setting constant. For each stage, record completed requests, errors, preemption delta, cache peak, queue latency, p50/p95/p99 time to first token, p95 end-to-end latency, and output tokens per second.

curl -fsS http://127.0.0.1:8000/metrics > /tmp/vllm-stage-01.metrics
nvidia-smi --query-gpu=timestamp,index,utilization.gpu,memory.used,memory.free \
  --format=csv,noheader > /tmp/vllm-stage-01.gpu.csv

Accept a configuration only when the intended workload completes without process OOM, preemption stays at the team’s documented tolerance, latency objectives hold, queue growth stops after the stage, and a repeated run produces comparable results. A zero-preemption target may be appropriate for strict latency; another service may deliberately accept rare recomputation to gain throughput.

Expose the chosen bounds in deployment configuration and monitoring. If Caddy owns the public TLS endpoint, follow Caddy zero-downtime reload procedure when changing upstream or timeout configuration; proxy tuning cannot create missing KV capacity.

The durable output is a capacity envelope: model and runtime version, hardware identity, cache budget, context ceiling, scheduler limits, request distribution, benchmark results, alert thresholds, and rollback command. Re-test it after upgrades or model changes instead of carrying old concurrency numbers forward.