GPU Memory Planning for Local LLM Inference

GPU memory planning starts with one equation: required VRAM = model weights + KV cache + runtime reserve. A checkpoint that fits by file size can still fail when several requests hold long contexts, while a model that loads successfully can still miss its latency target. The useful result is therefore not a GPU name. It is a capacity envelope tied to one model build, serving engine and traffic shape.

This article is for developers and technical builders choosing hardware for local or self-hosted inference. You need the model card and expected request shape, but not deep CUDA knowledge. By the end, you will have a reviewable estimate, a hardware shortlist and an acceptance test that can disprove the estimate before money or production traffic depends on it.

Three Memory Pools Define the Fit

Weights establish the fixed floor. During inference, the selected checkpoint parameters must be available to the serving runtime at their loaded precision. A 7.61-billion-parameter model loaded at 16 bits needs roughly 15.22 GB in decimal units for weights alone. GPU vendors usually advertise decimal gigabytes, while operating tools may report binary gibibytes; 15.22 GB is about 14.17 GiB. Write the unit beside every number so a conversion error does not consume the safety margin.

KV cache creates the variable part. A transformer reuses keys and values from previous tokens rather than recalculating all of them during each decode step. NVIDIA’s current TensorRT documentation exposes the relevant cache dimensions: batch, KV heads, maximum sequence and head dimension. The higher-level framework or application still owns allocation, which is why two engines can show different total use around the same theoretical cache.

Runtime reserve covers what the two neat formulas do not. The engine needs workspaces, temporary tensors, allocator space and sometimes CUDA graphs or other optimized execution structures. Hugging Face’s GPU memory anatomy also distinguishes stored tensors from temporary operation peaks. Training adds gradients and optimizer state that ordinary inference does not need, so a training-memory rule should not be copied into an inference purchase.

Treat a percentage such as 10% or 20% as an early filter, not a guarantee. Framework version, attention backend, quantization method, GPU architecture and enabled features change the measured peak. Reserve is the difference between observed peak and physical capacity after the representative run, not a universal constant.

Collect Workload Inputs Before Comparing GPUs

Start with the artifact that will actually run. Record its exact repository and revision, parameter count, loaded weight dtype or quantization, layer count, KV-head count, attention-head count, hidden size and supported context. A label such as “7B” is convenient marketing shorthand; it does not reveal whether the model uses multi-head attention, grouped-query attention or a mixture-of-experts architecture.

Next, describe traffic in tokens rather than users. One registered user may send nothing, while one agent can hold several active sequences. The capacity sheet needs:

• maximum input tokens admitted by the service;
• maximum generated tokens reserved for a response;
• simultaneously active sequences at the target percentile;
• expected prompt-length distribution, not only the advertised context maximum;
• target time to first token and decode throughput;
• one representative burst duration.

The distinction between configured context and occupied context matters. Some runtimes reserve cache pools during startup; others allocate blocks as sequences grow. Use documentation for the engine you will deploy, then inspect its own startup and load evidence. When vLLM is the engine, the reported KV-cache capacity and maximum concurrency provide a more useful acceptance boundary than a generic VRAM calculator.

Deployment architecture belongs on the same sheet. Decide whether the GPU also drives a display, shares memory with another model, or hosts embeddings and reranking beside generation. For a broader choice among local cards, rented accelerators and mixed capacity, compare GPU server, cloud GPU and hybrid infrastructure only after the workload envelope is written.

Calculate Weights and KV Cache Separately

Weight memory is the simpler line:

weight bytes = parameter count × loaded bits per parameter ÷ 8

That calculation is a floor. Quantized checkpoints can include scales, zero points or tensors kept at a different precision, and the serving format may add metadata. Prefer the checkpoint’s measured loaded footprint when available. File size can be a useful cross-check, but compression, sharding and memory mapping prevent it from being a universal runtime measurement.

KV cache needs architecture-specific inputs:

KV bytes = 2 × layers × KV heads × head dimension × bytes per cache element × cached tokens × active sequences

The factor of two represents keys and values. Head dimension is commonly hidden size divided by the number of attention heads, but verify the configuration rather than assuming. Grouped-query attention can use far fewer KV heads than query heads, sharply reducing cache per token. Parameter count alone therefore cannot compare two models’ long-context capacity.

Three scenarios make the estimate useful:

Do not multiply every request by the model’s advertised maximum unless the service truly admits and retains that many tokens. Equally, do not size only for the average when a small number of long conversations can occupy the cache pool. Admission limits are capacity controls, not merely API preferences.

For a simple Ollama deployment, the adjacent host-sizing guide for local LLMs helps inventory CPU, system RAM and storage that sit outside this GPU-memory calculation. Keep those budgets separate so spare host RAM is not mistaken for full-speed VRAM.

Worked Example from an Open Model Configuration

Consider Qwen2.5-7B-Instruct as a reproducible arithmetic example, not a recommendation. Its official model card lists 7.61 billion parameters, 28 layers, 28 attention heads and four KV heads. The public configuration file gives hidden size 3,584 and BF16 as the declared dtype. Dividing 3,584 by 28 gives a head dimension of 128.

At BF16, weights create a theoretical floor of 7.61 billion × 2 bytes = 15.22 GB, or about 14.17 GiB. The KV cache consumes 2 × 28 × 4 × 128 × 2 = 57,344 bytes per cached token per active sequence. That is exactly 56 KiB.

The cache then scales predictably:

Those subtotals still exclude runtime reserve. A nominal 24 GB card may appear comfortable in the first two cases and uncomfortably close in the third, especially if the display, another process or the engine owns additional memory. “It loads” is not proof that the busy scenario fits, and the table does not predict tokens per second.

Long-context claims need another check. The model card describes support beyond the configuration’s default position limit through additional scaling configuration and warns that the setting can affect shorter text. Capacity planning must use the context actually enabled and validated in the serving stack, not the largest number printed on a landing page.

Turn the Estimate into a Hardware Shortlist

Reject any option whose usable memory is below the busy subtotal plus a measured reserve. For the remaining candidates, compare memory bandwidth, supported numeric formats, reliability features, power, cooling, interconnect and hourly or ownership cost. A GPU can satisfy capacity and still deliver poor latency; another can have enough aggregate memory across several devices but suffer from an unsuitable topology.

Multi-GPU planning requires the engine’s parallelism model. Tensor parallelism can split weights across devices, but cache placement, communication and uneven memory use influence the real fit. Treat aggregate VRAM as an inventory number, not automatic shared memory. If one process needs two devices, validate that exact topology rather than adding card labels in a spreadsheet.

Quantization changes more than the weight line. Verify that the checkpoint format, kernels and GPU support the selected precision, and evaluate output quality on representative tasks. Weight quantization does not automatically quantize KV cache; the two decisions have different compatibility and quality boundaries. The current vLLM tuning guidance makes the trade-off explicit: cache sizing affects achievable batching and concurrency, while an optimistic fixed value can fail allocation.

Infrastructure cost should be compared only among configurations that pass the same test. A cheaper card that forces a smaller model, shorter context or lower concurrency is a different service. If retrieval-augmented generation is part of the design, budget vector payloads, indexes and optimizer workspace through a separate Qdrant storage-capacity model; disk headroom does not belong in the VRAM total.

Prove Capacity with a Representative Acceptance Run

Load the exact checkpoint and engine version on a shortlisted GPU. Capture idle GPU memory after startup, then send requests that reproduce the ordinary, busy and guardrail scenarios. Record peak memory, accepted active sequences, input and output tokens, time to first token, decode throughput, errors, preemptions and rejected requests.

Success needs an observable boundary: the busy scenario completes within the latency target, no request is silently moved to an unintended slow path, and free memory remains above the team’s declared reserve. The guardrail scenario should reject or queue excess work predictably before an out-of-memory crash. When Ollama unexpectedly uses CPU or only partially offloads a model, use the processor-placement diagnosis before blaming the estimate.

If vLLM reports preemption under realistic concurrency, the problem may be cache capacity rather than weight fit. Follow the vLLM KV-cache and concurrency investigation to decide whether to reduce admitted context, reduce active sequences, change cache precision where supported, or move to more capacity. Raising utilization without measuring the next peak merely narrows the failure margin.

Keep serving security outside the memory equation but inside deployment acceptance. A model endpoint that meets throughput while distributing provider or gateway keys to every client has passed the wrong test. When the design fronts several models or providers, the LiteLLM credential-boundary guide shows how to separate client access from upstream credentials.

Keep a Four-Line Capacity Receipt

Preserve four lines beside the deployment decision: exact model revision and precision; weight and KV-cache arithmetic with units; busy scenario with context and active sequences; measured peak, latency and remaining reserve. Add the engine version, GPU model and test date so the next upgrade can rerun the same boundary.

That receipt turns “this GPU should fit” into a falsifiable claim. When the model, context policy, concurrency or serving engine changes, update the relevant line and repeat the busy test before expanding traffic.