Why Ollama Uses CPU Even When Your GPU Is Available

nvidia-smi can show a healthy GPU while an Ollama request still runs partly—or entirely—on CPU. That contradiction does not have one universal cause. The loaded model’s PROCESSOR field is the starting evidence: it separates full GPU placement, full CPU placement and a mixed split before you touch drivers, packages or model files.

Several inputs decide the result. The host must expose a supported accelerator, the Ollama server process must be allowed to discover it, the request must not force a conflicting runner placement, and enough usable VRAM must remain for the model plus its allocated context. A container adds another visibility boundary because a GPU visible on the host is not automatically visible inside it.

Begin with the live request, not a remembered configuration. An Ollama sizing guide can help you compare model weight, RAM and VRAM before buying capacity; VPS specs for Ollama and local LLMs covers that planning intent. Here, the task is narrower: identify why the current loaded model chose CPU, GPU or both.

Read the placement Ollama actually chose

Ollama’s current FAQ defines ollama ps as the authoritative quick check for loaded-model placement. Run a real request first, then inspect the process list while the model remains loaded:

ollama list
ollama run REPLACE_WITH_INSTALLED_MODEL "Reply with the word ready"
ollama ps

The PROCESSOR column describes where the model was loaded. A result such as 100% GPU means the loaded model is entirely on GPU memory. 100% CPU means system memory and CPU execution. A mixed value such as 48%/52% CPU/GPU means Ollama split placement between system and GPU memory.

Do not diagnose from Task Manager, top or one utilization percentage alone. Tokenization, request handling and other work can use CPU even when model layers are on the GPU. Conversely, a visible GPU process does not prove the whole model fits. Placement and utilization answer different questions.

Freeze one request across every evidence source

A useful snapshot must describe the same request window. Record the model name, quantization or tag, allocated context, Ollama version, service log, device inventory and GPU memory state before restarting anything.

ollama --version
ollama ps
systemctl status ollama --no-pager
journalctl -u ollama --no-pager -n 200

On NVIDIA hosts, capture both the inventory and the live process view:

nvidia-smi -L
nvidia-smi

For AMD on Linux, the equivalent boundary includes ROCm device discovery and permissions. Current Ollama documentation recommends checking rocminfo, /dev/kfd, /dev/dri and server logs when acceleration is missing:

rocminfo | sed -n '1,120p'
ls -lnd /dev/kfd /dev/dri /dev/dri/*
journalctl -u ollama --no-pager -n 200

Capture first because a restart can erase the most useful discovery message. The official Ollama troubleshooting guide lists Linux logs, GPU-discovery errors and platform-specific recovery paths. Treat those messages as evidence about a boundary, not as permission to try every workaround at once.

A healthy host GPU is only the first boundary

When nvidia-smi fails on the host, Ollama cannot repair the host driver or missing hardware assignment. Resolve that layer first. Current Ollama hardware support documents supported NVIDIA compute capabilities and driver floors, plus the AMD ROCm and Vulkan paths that apply to current releases.

Virtualization changes what “available” means. A provider dashboard can name a GPU product while the guest lacks direct device access, the required virtual function or a supported passthrough path. Compare the deployment with Voxfor’s GPU server, cloud GPU and hybrid GPU guide before changing software inside a guest that never received an accelerator.

Physical ownership matters as well. Dedicated hosts expose different limits around device assignment, VRAM, PCIe topology and driver control; Voxfor’s dedicated server hardware map provides the broader hardware context. No Ollama setting can create passthrough that the hypervisor or provider withheld.

Supported is not the same as visible

A card may appear in PCI inventory yet remain unusable by the selected backend. For NVIDIA, Ollama documents compute-capability and driver requirements. For AMD, current Linux guidance requires a compatible ROCm v7 driver for the documented ROCm path, while Vulkan expands support with its own permissions and VRAM-reporting considerations.

Avoid copying an override meant for another GPU family. Variables such as CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES and GGML_VK_VISIBLE_DEVICES can deliberately hide or select devices. An inherited value of -1, an invalid index or an unstable integrated-GPU selection can explain why the server behaves differently from an interactive shell.

The daemon may not inherit your working shell

nvidia-smi succeeding as your SSH user proves that user’s host view. Ollama installed as a systemd service normally runs under another identity with its own groups, environment and startup timing. Read the effective unit before adding an override:

systemctl show ollama -p User -p Group -p SupplementaryGroups -p Environment -p ExecStart
systemctl cat ollama

Look for device-selection variables, a forced OLLAMA_LLM_LIBRARY, custom paths and service hardening that blocks device access. Redact tokens, private endpoints and other sensitive environment values before sharing the output. A manual ollama serve test launched from the shell can differ because it inherits your login environment. That difference is useful evidence, but it is not a production fix.

Correct the service boundary, not the login profile

When evidence proves the service needs one environment change, place it in a systemd drop-in rather than .bashrc. Use systemctl edit ollama.service, add only the required setting under [Service], then reload and restart during a controlled window. The exact value must come from the detected device and current Ollama documentation; do not paste another host’s GPU index.

[Service]
Environment="CUDA_VISIBLE_DEVICES=GPU-REPLACE_WITH_DISCOVERED_UUID"

sudo systemctl daemon-reload
sudo systemctl restart ollama
ollama run REPLACE_WITH_INSTALLED_MODEL "Reply with the word ready"
ollama ps

Keep the previous unit output with the incident. If placement or service health worsens, remove the new drop-in line, reload systemd and restart the service. One override, one restart and one acceptance request creates a reviewable change; a driver reinstall plus new model plus several environment variables does not.

Mixed placement often points to memory demand

A split PROCESSOR value proves that Ollama is using both CPU and GPU memory. Before treating that as failed discovery, compare the model’s loaded size and context with free VRAM during the same window.

Ollama’s context-length documentation states that larger context increases memory requirements and directs operators to verify both CONTEXT and PROCESSOR through ollama ps. Current defaults can vary by release and available VRAM, so the live CONTEXT column is safer evidence than a remembered default.

Parallel requests multiply the context allocation for a loaded model. The current Ollama FAQ explains that required memory scales with parallel request count and context length. Other loaded models, display workloads and unrelated GPU processes can reduce the remaining VRAM even when the model fitted during an earlier test.

Change demand before changing drivers

Run a controlled single-request comparison with the same prompt and model. Stop idle Ollama models you intentionally no longer need, confirm competing GPU processes, and test an approved smaller context or model variant. Do not kill unknown GPU processes on a shared host.

ollama ps
nvidia-smi
ollama stop REPLACE_WITH_IDLE_MODEL

Flash Attention and K/V-cache quantization can change memory demand on supported backends, but they also change compatibility or precision tradeoffs. Apply those settings only after the processor split and context evidence prove a capacity problem. A smaller context may be unacceptable for coding, agent or retrieval workloads; fitting entirely in VRAM is not success if the application loses required context.

Docker creates another device boundary

Host GPU success does not cross a container boundary automatically. An Ollama container needs the correct runtime/device exposure, and the process inside the container must be able to query the device. Current Ollama Docker guidance uses NVIDIA Container Toolkit for NVIDIA acceleration and explicit device mappings for AMD.

First prove the runtime independently of Ollama. Ollama’s troubleshooting documentation recommends a disposable NVIDIA check such as:

docker run --rm --gpus all ubuntu nvidia-smi
docker exec REPLACE_WITH_OLLAMA_CONTAINER ollama ps
docker logs --tail 200 REPLACE_WITH_OLLAMA_CONTAINER

Replace the container placeholder with the name returned by docker ps; the common name ollama is not guaranteed. If the disposable check fails, repair the container runtime boundary before recreating the Ollama application. NVIDIA’s Container Toolkit installation guide is the current source for configuring Docker or another supported runtime.

Compose users should inspect the effective declaration rather than assuming a YAML file was applied. Voxfor’s Docker versus Docker Compose guide explains the deployment boundary; the Ollama service still needs explicit accelerator access in the rendered configuration.

Compare the client request with the direct backend path

Open WebUI, an agent or a custom API client can change more than the route and timeout. Ollama’s current API types expose load-time runner options including num_gpu, main_gpu and num_ctx; an advanced client setting can therefore force CPU placement, limit GPU layers or increase memory demand. Inspect the client’s model options or outbound request before dismissing the UI as presentation-only. The Ollama API type definitions are the current primary reference for those load-time fields.

Compare like with like. Stop the model intentionally, load it once through the client, record ollama ps, then repeat with the same model and context through the direct local CLI. A placement difference under the same service and device state points to request options or client-selected model configuration; an identical result sends the investigation back to service, memory or container evidence.

Voxfor’s Open WebUI deployment workflow is a useful next step once backend placement is proven. Keeping those tasks separate prevents a reverse-proxy or browser symptom from being misdiagnosed as GPU fallback.

Close the incident with placement and workload proof

Successful repair needs more than a changed percentage. Repeat the same prompt, context and concurrency used in the baseline; record ollama ps, service logs, device memory, latency and the application’s real response. Watch the host for system-memory pressure because partial offload can shift load rather than remove it.

A direct-hardware environment becomes the relevant infrastructure decision when the guest cannot receive a supported accelerator or the team needs deterministic driver and device control. Evaluate that hosting model after the software boundaries are proven, not as a substitute for diagnosis.

Store the before/after evidence with four outcomes: GPU visible to the host, GPU visible to the Ollama service or container, expected PROCESSOR placement, and representative workload acceptance. That record makes rollback possible and stops the next operator from repeating a blind reinstall.

Continue from the evidence, not the symptom

For more adjacent deployment and operations guidance, browse Voxfor’s AI and hosting field guides.