You update Ollama, pull the model you use every day, and everything gets slower. Not broken, not erroring. Just slow. ollama ps
says something like 12% CPU / 88% GPU
, so you shrug and carry on. Then you open nvidia-smi mid-generation and see Ollama using 0 MiB of VRAM.
It's been running on CPU this whole time. Nobody told you.
This is what's happening to a lot of RTX 30-series owners after Ollama 0.32.14, and to people with A40/A6000/A10 cards too.
The giveaway is buried in the server log. Somewhere in the noise:
msg="skipping CUDA device - compute capability not in compiled architectures"
device="NVIDIA RTX A6000" cc=860 archs="[750 890 1000 1200]"
cc=860 means your GPU is compute capability 8.6. The arch list is the set of architectures the bundled CUDA kernels were compiled for: 7.5 (RTX 20-series), 8.9 (RTX 40-series), 10.0 and 12.0. No 8.6. Your card isn't in the build.
Affected hardware: every sm_86 card. RTX 3090, 3080, 3070, 3060, plus the A40, A6000, A5000, A10 and friends. A big chunk of the people running local LLMs are on exactly these cards.
The missing sm_86 kernel isn't new. Older builds skipped the same arch. But they had a safety net: when the CUDA 13 kernels didn't cover the card, they fell back to the bundled CUDA 12 library, which does include sm_86. Same skip line, then:
msg="inference compute" ... library=CUDA compute=8.6
In 0.32.14 that fallback path broke. So instead of dropping to the CUDA 12 lib, the runner skips straight to library=cpu
. Nothing errors, so you don't notice until the token rate starts to hurt.
Upstream hasn't shipped a fix as of writing. The issue is open and the only maintainer reply so far is a request for more logs. The dependable move: pin to the last version where the CUDA 12 fallback worked, which for this one is 0.32.13.
Windows:
ollama
service if you run it as one.Linux:
sudo systemctl stop ollama
sudo systemctl start ollama
This is the step everyone skips, and honestly the reason this post exists. ollama ps
can lie. It showed a GPU split while the GPU sat idle. The reliable check:
nvidia-smi
during a generation. VRAM allocated to the ollama process, nonzero, means real GPU.library=CUDA compute=8.6
again, not library=cpu
.CUDA_VISIBLE_DEVICES=0
. The GPU is visible. It's being skipped at the arch check, which is compile-time. No env var brings a missing arch back.OLLAMA_LOAD_LIBRARY
. Some builds respect it, but in 0.32.14 the fallback path itself was the broken part, so don't count on it for this version.After any update to a local-model stack (Ollama, llama.cpp, LM Studio, whatever), spend 30 seconds proving acceleration before you trust it. One nvidia-smi
during a generation is the cheapest smoke test there is.
Silent degradation is worse than an error. An error tells you something's wrong. A silent CPU fallback just makes everything sluggish and you end up blaming the model.
And prebuilt-kernel coverage is decided at build time. When a release drops your GPU architecture, there's no flag that fixes it. Pin the last good version, watch the release notes, and move back up when the coverage returns.
The thread that started this: https://github.com/ollama/ollama/issues/17841