Silent CPU Fallback in Ollama: How to Detect and Prevent CUDA Spillover A developer's technical guide documents how Ollama silently offloads transformer layers from GPU VRAM to system RAM when a model and its expanding KV cache exceed available CUDA memory, dropping throughput from roughly 35-45 tokens per second to about 1.2 tokens per second without any error. The guide shows how to detect the spillover using `ollama ps` (checking the PROCESSOR column for anything other than 100% GPU) and server logs that report layer offload counts, and explains how default context windows of 32K to 128K tokens pre-allocate VRAM before the first prompt. You pull a 14B or 32B model in Ollama. On the first prompt, response generation is blazing fast—delivering a smooth 35 to 45 tokens per second. But on your third turn, after feeding in a 3,000-token function definition or log trace, generation suddenly grinds to a halt. The model stutters, your CPU fans rev to 100%, and token generation plummets to a painful 1.2 tokens per second . Your GPU has not crashed, and Ollama hasn't thrown an error. What you are witnessing is silent CPU fallback —one of the most frustrating performance traps in local AI engineering. In this technical guide from The Zero-Dollar Dev https://thezerodollardev.blogspot.com/ , we break down why Ollama silently offloads transformer layers from VRAM to system RAM, how to verify it instantly with CLI diagnostics, and the exact configuration flags to lock your models into GPU memory permanently. When Ollama initializes an LLM, its runner built on llama.cpp inspects your system's available CUDA or Metal memory. If the entire model cannot fit into contiguous GPU VRAM, the runtime does not throw an out-of-memory OOM error by default. Instead, it partitions the model's neural network layers between two distinct memory spaces: | Memory Domain | Hardware Bus | Peak Theoretical Bandwidth | Latency Profile | |---|---|---|---| | GPU VRAM GDDR6X | Direct On-Die Memory Bus e.g. RTX 3080/4080 | 760 – 1,008 GB/s | Sub-microsecond tensor transfers | | System RAM DDR4 / DDR5 | PCIe Gen4 / Gen5 Bus x16 slot | 38 – 64 GB/s PCIe bottleneck: ~31.5 GB/s | Severe bus transit latency | Notice the massive discrepancy: GPU VRAM is 15x to 25x faster than system DDR5 RAM , and the PCIe interconnect creates a severe data transfer bottleneck. During inference, every single autoregressive token requires forward passes through all layers sequentially. If even 2 out of 48 layers are offloaded to system RAM, the entire inference pipeline stalls at every layer transition while weights and activation tensors are copied back and forth across the PCIe bus. Because Ollama does not print warning messages to standard output when offloading layers, you must inspect the runtime engine directly. While your model is running a query, open a second terminal and execute: ollama ps Examine the PROCESSOR column in the output: NAME ID SIZE PROCESSOR UNTIL qwen2.5-coder:14b b38006e8e6e5 9.0 GB 68%/32% CPU/GPU 4m52s If you see anything other than 100% GPU , silent CPU fallback is actively degrading your throughput. Ollama logs the exact layer allocation breakdown during model initialization. To view this: On Linux systemd journalctl -u ollama --no-pager -n 50 | grep -E "load tensors|offloaded" On macOS cat ~/.ollama/logs/server.log | grep -E "load tensors|offloaded" On Windows PowerShell Get-Content "$env:LOCALAPPDATA\Ollama\server.log" -Tail 100 | Select-String -Pattern "offload" You will see log entries detailing layer assignments: llm load tensors: offloaded 34/48 layers to GPU llm load tensors: 14 layers remaining on CPU llm load tensors: VRAM used: 7842 MB, CPU RAM used: 3120 MB When you start Ollama, it might fit 40 layers into an 8GB GPU, leaving only 300MB of free VRAM. As your conversation lengthens to 8,000 tokens, the Key-Value KV cache dynamically expands requiring up to 2.5GB of additional memory . The CUDA memory allocator fails, forcing the runtime to evict layer tensors or spill dynamic activations to host system memory. See our deep dive: Llama 3.3 70B VRAM Calculation Math https://thezerodollardev.blogspot.com/2026/11/llama-33-70b-on-consumer-gpu-vram.html . In modern GGUF models, the default context window num ctx can be 32,768 or even 128,000 tokens. Ollama attempts to pre-allocate memory structures for this full context window, consuming multiple gigabytes of VRAM before the first prompt token is even processed. Operating systems, display servers Xorg/Wayland/Windows DWM , and open browser tabs hardware-accelerated by Chromium consume between 600MB and 1.5GB of GPU VRAM. If Ollama calculates available VRAM without accounting for display buffer spikes, it over-allocates and trips fallback. By default, Ollama stores KV-cache tensors in 16-bit floating point precision FP16 . By switching to 8-bit quantized KV-cache Q8 0 , you instantly cut KV-cache VRAM consumption in half with zero loss in coding precision: Linux / macOS export OLLAMA FLASH ATTENTION=1 export OLLAMA KV CACHE TYPE=q8 0 Windows Persistent System Environment Variable System.Environment ::SetEnvironmentVariable 'OLLAMA FLASH ATTENTION', '1', 'Machine' System.Environment ::SetEnvironmentVariable 'OLLAMA KV CACHE TYPE', 'q8 0', 'Machine' Never leave num ctx unbound when operating near your GPU's VRAM ceiling. Create a custom Modelfile to set a deterministic context limit: FROM qwen2.5-coder:14b-instruct-q4 K M Pin context window to 8,192 tokens to guarantee deterministic VRAM headroom PARAMETER num ctx 8192 Pin GPU layer offload count e.g. 48 for full offload PARAMETER num gpu 48 Low temperature for precise code generation PARAMETER temperature 0.2 PARAMETER top p 0.95 Build and run the locked model: ollama create qwen14b-locked -f ./Modelfile ollama run qwen14b-locked To automatically detect when an inference query spills over to CPU and restart Ollama before your machine locks up, save this lightweight Bash watcher: bash /usr/bin/env bash ollama-watchdog.sh - Detect and alert on CPU fallback while true; do PROCESSOR=$ ollama ps | awk 'NR 1 {print $4}' if "$PROCESSOR" =~ "CPU" ; then echo " ALERT Silent CPU Fallback Detected: $PROCESSOR" notify-send "Ollama Fallback" "Model running on CPU: $PROCESSOR" fi sleep 5 done Explore our complete library of local AI hardware benchmarks: