{"slug": "silent-cpu-fallback-in-ollama-how-to-detect-and-prevent-cuda-spillover", "title": "Silent CPU Fallback in Ollama: How to Detect and Prevent CUDA Spillover", "summary": "A developer's technical guide documents how Ollama silently offloads transformer layers from GPU VRAM to system RAM when a model and its expanding KV cache exceed available CUDA memory, dropping throughput from roughly 35-45 tokens per second to about 1.2 tokens per second without any error. The guide shows how to detect the spillover using `ollama ps` (checking the PROCESSOR column for anything other than 100% GPU) and server logs that report layer offload counts, and explains how default context windows of 32K to 128K tokens pre-allocate VRAM before the first prompt.", "body_md": "You pull a 14B or 32B model in Ollama. On the first prompt, response generation is blazing fast—delivering a smooth 35 to 45 tokens per second. But on your third turn, after feeding in a 3,000-token function definition or log trace, generation suddenly grinds to a halt. The model stutters, your CPU fans rev to 100%, and token generation plummets to a painful **1.2 tokens per second**.\n\nYour GPU has not crashed, and Ollama hasn't thrown an error. What you are witnessing is **silent CPU fallback**—one of the most frustrating performance traps in local AI engineering.\n\nIn this technical guide from [The Zero-Dollar Dev](https://thezerodollardev.blogspot.com/), we break down why Ollama silently offloads transformer layers from VRAM to system RAM, how to verify it instantly with CLI diagnostics, and the exact configuration flags to lock your models into GPU memory permanently.\n\nWhen Ollama initializes an LLM, its runner (built on `llama.cpp`) inspects your system's available CUDA or Metal memory. If the entire model cannot fit into contiguous GPU VRAM, the runtime does not throw an out-of-memory (OOM) error by default. Instead, it partitions the model's neural network layers between two distinct memory spaces:\n\n| Memory Domain | Hardware Bus | Peak Theoretical Bandwidth | Latency Profile | \n|---|---|---|---|\n| **GPU VRAM (GDDR6X)** | Direct On-Die Memory Bus (e.g. RTX 3080/4080) | **760 – 1,008 GB/s** | Sub-microsecond tensor transfers | \n| **System RAM (DDR4 / DDR5)** | PCIe Gen4 / Gen5 Bus (x16 slot) | **38 – 64 GB/s** (PCIe bottleneck: ~31.5 GB/s) | Severe bus transit latency | \n\nNotice the massive discrepancy: **GPU VRAM is 15x to 25x faster than system DDR5 RAM**, and the PCIe interconnect creates a severe data transfer bottleneck. During inference, every single autoregressive token requires forward passes through *all* layers sequentially. If even **2 out of 48 layers** are offloaded to system RAM, the entire inference pipeline stalls at every layer transition while weights and activation tensors are copied back and forth across the PCIe bus.\n\nBecause Ollama does not print warning messages to standard output when offloading layers, you must inspect the runtime engine directly.\n\nWhile your model is running a query, open a second terminal and execute:\n\n```\nollama ps\n```\n\nExamine the **PROCESSOR** column in the output:\n\n```\nNAME                 ID           SIZE     PROCESSOR        UNTIL\nqwen2.5-coder:14b    b38006e8e6e5 9.0 GB   68%/32% CPU/GPU  4m52s\n```\n\nIf you see anything other than **100% GPU**, silent CPU fallback is actively degrading your throughput.\n\nOllama logs the exact layer allocation breakdown during model initialization. To view this:\n\n```\n# On Linux (systemd)\njournalctl -u ollama --no-pager -n 50 | grep -E \"load_tensors|offloaded\"\n\n# On macOS\ncat ~/.ollama/logs/server.log | grep -E \"load_tensors|offloaded\"\n\n# On Windows (PowerShell)\nGet-Content \"$env:LOCALAPPDATA\\Ollama\\server.log\" -Tail 100 | Select-String -Pattern \"offload\"\n```\n\nYou will see log entries detailing layer assignments:\n\n```\nllm_load_tensors: offloaded 34/48 layers to GPU\nllm_load_tensors:   14 layers remaining on CPU\nllm_load_tensors: VRAM used: 7842 MB, CPU RAM used: 3120 MB\n```\n\nWhen you start Ollama, it might fit 40 layers into an 8GB GPU, leaving only 300MB of free VRAM. As your conversation lengthens to 8,000 tokens, the Key-Value (KV) cache dynamically expands (requiring up to 2.5GB of additional memory). The CUDA memory allocator fails, forcing the runtime to evict layer tensors or spill dynamic activations to host system memory. (See our deep dive: [Llama 3.3 70B VRAM Calculation Math](https://thezerodollardev.blogspot.com/2026/11/llama-33-70b-on-consumer-gpu-vram.html)).\n\nIn modern GGUF models, the default context window (`num_ctx`) can be 32,768 or even 128,000 tokens. Ollama attempts to pre-allocate memory structures for this full context window, consuming multiple gigabytes of VRAM before the first prompt token is even processed.\n\nOperating systems, display servers (Xorg/Wayland/Windows DWM), and open browser tabs hardware-accelerated by Chromium consume between 600MB and 1.5GB of GPU VRAM. If Ollama calculates available VRAM without accounting for display buffer spikes, it over-allocates and trips fallback.\n\nBy default, Ollama stores KV-cache tensors in 16-bit floating point precision (FP16). By switching to 8-bit quantized KV-cache (Q8_0), you instantly cut KV-cache VRAM consumption in half with zero loss in coding precision:\n\n```\n# Linux / macOS\nexport OLLAMA_FLASH_ATTENTION=1\nexport OLLAMA_KV_CACHE_TYPE=q8_0\n\n# Windows (Persistent System Environment Variable)\n[System.Environment]::SetEnvironmentVariable('OLLAMA_FLASH_ATTENTION', '1', 'Machine')\n[System.Environment]::SetEnvironmentVariable('OLLAMA_KV_CACHE_TYPE', 'q8_0', 'Machine')\n```\n\nNever leave `num_ctx` unbound when operating near your GPU's VRAM ceiling. Create a custom Modelfile to set a deterministic context limit:\n\n```\nFROM qwen2.5-coder:14b-instruct-q4_K_M\n\n# Pin context window to 8,192 tokens to guarantee deterministic VRAM headroom\nPARAMETER num_ctx 8192\n\n# Pin GPU layer offload count (e.g. 48 for full offload)\nPARAMETER num_gpu 48\n\n# Low temperature for precise code generation\nPARAMETER temperature 0.2\nPARAMETER top_p 0.95\n```\n\nBuild and run the locked model:\n\n```\nollama create qwen14b-locked -f ./Modelfile\nollama run qwen14b-locked\n```\n\nTo automatically detect when an inference query spills over to CPU and restart Ollama before your machine locks up, save this lightweight Bash watcher:\n\n``` bash\n#!/usr/bin/env bash\n# ollama-watchdog.sh - Detect and alert on CPU fallback\nwhile true; do\n  PROCESSOR=$(ollama ps | awk 'NR>1 {print $4}')\n  if [[ \"$PROCESSOR\" =~ \"CPU\" ]]; then\n    echo \"[ALERT] Silent CPU Fallback Detected: $PROCESSOR\"\n    notify-send \"Ollama Fallback\" \"Model running on CPU: $PROCESSOR\"\n  fi\n  sleep 5\ndone\n```\n\nExplore our complete library of local AI hardware benchmarks:", "url": "https://wpnews.pro/news/silent-cpu-fallback-in-ollama-how-to-detect-and-prevent-cuda-spillover", "canonical_source": "https://dev.to/thezerodollardev/silent-cpu-fallback-in-ollama-how-to-detect-and-prevent-cuda-spillover-56lc", "published_at": "2026-11-21 09:05:29+00:00", "updated_at": "2026-11-21 09:12:05.470748+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["Ollama", "llama.cpp", "CUDA", "The Zero-Dollar Dev", "Qwen2.5-Coder"], "also_reported_by": [], "takeaway": "Check `ollama ps` for a PROCESSOR value other than 100% GPU and grep server logs for `llm_load_tensors` offload lines, since even 2 of 48 layers on CPU can collapse generation from ~40 tokens/s to ~1.2 tokens/s.", "recommendation": {"advice": "Verify GPU-only inference with `ollama ps` and the `llm_load_tensors` log lines, and cap `num_ctx` so the KV cache fits in VRAM instead of spilling layers to CPU.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/silent-cpu-fallback-in-ollama-how-to-detect-and-prevent-cuda-spillover", "markdown": "https://wpnews.pro/news/silent-cpu-fallback-in-ollama-how-to-detect-and-prevent-cuda-spillover.md", "text": "https://wpnews.pro/news/silent-cpu-fallback-in-ollama-how-to-detect-and-prevent-cuda-spillover.txt", "jsonld": "https://wpnews.pro/news/silent-cpu-fallback-in-ollama-how-to-detect-and-prevent-cuda-spillover.jsonld"}}