Why I Switched from Ollama to llama.cpp — and What I Learned

A developer switched from Ollama to llama.cpp to run a Qwen2.5-14B model on a dual-GPU AMD handheld, gaining efficiency and hardware control. The switch involved locating the Ollama blob, symlinking it, and launching llama-server with -ngl 32 to offload 32 of 48 layers to the GPU. The author notes that output quality depends on the model, not the backend, and advises verifying device selection on multi-GPU systems.

I’m running a local LLM on my Arch Linux + Hyprland setup, and already have qwen2.5:7b with Ollama. I want to maximize the usage of my hardware. Therefore I decided to download a slightly bigger model to cover my daily document writing and Python coding work, and switch from Ollama to llama.cpp. My hardware: a GPD handheld running Arch + Hyprland with an AMD Ryzen 7 7840U — a Radeon 780M integrated GPU plus a discrete Radeon RX 7600M XT 8GB each , driven through Vulkan/RADV. The exact specs don’t matter; what matters is that it’s a two-GPU machine a shared-memory iGPU and a discrete dGPU , and the point of this article is how to control and tune llama.cpp on such a setup. The switch mainly buys efficiency and hardware control. Output quality is determined by the model itself, not the backend — the better answers in this article come from the larger 14B model, not from llama.cpp. There is more detailed explanation on the web; I’ll just summarize the main points here bash jeff@gpd blobs $ free -h total used free shared buff/cache availableMem: 23Gi 9.7Gi 2.3Gi 503Mi 11Gi 13GiSwap: 11Gi 0B 11Gi jeff@gpd blobs $ glxinfo | grep -i "video memory" Video memory: 8192MB Dedicated video memory: 8192 MB Currently available dedicated video memory: 7079 MB This is an AMD APU with unified memory, so the 8GB “VRAM” is carved out of the same 23Gi system RAM that free -h shows — and the 7079MB currently available is what the GPU reports after the desktop has taken its share. That number is a good starting point for picking -ngl below. On a two-GPU machine, double-check which device llama.cpp actually picked before trusting those numbers — the GPU you measure with glxinfo isn’t necessarily the one the server runs on. llama.cpp defaults to the first device it finds Vulkan enumerates the iGPU first here, leaving the discrete GPU idle , and the “dedicated VRAM” figure is only what one API reports — on shared-memory devices llama.cpp may see a larger budget, so treat the -ngl math as a heuristic, not a hard limit. To list and override the device, use --list-devices / --device or the GGML VK VISIBLE DEVICES Vulkan , HIP VISIBLE DEVICES ROCm , or CUDA VISIBLE DEVICES CUDA env vars — the same approach works whether the second GPU is internal or an eGPU. A smaller model that fits entirely in the discrete GPU's VRAM typically runs several times faster there. Download qwen2.5:14b Find out where it is located bash jeff@gpd blobs $ pwd/var/lib/ollama/.ollama/models/blobs jeff@gpd blobs $ find . -size +8G -size -10G./sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54 After the download finishes, Ollama won’t be needed and can be stopped sudo systemctl stop ollama Create a symlink to the model downloaded via ollama. First create a local directory to hold the symlink mkdir -p ~/llama.cpp/ ln -s /var/lib/ollama/.ollama/models/blobs/./sha256-2049f5674b1e92b4464e5729975c9689fcfbf0b0e4443ccf10b5339f370f9a54 \ ~/llama.cpp/qwen2.5-14b.gguf Change the sha256 hash to yours Also adjust the blob path if needed — it depends on how Ollama was installed /usr/share/ollama/.ollama/models/blobs for the systemd package, /root/.ollama/models/blobs for the official installer . The blob is tied to that exact download: re-pulling a different quantization or ollama rm invalidates the hash. If you'd rather not depend on Ollama at all, download the GGUF directly with hf download Qwen/Qwen2.5-14B-Instruct-GGUF qwen2.5-14b-instruct-q4 k m.gguf --local-dir ~/llama.cpp/ Start the server no sudo needed — it runs as your user and stays reachable at localhost bash jeff@gpd ~ $ llama-server -m ~/llama.cpp/qwen2.5-14b.gguf -ngl 32 -c 8192 --flash-attn on -np 1 --port 8080 -ngl 32 offloads the first 32 of Qwen2.5-14B's 48 layers to the GPU and leaves the rest on the CPU. Why 32? The Q4 K M weights are ~9.04GB in total, so 32 layers ≈ 6.0GB of weights plus ~1.1GB of KV cache at -c 8192 ≈ 7.1GB — just under the 7079MB available. Those leftover 16 layers running on CPU are exactly why generation speed in the log below lands around 6 tokens/s while prompt processing reaches ~120–136 t/s. You’d see something like ...0.02.762.489 I srv llama server: model loaded0.02.762.494 I srv llama server: listening on http://127.0.0.1:8080 llama-server already exposes an OpenAI-compatible API at http://127.0.0.1:8080/v1, so any Obsidian plugin that accepts a custom OpenAI-compatible endpoint can talk to it. Using the Karpathy LLM Wiki plugin same one as the previous article : That’s it — your notes now query the local 14B model. The same endpoint also works with any other OpenAI-compatible Obsidian plugin Copilot, Smart Connections, Text Generator, … for document writing and Python coding assistance. To maximize local LLM capability through llama.cpp, the parameters can vary depending on your GPU or hardware spec, so it's important to tune them. One strong reason to run everything through the command line is that any error or warning message is printed clearly, which helps debugging. If you get an error, check Google or similar first. Here’s my printed log ...0.00.038.015 I cmn common param: common params print info: verbosity = 3 adjust with the -lv N CLI arg 0.00.038.358 W srv llama server: -----------------0.00.038.360 W srv llama server: CORS is set to allow all origins ' ' and no API key is set0.00.038.360 W srv llama server: this can be a security risk cross-origin attacks 0.00.038.360 W srv llama server: more info: https://github.com/ggml-org/llama.cpp/pull/256550.00.038.360 W srv llama server: -----------------0.00.039.555 I srv load model: loading model '/home/jeff/llama.cpp/qwen2.5-14b.gguf'0.00.268.690 W load: control-looking token: 128247 '</s ' was not control-type; this is probably a bug in the model. its type will be overridden0.02.757.937 I srv load model: initializing, n slots = 1, n ctx slot = 8192, kv unified = 'false'0.02.762.489 I srv llama server: model loaded0.02.762.494 I srv llama server: listening on http://127.0.0.1:80801.07.002.103 I slot get availabl: id 0 | task -1 | selected slot by LRU, t last = -11.07.002.163 I slot launch slot : id 0 | task 0 | processing task, is child = 01.22.014.086 I slot print timing: id 0 | task 0 | prompt processing, n tokens = 2048, progress = 0.42, t = 15.01 s / 136.43 tokens per second1.41.426.409 I slot print timing: id 0 | task 0 | prompt processing, n tokens = 4096, progress = 0.84, t = 34.42 s / 118.99 tokens per second2.08.679.214 I slot print timing: id 0 | task 0 | n decoded = 100, tg = 5.88 t/s, tg 3s = 5.88 t/s2.11.798.583 I slot print timing: id 0 | task 0 | n decoded = 118, tg = 5.86 t/s, tg 3s = 5.77 t/s A few notes on this log: the control-looking token: 128247 '</s ' warning is a known harmless artifact of some Qwen GGUF re-quants, so you can ignore it. The CORS + no-API-key warning is acceptable because llama-server binds to 127.0.0.1 by default — it's only reachable from your machine; if you ever expose it with --host 0.0.0.0, set --api-key first. And -np 1 allows one parallel request — raising it to 2 keeps the UI responsive while generating, but the KV cache roughly doubles, so keep an eye on the 7079MB budget. We can see the local LLM handles roughly 5~6 tokens/second, which seems low — the bottleneck is the 16 layers left on the CPU plus the shared memory bus on this APU — but it maintains better output quality than qwen2.5:7b. Certainly, if you prioritize faster processing — often a few times faster — you could try qwen2.5:7b for similar use cases like knowledge management with Karpathy LLM Wiki. One more check before blaming tuning: multiply your generation speed by the model size. At ~6 tokens/s the 14B model reads ~9GB per token, which works out to ~54GB/s — already near the effective shared-memory bandwidth of this APU, so pushing all 48 layers onto the iGPU wouldn’t have changed much. On any machine, if t/s × model size is close to your memory bandwidth, you’ve hit the wall, not a config bug. Two weeks of daily use later, the verdict is a trade-off between control and convenience. llama.cpp — maximum control Ollama — maximum convenience Neither is better in absolute terms — they simply optimize for different goals. btw, i use arch Why I Switched from Ollama to llama.cpp — and What I Learned https://blog.devgenius.io/why-i-switched-from-ollama-to-llama-cpp-and-what-i-learned-0ec71bf2fc95 was originally published in Dev Genius https://blog.devgenius.io on Medium, where people are continuing the conversation by highlighting and responding to this story.