I got myself a Tesla V100 a while ago and thought why not put it to some good use for once: Curious about the feasibility of the somewhat controversial ollama in comparison to straight llama-server (llama.cpp) for serving a small scale llm-based service, I decided to run a little benchmark myself.
For the overzealous: Yes, I know that ollama itself uses llama-server under the hood, but I still wanted to see if there is any significant overhead in practice, which in fact seems to be the case: Keep reading to see for yourself!
1. Results #
For the impatient, here are the bottom line results at a glance: Both backends were tested with Qwen3 8B Q4_K_M GGUF with the same prompt and decoding settings.
| Backend | Prompt processing | Generation |
|---|---|---|
| llama-server | 1255.29 tok/s | 111.00 tok/s |
| Ollama | 1028.02 tok/s | 110.61 tok/s |
Generation speed differed by about 0.35% – Practically a tie.
However, llama-server processed the short prompt about 22% faster. Is this conclusive? See the caveats section below.
2. Setup #
The environment is a small scale VPS running Debian 12 with a Tesla V100-SXM2 16 GB, NVIDIA driver 550.163.01, and CUDA 12.4.
llama.cpp was built from commit 0749449
with CUDA enabled for compute capability 7.0. Ollama was running at version 0.31.2.
The model used to run the benchmark was Qwen3 8B, quantized as Q4_K_M with context length 4096 tokens. Each run generated 256 tokens with temperature 0, top-k 1, top-p 1.0, repeat penalty 1.0, and seed 42.
For each backend, I ran one warm-up run followed by five measurement runs. Ollama kept using prompt cache inflating the results, so I had to ensure ollama was unloaded before each cold run to prevent that.
3. Caveats #
The prompt contained only 60 tokens, which is admittedly a bit short for a strong prompt-processing comparison because fixed overhead and timing noise matter more. Perhaps I'll get around to running a more expansive test soon, so stay tuned!
Also, Ollama kept making use of prompt caching (apparently there is no flag to disable that anymore?) which led to prompt throughput rising to roughly 6000 tok/s. Good to know though!
There are also many other factors to vary: My setup tested only one model, one quantization, one GPU, one context size, and one concurrent request.
4. Conclusions #
So overall Ollama and llama-server are effectively equally fast at token generation.
For long prompts, which is probably quite a common scenario, llama-server definitely seems to be the better choice. There is also another candidate: vLLM. I don't think my V100 (Volta architecture) is supported for it, but I would be curious to know if it performs even better on supported hardware.