Running LLMs on a CPU-Only Server: My Benchmark of Gemma, Qwen, Spark, and MiniCPM

A developer benchmarked four LLMs — Qwen3.5:9b, Spark-X2.6-1.7B, Gemma3:4b-it, and MiniCPM-V 4.7 1B — on a CPU-only AMD EPYC server with 32 GB of RAM using Ollama, measuring generation speed and Russian-language quality. MiniCPM-V 4.7 1B was fastest at 52.63 tokens/sec, Gemma3:4b was the best balance of speed (13.29 tok/s) and Russian fluency, while Qwen3.5:9b lagged at 6.71 tok/s. The developer concluded Gemma3:4b works as a daily driver for chat and summarization without a GPU.

We all know the default answer when someone asks about running local LLMs: "Get a GPU." Preferably an NVIDIA one. And honestly, for fine-tuning or heavy multimodal work, that's true. But what if you just want to run inference? A chatbot for your internal tools, a quick summarizer, or a private assistant that doesn't send your data to the cloud? Do you really need a discrete graphics card for that? I decided to find out. This is a write-up of my benchmark running four different models on a CPU-only server — no GPU, just an AMD EPYC and 32 GB of RAM. Spoiler: it works, and some results were genuinely surprising. The Testbed Here's the hardware and software setup: CPU: AMD EPYC-Genoa Processor 8 cores, 16 threads via SMT RAM: 32 GB OS: Ubuntu 24.04 Runtime: Ollama for easy model switching and built-in quantization support The goal was simple: measure generation speed tokens/sec and subjective quality of Russian-language responses. I used similar prompts across models to keep the comparison fair. Note that on CPU, prompt evaluation time can differ significantly from generation time — I focused on the latter, since that's what affects the "feel" of chatting with a model in real time. The Contenders I picked four models with different sizes and quantization levels: Qwen3.5:9b — the heavyweight, 9B parameters. SparkLLM/Spark-X2.6-1.7B:q8 0 — a lightweight 1.7B model with 8-bit quantization. Gemma3:4b-it-q4 K M — 4B parameters, 4-bit quantization the "golden middle" . MiniCPM-V 4.7 1B — a tiny 1B multimodal model. Results: Speed Here's the raw data. Generation speed is measured in tokens per second tok/s , and total time is for the full response. Model Generation tok/s Input tokens Output tokens Total time Qwen3.5:9b 6.71 19 467 70.46 s Spark-X2.6-1.7B:q8 0 16.72 88 4,008 240.47 s Gemma3:4b-it-q4 K M 13.29 27 296 22.73 s MiniCPM-V 4.7 1B 52.63 806 277 5.94 s A few observations: Qwen3.5:9b is painfully slow. 6.71 tok/s means you're waiting over a minute for a medium-length answer. It works, but it's not something you'd want to use interactively. Spark is decent at 16.72 tok/s, but it generated a massive 4,008-token response, which took 4 minutes total. Gemma3:4b hits a sweet spot at 13.29 tok/s. Fast enough to feel responsive, slow enough that you notice it's "thinking." MiniCPM-V is blazing fast at 52.63 tok/s — over 7x faster than Qwen and 4x faster than Gemma. A note on fairness I have to be honest here: MiniCPM's prompt evaluation rate 345 tok/s looks impressive, but 577 of its 806 input tokens were already cached. So that number isn't directly comparable to models that processed prompts from scratch. However, the generation speed of 52.63 tok/s is real — it produced 277 tokens in under 6 seconds. Results: Quality Russian Language Speed is great, but if the model outputs gibberish, it's useless. This is where things got interesting. I ran a few diagnostic prompts in Russian and evaluated the responses subjectively. Here's what I found: MiniCPM-V 4.7 1B — despite its speed, it was the weakest in Russian. It's primarily designed for multimodal tasks images, video and English. For complex text tasks, 1B parameters just aren't enough. Gemma3:4b — the clear winner for Russian. It officially supports 140+ languages and was clearly trained with multilingual text in mind. Responses felt natural, without the "machine accent" or hallucinated facts. Qwen3.5:9b — the heavy artillery. Slow, but potentially better for complex reasoning tasks. I'd keep it around for deep analysis, not quick chats. Practical Takeaways: What Actually Works on CPU? After all the tests, I settled on three use cases: Everyday tasks and chat Gemma3:4b is my daily driver. ~13 tok/s, good Russian quality, and it fits comfortably in 32 GB of RAM. Ultra-fast responses and multimodal tasks MiniCPM-V 4.7 1B is my go-to when I need a quick answer or need to process an image. Yes, its Russian is weaker, but 52 tok/s on CPU is genuinely impressive. Complex reasoning Qwen3.5:9b stays as a "heavy alternative." I wouldn't use it for simple questions, but for deep analysis, it might be worth the wait. The Gotchas It wasn't all smooth sailing. Here are the pain points I ran into: Thermal throttling. During long generations, the CPU heats up and clock speeds can drop. This affects consistency. 32 GB RAM is not infinite. If you run multiple models simultaneously, the system can start swapping to disk, and performance tanks. Context window matters. The more text you feed into the model, the slower it gets. On CPU, this is especially noticeable. Final Verdict Can you live without a GPU? Absolutely. If you're not fine-tuning models or generating hundreds of images per minute, modern CPUs handle inference just fine. For my AMD EPYC with 32 GB RAM on Ubuntu 24.04, Gemma3 4B is my primary model. It offers the best balance of speed and Russian-language quality. MiniCPM is a great tool for fast experiments and multimodal tasks. Have you tried running LLMs on CPU? Which model worked best for you? Share your experience in the comments