{"slug": "llms-on-consumer-hardware-part-1-the-stack-and-first-benchmarks", "title": "LLMs on Consumer Hardware — Part 1: The Stack and First Benchmarks", "summary": "A developer documented a local LLM project running models on consumer hardware, benchmarking Gemma 4 26B against smaller models. The primary desktop with an AMD RX 6900XT achieved 18.86 tokens/sec on Gemma 4 26B, which scored 50/50 on a five-task quality suite, leading to its adoption as the primary model despite slower throughput.", "body_md": "This is the first in a series of build-log posts documenting a local LLM project, in which models are run on owned consumer hardware rather than through a cloud API. The present entry covers the hardware, the software stack, and the benchmarks by which a primary model was selected.\n\nTwo machines are used, both consumer-grade. All benchmarks reported below were obtained on the primary desktop.\n\n| Machine | CPU | RAM | GPU |\n|---|---|---|---|\nPrimary desktop |\nRyzen 5950X | ~80 GB DDR4 | AMD RX 6900XT (16 GB) |\nSecondary box |\nRyzen 5600G | 32 GB | NVIDIA GTX 1060 (6 GB) |\n\nOllama serves as the model runner across two GPU vendors: ROCm 5.7 for the AMD card on the primary desktop, and CUDA for the NVIDIA card on the secondary box.\n\nThe primary model is Gemma 4 26B, a mixture-of-experts model with roughly 3.8B active parameters, quantized to Q4_K_M and occupying approximately 18 GB on disk. On the RX 6900XT it is run with an automatic GPU/CPU layer split, as the Q4 weights together with the KV cache exceed the 16 GB of available VRAM. Several Ollama settings were enabled to recover headroom: flash attention, and an 8-bit (`q8_0`\n\n) KV cache, the latter approximately halving the cache footprint. A free cloud tier is retained for occasional heavier tasks, though the objective is to run as much as possible locally.\n\nBefore a primary model was chosen, the installed models were benchmarked. Two properties were of interest: throughput and output quality.\n\nThroughput was measured on the primary desktop with a 500-word essay prompt (`ollama run <model> --verbose`\n\n):\n\n| Model | Tokens/sec | Duration | Tokens out |\n|---|---|---|---|\n| gemma4:26b | 18.86 | 50.11s | 945 |\n| gemma4-26b (64K ctx) | 17.96 | 51.99s | 934 |\n| mistral:7b-instruct | 34.81 | 10.17s | 354 |\n| llama3.2 | 57.11 | 3.99s | 228 |\n\nThe smaller models are substantially faster; their token counts, however, are lower, and in practice their responses were correspondingly shallower.\n\nQuality was assessed with a five-task suite spanning logic, coding, summarization, creative writing, and instruction-following:\n\n| Model | Score | Note |\n|---|---|---|\n| gemma4-26b (64K) | 50/50 | Flawless instruction-following |\n| glm-4.7-flash | 42/50 | Missed only a complex string-formatting task |\n| qwen3-coder:30b | 40/50 | Overthinks logic; hallucinated a fake country |\n| llama3.2 | 28/50 | Failed logic entirely |\n| mistral:7b | 26/50 | Failed logic and negative constraints |\n\nA harder ten-task variant — incorporating a lipogram, a theory-of-mind question, and a riddle — was subsequently administered, on which Gemma 4 26B scored 99/100 while sustaining approximately 17 tokens/sec. This represents the local optimum: strong quality at a workable speed.\n\nThe principle that follows is that throughput and quality trade off against one another, and the fastest available model is rarely the appropriate default for substantive work. Gemma 4 26B is slower than the 7B models yet markedly more accurate, and it was therefore adopted as the primary model.\n\nWith a runner, a model, and a hardware baseline established, subsequent entries turn to the extraction of useful work from the setup. We intend to address context length and KV-cache trade-offs; the distinction between prefill and generation, and why a machine without a GPU can sustain conversation yet falter on large prompts; and the practice of keeping models resident to avoid cold-start reload penalties.\n\nThe measured figures are reported as-is, dead ends included. Part 2 will follow.", "url": "https://wpnews.pro/news/llms-on-consumer-hardware-part-1-the-stack-and-first-benchmarks", "canonical_source": "https://dev.to/sven_welack/running-llms-locally-on-consumer-hardware-part-1-the-stack-and-first-benchmarks-3k09", "published_at": "2026-08-05 01:07:09+00:00", "updated_at": "2026-08-05 01:39:24.000556+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Ollama", "AMD RX 6900XT", "NVIDIA GTX 1060", "Gemma 4 26B", "Mistral 7B", "Llama 3.2", "GLM-4.7-flash", "Qwen3-Coder"], "alternates": {"html": "https://wpnews.pro/news/llms-on-consumer-hardware-part-1-the-stack-and-first-benchmarks", "markdown": "https://wpnews.pro/news/llms-on-consumer-hardware-part-1-the-stack-and-first-benchmarks.md", "text": "https://wpnews.pro/news/llms-on-consumer-hardware-part-1-the-stack-and-first-benchmarks.txt", "jsonld": "https://wpnews.pro/news/llms-on-consumer-hardware-part-1-the-stack-and-first-benchmarks.jsonld"}}