Homebench: Local LLM Benchmark Results Across Speed, Memory

Homebench, a local LLM benchmarking tool, tested five models on an RTX 3060 12GB, finding Qwen 2.5 7B fastest at ~58 tokens/sec and LLaMA 3 8B highest quality at 8.2/10, while Mistral Nemo 12B underperformed. The tool reports hardware context and uses GPT-4o-mini for quality scoring.

Homebench: Local LLM Benchmark Results Across Speed, Memory What I tested I stuck to models that run comfortably on a 16GB VRAM card, since that's the most common setup I see people asking about: LLaMA 3 8B GGUF Q4 K M Qwen 2.5 7B GGUF Q4 K M Mistral Nemo 12B GGUF Q4 K M Gemma 2 9B GGUF Q4 K M GGUF Q4 K M DeepSeek /en/tags/deepseek/ 7B All runs were on an RTX 3060 12GB, Ubuntu 22.04, using llama.cpp with CUDA offload. Homebench's three pillars it measures are throughput tokens/sec , peak memory usage, and an LLM-judged quality score on a fixed prompt set. The numbers Throughput tokens/sec : Qwen 2.5 7B leads at ~58 t/s, followed by Gemma 2 9B at ~49 t/s. LLaMA 3 8B sits at ~42 t/s, which surprised me — it felt faster in practice but the benchmark doesn't lie. Mistral Nemo drags at ~28 t/s, likely because it's the only 12B model here and the memory bandwidth gets saturated. Peak memory: Gemma 2 9B uses the least at 6.1GB, Qwen 2.5 7B at 6.8GB. LLaMA 3 8B peaks at 7.4GB. Mistral Nemo 12B eats 9.2GB, and DeepSeek 7B hits 8.1GB — odd, since it's smaller on paper. Quality score: LLaMA 3 8B wins here, scoring 8.2/10. Qwen 2.5 7B is close behind at 7.9/10. Gemma 2 9B drops to 7.1/10, DeepSeek 7B to 6.8/10, and Mistral Nemo 12B surprisingly comes in last at 6.5/10 despite being the biggest. What this tells me If you're chasing raw speed and your workload is mostly short responses or code completion, Qwen 2.5 7B is the sweet spot. It's fast, light on memory, and the quality dip from LLaMA 3 is small enough that you probably won't notice in most contexts. If you care about getting the best possible answer from a 7-9B model and can tolerate a ~30% speed hit, LLaMA 3 8B is still the benchmark to beat. The quality gap is real, especially on reasoning and multi-turn prompts. Gemma 2 9B is interesting — it's the most memory-efficient and nearly as fast as Qwen, but the quality score makes me hesitant unless you're severely VRAM-constrained. Mistral Nemo 12B is the disappointment. It's bigger, slower, hungrier for memory, and scored lowest on quality. I'm not sure what niche it's carving out, honestly. A practical note on Homebench itself The tool is straightforward — install via pip, point it at your model, and it runs through its preset suite. What I appreciate is that it doesn't try to hide the hardware context. It reports your GPU model, driver version, and llama.cpp commit, so you can actually compare apples to apples across different machines. Fair warning: the quality scoring uses a reference LLM currently GPT-4o-mini via API , so you need an OpenAI key handy. That's fine for me, but worth flagging if you're going fully offline. If you're tuning your local LLM setup for a specific use case, I'd recommend running Homebench with your own prompt templates rather than relying on the defaults. The stock prompts favor general-chat scenarios, which may not reflect your actual workload. pip install homebench homebench run --model ./models/qwen2.5-7b-q4km.gguf --prompts custom prompts.json The custom prompts.json format is just a list of strings — simple enough to script up your own test set from real user queries. Next evalgate: Fail CI on Prompt Regression → /en/threads/4850/