{"slug": "homebench-benchmark-local-llms-for-speed-memory-and-quality", "title": "Homebench – Benchmark local LLMs for speed, memory, and quality", "summary": "Homebench, a new open-source command-line tool, benchmarks local large language models for speed, memory, and quality, presenting results in a live terminal leaderboard. The tool, installable via pip, supports Ollama, LM Studio, llama.cpp, vLLM, and any OpenAI-compatible server, measuring tokens/sec, time-to-first-token, memory footprint, and running a 31-task quality suite. It requires Python 3.9+ and is available on GitHub.", "body_md": "**Benchmark the local LLMs you already have — speed, memory, and quality — as a live terminal leaderboard.**\n\n`homebench`\n\nis a single-command TUI that discovers the models installed in your local runner (**Ollama**, **LM Studio**, **llama.cpp**, **vLLM**, or any **OpenAI-compatible** server), runs a curated quality suite, measures **tokens/sec**, **time-to-first-token**, and **memory footprint** on *your actual machine*, and renders a live comparison leaderboard.\n\n```\npip install homebench\nhomebench\n```\n\nThat's it. No config, no API keys, no cloud.\n\nThere are great tools for *one* half of this problem, but nothing local-first that does both:\n\n(inside llama.cpp) measures`llama-bench`\n\n**speed only**.measures`lm-evaluation-harness`\n\n**quality** but has no polished laptop UX and isn't built around the model runners most people actually use locally.\n\n`homebench`\n\nfills the gap: **local-first, zero-config, UX-driven.** Clone-and-run, point it at the models you already pulled, and get an at-a-glance answer to *\"which of my local models is actually good, and how fast is it on this laptop?\"*\n\n| Metric | How |\n|---|---|\ntok/s |\nOutput tokens ÷ generation time. Ollama reports server-side eval timing; OpenAI-compatible backends are timed client-side from the token stream. Excludes prompt processing and model load. |\nTTFT |\nWall-clock time to the first streamed token (minus model-load time where the runner reports it). |\nMemory |\nResident model size when the runner exposes it (Ollama `/api/ps` , LM Studio `/api/v0` ), plus a best-effort peak-RSS sample of the backend's processes. |\nQuality |\n31 deterministically-graded tasks across math, reasoning, factual recall, instruction-following/structured-output, extraction, and code understanding. Optional LLM-as-judge adds open-ended tasks (summaries, email, haiku, explanations). |\n\n```\npip install homebench        # then run:  homebench\n```\n\nPrefer an isolated install? Use [pipx](https://pipx.pypa.io):\n\n```\npipx install homebench\n```\n\nOr from source:\n\n```\ngit clone https://github.com/david-g-3654/homebench\ncd homebench\npip install .\n```\n\nRequires **Python 3.9+**.\n\n```\nhomebench                        # fast default: 3 smallest models, quick suite (TUI)\nhomebench --all                  # benchmark every discovered model\nhomebench --full                 # run the full quality suite (not just the fast subset)\nhomebench --no-tui               # plain live renderer (great for piping / CI)\nhomebench -m llama3.2,qwen3:8b   # only these models\nhomebench --limit 3              # cap the number of models\nhomebench --provider lmstudio    # use LM Studio instead of auto-detect\nhomebench --provider llamacpp    # llama.cpp server (llama-server)\nhomebench --provider vllm        # vLLM\nhomebench --provider openai --host http://localhost:5000   # any OpenAI-compatible server\nhomebench --refresh-cache        # recompute instead of reusing cached responses\nhomebench --no-quality           # speed + memory only (fast)\nhomebench --no-speed             # quality only\nhomebench --judge qwen3:8b       # enable LLM-as-judge (adds open-ended tasks)\nhomebench --tasks mypack.yaml    # use a custom task pack instead of the built-in suite\nhomebench --add-tasks mypack.yaml  # add a pack on top of the built-in suite\nhomebench --label \"before tuning\"  # tag this run for later diffing\nhomebench --md results.md        # also export a Markdown report\nhomebench --json results.json    # also export raw JSON\n\nhomebench list                   # just list discovered models\nhomebench tasks                  # show the quality suite (add --tasks to preview a pack)\nhomebench history                # list past runs (saved automatically)\nhomebench diff                   # diff the two most recent runs\nhomebench diff 3 1               # diff run #3 (base) against run #1 (newer)\nhomebench throughput             # batch-throughput sweep (concurrency 1,2,4,8)\nhomebench throughput --concurrency 1,8,16 --provider vllm\nhomebench fit                    # which popular models fit YOUR hardware?\n```\n\nRun `homebench --help`\n\nfor the full flag list.\n\nA real quick-suite run on an Apple M1 (16 GB), via Ollama:\n\n```\n                               Final leaderboard\n┏━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━┳━━━━━━━━┓\n┃ # ┃ Model                ┃ Params ┃ Quality ┃ Pass ┃ tok/s ┃   TTFT ┃ Memory ┃\n┡━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━╇━━━━━━━━┩\n│ 1 │ llama3.2:latest      │   3.2B │     75% │  6/8 │  16.8 │ 545 ms │ 2.4 GB │\n│ 2 │ alibayram/smollm3    │   3.1B │     38% │  3/8 │  16.9 │ 829 ms │ 2.1 GB │\n└───┴──────────────────────┴────────┴─────────┴──────┴───────┴────────┴────────┘\n```\n\n(Numbers are for *that* laptop at *that* moment — see [Limitations](#limitations).)\n\nAt least one local model runner must be reachable:\n\n| Provider | `--provider` |\nDefault host | Host env var | Notes |\n|---|---|---|---|---|\n| Ollama | `ollama` |\n`http://localhost:11434` |\n`OLLAMA_HOST` |\nNative API; reports model memory via `/api/ps` . |\n| LM Studio | `lmstudio` |\n`http://localhost:1234` |\n`LMSTUDIO_HOST` |\nEnriches metadata + memory via native `/api/v0` . |\n| llama.cpp | `llamacpp` |\n`http://localhost:8080` |\n`LLAMACPP_HOST` |\n`llama-server` , OpenAI-compatible. |\n| vLLM | `vllm` |\n`http://localhost:8000` |\n`VLLM_HOST` |\nSet `VLLM_API_KEY` if started with `--api-key` . |\n| OpenAI-compatible | `openai` |\n— | `OPENAI_BASE_URL` |\nAny `/v1` server (Jan, LocalAI, TGI, …); pass `--host` . |\n\nAuto-detection tries Ollama → LM Studio → llama.cpp → vLLM (the generic `openai`\n\nprovider is explicit-only). Force one with `--provider`\n\n. Override host with `--host`\n\nor the env var above.\n\nThe suite is small on purpose — enough tasks across categories to *separate* models, few enough that every model runs in a couple of minutes on a laptop. Each task is graded deterministically (exact numeric match, multiple-choice letter, substring, valid-JSON, regex). Temperature is 0 and a fixed seed is used for reproducibility. See `homebench tasks`\n\nfor the list.\n\nThe optional `--judge MODEL`\n\nflag turns on an LLM-as-judge (any local model) that scores open-ended tasks 1–5 against a reference answer. It's a signal, not an oracle.\n\nBenchmarking every model on the full suite takes a while on a laptop, so the defaults are tuned for a quick first look:\n\n**3 smallest models** by default (smallest first, so results appear fast) —`--all`\n\nfor everything,`-m`\n\nto choose.**A fast quality subset**(~8 tasks across all categories) —`--full`\n\nfor all 31.**Response caching**: quality runs use temperature 0 + a fixed seed, so responses are deterministic and cached under`~/.homebench`\n\n. Re-running only regenerates*new*models/tasks (unchanged ones are re-graded from cache in milliseconds);`--refresh-cache`\n\nforces recompute,`--no-cache`\n\ndisables it.\n\nIn practice this turns a first run from ~15–25 min (all models, full suite) into ~1–2 min, and a re-run into seconds. For a thorough pass (CI, final numbers) use `homebench --all --full`\n\n.\n\nBring your own evals with a JSON or YAML pack — no Python required. `--tasks`\n\nreplaces the built-in suite; `--add-tasks`\n\nappends to it. YAML needs the optional extra (`pip install \"homebench[yaml]\"`\n\n); JSON works out of the box.\n\n```\n# mypack.yaml  —  homebench --tasks mypack.yaml\nname: my-pack\ntasks:\n  - id: capital_japan\n    category: factual\n    prompt: \"What is the capital of Japan? Answer with just the city name.\"\n    grader: {type: contains_any, values: [\"Tokyo\"]}\n    reference: Tokyo\n  - id: add\n    category: math\n    prompt: \"What is 12 + 30? End with the answer on its own line.\"\n    grader: {type: exact_number, value: 42}\n  - id: explain          # no grader -> open-ended, scored only with --judge\n    category: open\n    prompt: \"Explain photosynthesis in one sentence.\"\n    reference: \"Plants convert sunlight, water, and CO2 into glucose and oxygen.\"\n```\n\nGrader `type`\n\nvalues: `exact_number`\n\n(`value`\n\n, `tol`\n\n), `multiple_choice`\n\n(`value`\n\n), `contains_any`\n\n(`values`\n\n), `regex`\n\n(`pattern`\n\n, `ignorecase`\n\n), `valid_json`\n\n(`keys`\n\n), `valid_json_array`\n\n(`length`\n\n). Omit `grader`\n\nfor a judge-only task. Runnable examples live in [ examples/](/david-g-3654/homebench/blob/main/examples); preview any pack with\n\n`homebench tasks --tasks mypack.yaml`\n\n.Every run is saved automatically to `$HOMEBENCH_HOME/runs`\n\n(default `~/.homebench/runs`\n\n); disable with `--no-save`\n\n, and tag runs with `--label`\n\n.\n\n```\nhomebench history            # table of past runs (newest first)\nhomebench diff               # previous run -> latest\nhomebench diff 3             # run #3 -> latest\nhomebench diff 3 1           # run #3 (base) -> run #1 (newer)\n```\n\n`diff`\n\ncompares models by name and shows per-model deltas in quality and throughput, plus which models were added or removed between runs — handy for \"did that quantization / setting actually help?\"\n\nThe main leaderboard measures **single-stream** tok/s. Servers that batch requests (vLLM, llama.cpp continuous batching, Ollama with `OLLAMA_NUM_PARALLEL>1`\n\n) can do far more total work under concurrency — `homebench throughput`\n\nmeasures that:\n\n```\nhomebench throughput -m my-model --concurrency 1,2,4,8\n```\n\nIt fires N requests at each concurrency level (N defaults to 3×concurrency) and reports **aggregate** tok/s (total output ÷ wall-clock), the speedup vs. concurrency 1, mean per-request rate, and latency (mean / p95):\n\n```\n             Batch throughput — my-model (vllm)\n┏━━━━━━┳━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┓\n┃ Conc ┃ Reqs ┃ Agg tok/s ┃ Speedup ┃ Req tok/s ┃ Mean lat ┃ p95 lat ┃ Errors ┃\n┡━━━━━━╇━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━┩\n│    1 │    4 │      95.0 │   1.00× │      95.0 │   1.35 s │  1.4 s  │      0 │\n│    4 │   12 │     320.0 │   3.37× │      82.0 │   1.56 s │  1.9 s  │      0 │\n│    8 │   24 │     540.0 │   5.68× │      70.0 │   1.83 s │  2.6 s  │      0 │\n└──────┴──────┴───────────┴─────────┴───────────┴──────────┴─────────┴────────┘\n```\n\nOn a non-batching setup, aggregate throughput stays flat while latency climbs — which is itself a useful thing to see. Add `--json FILE`\n\nto export.\n\nBefore benchmarking, `homebench fit`\n\ncaptures your hardware (RAM, CPU, GPU/VRAM, Apple unified memory) and checks a **catalog of ~50 popular models** — SmolLM2, Qwen2.5, Llama 3.x, Gemma 2, Phi-3.5/4, Mistral/Mixtral, DeepSeek-R1, CodeLlama, Yi, Command-R, and more, from 135M up to 141B — against your memory budget, showing which fit and at what quantization:\n\n```\nhomebench fit                    # what fits, at the best quant\nhomebench fit --all              # include models that don't fit\nhomebench fit --context 8192     # budget a larger KV cache\nhomebench fit --quant Q4_K_M     # evaluate a specific quant\nhomebench fit --vram 24          # what-if: \"if I had a 24 GB GPU…\"\nhomebench fit --catalog my.json  # add your own models to the catalog\n```\n\nInstead of the built-in catalog, pull the **currently most popular models straight from the HuggingFace Hub** — their parameter counts (from safetensors metadata) are sized against your hardware in real time:\n\n```\nhomebench fit --online              # top 50 text-generation models by downloads\nhomebench fit --online --top 100    # cast a wider net\nhomebench fit --online --sort trending   # or: likes\nhomebench fit --online --refresh    # bypass the 1-day cache\n```\n\nResults are cached under `$HOMEBENCH_HOME`\n\n(`~/.homebench`\n\n), so repeat runs are fast and work offline; if the Hub is unreachable, `homebench`\n\nfalls back to the cache (or the built-in catalog).\n\nThe built-in catalog also ships each model's **Ollama tag** (`ollama pull …`\n\n) and **HuggingFace repo** (which LM Studio and vLLM pull from). Add your own with a JSON catalog (see [ examples/models.example.json](/david-g-3654/homebench/blob/main/examples/models.example.json)): a list of\n\n`{name, params_b, family?, ollama?, hf?}`\n\n. Sizes are estimates (weights + KV cache + overhead), so treat \"fits\"/\"tight\" as guidance. Add `--json FILE`\n\nto export the hardware profile and results.`homebench`\n\nis a fast, local **first look** — not a rigorous benchmark of record. Keep these in mind:\n\n**Quality is a signal, not a leaderboard of record.** The suite is small and English-only (8 tasks in the fast default, 31 with`--full`\n\n); it's designed to*separate*your models, not to rank them authoritatively. For serious evals use[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). The optional LLM-as-judge is noisy, especially with small local judges.**Speed is your-machine-at-that-moment.** tok/s and TTFT depend on current load, thermal state, and memory pressure — a busy laptop (or swapping when low on RAM) will read slower. Numbers are meaningful*relative*to each other on the same run, not as absolute model specs.**Memory is best-effort.** It uses the runner's resident size where exposed (Ollama`/api/ps`\n\n, LM Studio`/api/v0`\n\n) plus RSS sampling; on unified-memory Macs it's approximate, and client-timed for OpenAI-compatible backends.(weights + KV cache + overhead) — treat \"fits/tight\" as guidance, not a guarantee. HuggingFace param counts come from safetensors metadata, which is missing for GGUF-only or gated repos.`fit`\n\nsizes are estimates**Throughput scaling only appears on batching servers**(vLLM, etc.); a single local model serializes requests.\n\n```\ngit clone https://github.com/david-g-3654/homebench\ncd homebench\npython -m venv .venv && source .venv/bin/activate\npip install -e \".[dev]\"\npytest -q\n```\n\nThe codebase is small and layered: `providers/`\n\n(pluggable backends), `quality/`\n\n(tasks, graders, judge), `metrics/`\n\n(memory sampling), `runner.py`\n\n(orchestration), `report.py`\n\n(export + tables), and `tui/`\n\n+ `plainui.py`\n\n(rendering). Adding a provider means subclassing `Provider`\n\n(or `OpenAICompatibleProvider`\n\n) and registering it; adding a task means appending to the suite in `quality/tasks.py`\n\nwith a reference that satisfies its grader (enforced by the tests).\n\nContributions welcome — new providers, task packs, and metrics especially.\n\n- PyPI release\n- HTML / shareable report export\n- Per-run environment capture (OS, RAM, GPU) for comparable results\n- Community task-pack sharing\n- GitHub Action for automated benchmarking in CI\n\nMIT", "url": "https://wpnews.pro/news/homebench-benchmark-local-llms-for-speed-memory-and-quality", "canonical_source": "https://github.com/david-g-3654/homebench", "published_at": "2026-08-04 09:48:43+00:00", "updated_at": "2026-08-04 09:52:41.881846+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "machine-learning"], "entities": ["Homebench", "Ollama", "LM Studio", "llama.cpp", "vLLM", "OpenAI", "Python", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/homebench-benchmark-local-llms-for-speed-memory-and-quality", "markdown": "https://wpnews.pro/news/homebench-benchmark-local-llms-for-speed-memory-and-quality.md", "text": "https://wpnews.pro/news/homebench-benchmark-local-llms-for-speed-memory-and-quality.txt", "jsonld": "https://wpnews.pro/news/homebench-benchmark-local-llms-for-speed-memory-and-quality.jsonld"}}