{"slug": "frontier-class-llm-inference-on-a-laptop-cpu", "title": "Frontier-class LLM inference on a laptop CPU", "summary": "A new CPU-only inference runtime called cpubrrr achieves up to 5× faster token generation than llama.cpp's CPU path on frontier-class mixture-of-experts models, running on an Apple M4 Max without GPU access. The research runtime, which links only the C standard library, delivers ~77 tok/s on gpt-oss:20b (MXFP4) versus llama.cpp's ~14 tok/s, and ~92 tok/s on Qwen3-Coder-30B (Q4_K/Q6_K) versus ~82 tok/s, using hand-written NEON integer kernels and worker-driven execution. The project's author documented how earlier over-optimistic claims were corrected through rigorous re-measurement, emphasizing that the most transferable part is the methodology for identifying and fixing performance bottlenecks.", "body_md": "## cpubrr_low.mp4\n\n**From-scratch CPU-only LLM inference that beats llama.cpp's CPU path — on both quant formats it runs, no GPU.**\n\n`cpubrrr`\n\nis a research runtime that runs frontier-class **mixture-of-experts** models\non an Apple M4 Max **CPU only** — the binary links nothing but the C standard library,\nso it *physically cannot* touch the GPU (`otool -L target/release/engine`\n\nto verify). It\nstarted as a one-model experiment (gpt-oss:20b) and now runs four MoE models across two\narchitectures and two quantization formats from one config-driven engine — and it is\n**faster than llama.cpp's CPU path on both formats**, including Q4_K, llama.cpp's own\nhand-tuned home turf.\n\n| model | quant | cpubrrr | llama.cpp CPU | |\n|---|---|---|---|---|\ngpt-oss:20b |\nMXFP4 | ~77 tok/s |\n~14 tok/s | ~5× |\nQwen3-Coder-30B |\nQ4_K/Q6_K | ~92 tok/s |\n~82 tok/s | ~1.1–1.2× |\n\nDecode throughput, several runs each. Output verified correct in both cases. llama.cpp\nplacement confirmed CPU-only from Ollama's own server logs (it defaults to Metal GPU on\nmacOS — `num_gpu:0`\n\nis a *request*, not a fact; see the benchmark-integrity note below).\n\nThese numbers replace this repo's earlier, over-optimistic claims (a \"7.5× / 110 tok/s\"\nheadline that did not survive rigorous re-measurement — the reproducible figure is\n~77, recovered from a 52 tok/s regression by fixing dispatch overhead the hard way).\nThe story of *how* the early numbers were wrong — a contaminated baseline, unverified\nGPU/CPU placement, thermal throttling, and a thread-pool whose condvar wakeups silently\ncost 7.5 ms/token — is documented in full, in order, with evidence. We consider that\nthe most transferable part of the project.\n\nToken generation for an MoE model is **memory-bandwidth bound**, not compute bound: a\n21B model activates only ~3.6B params per token, so decode speed ≈ memory bandwidth ÷\nactive-bytes-per-token. llama.cpp's MXFP4 MoE CPU path uses a small fraction of the\navailable bandwidth; cpubrrr streams expert weights at close to what the cores can\nsustain. Core techniques:\n\n- Hand-written\n**NEON**(ARM SIMD) integer kernels using`sdot`\n\n/`tbl`\n\nfor exact 4-bit arithmetic on integer hardware — no float dequant in the inner loop. **Quad-interleaved weight layout** so each core reads one sequential stream (the single biggest MXFP4 win — a byte-reordering, not a code change).**Integer-accumulation Q8_K kernel** for Q4_K/Q6_K — llama.cpp's own algorithm, found by reading its ARM source: quantize activations to Q8_K, accumulate sub-block integer dot products weighted by the 6-bit scales in int32, and convert to float*once per 256-value superblock*. This is what took Q4_K from losing (~71 vs ~86) to winning (~92 vs ~82). Kernels verified bit-exact against a dequant reference.**Worker-driven execution**: 12 persistent workers run the whole forward pass with a*yielding*spin-barrier (spin briefly, then let the OS in) — saturates all cores without the collapse-under-jitter that a pure spin-barrier suffers.- MoE-aware scheduling, block-wise Q8 activation quantization,\n`mmap`\n\n'd weights (so a 117B model pages in under memory pressure instead of OOM-killing).\n\nThe lesson behind the Q4_K win: to beat a mature kernel, don't out-clever it with\nmicro-tweaks — read its source and the research, find the *algorithmic* edge, adopt it,\nthen out-schedule it.\n\nOne config-driven engine (dimensions read from the model at setup — same-family models run with zero code changes):\n\n**gpt-oss:20b**(MXFP4) — the original target, verified end-to-end.** gpt-oss-120b**(117B / ~5.1B active, MXFP4) — 6× bigger, same family; runs on the laptop CPU via`mmap`\n\n'd weights.**Qwen3-Coder-30B**(`qwen3moe`\n\n, Q4_K/Q6_K) — a different architecture*and*a different quant format; writes correct code.**Qwen3-30B** general — same arch, drop-in.\n\nArchitecture math was recovered from llama.cpp source and the 4-bit unpacking verified\nbit-for-bit against the official `gguf`\n\nlibrary *before* writing the forward pass — so\nthe new architecture produced correct output on the first run.\n\n— chronological lab log: every measured number, every wrong number, every correction, in order.[docs/RESEARCH_LOG_V2.md](/arizqi/cpubrrr/blob/main/docs/RESEARCH_LOG_V2.md)— the original single-model log.[docs/RESEARCH_LOG.md](/arizqi/cpubrrr/blob/main/docs/RESEARCH_LOG.md)— first-principles ceilings and the experiment ladder.[docs/PLAN.md](/arizqi/cpubrrr/blob/main/docs/PLAN.md)\n\n- Apple silicon Mac (M-series; developed on\n**M4 Max**, ARMv9 + SME). Other ARM/x86 targets need a port — see the honest-limits section below. [Rust](https://rustup.rs/)(stable), Python 3, and[Ollama](https://ollama.com/)with a model pulled (e.g.`ollama pull gpt-oss:20b`\n\nor`ollama pull qwen3-coder:30b`\n\n).\n\n```\n# 1. build (uses target-cpu=native)\ncargo build --release\n\n# 2. prepare runtime data from your local Ollama copy (no weights are copied)\n./scripts/setup_model.sh gpt-oss:20b     # or qwen3-coder:30b, etc.\n                                         # writes data-<slug>/{tokens.bin,config.txt,...}\n\n# 3. generate (gpt-oss uses the `engine` binary; qwen uses `engine_qwen2`)\n./target/release/engine data-gpt-oss_20b \"$(cat data-gpt-oss_20b/blob_path.txt)\" \"Why is the sky blue?\"\n```\n\nA local web page streams cpubrrr vs. llama.cpp/Ollama (CPU) with live tok/s counters:\n\n```\npython3 scripts/demo_server.py     # then open http://localhost:8642\n```\n\nEach is a standalone binary (`cargo build --release`\n\n, then `./target/release/<name>`\n\n):\n\n| binary | what it measures |\n|---|---|\n`engine` |\nfull gpt-oss (MXFP4) generation engine |\n`engine_qwen2` |\nfull Qwen3 (Q4_K/Q6_K) generation engine — the worker-driven, integer-accum path |\n`q8k_verify` |\nQ4_K/Q6_K integer-accum kernels vs a dequant-f64 reference |\n`qk_verify` |\nQ4_K/Q6_K dequant vs the official `gguf` library (bit-exact) |\n`bench_real` |\nreal gpt-oss MXFP4 weights through the decode kernel |\n`bench_moe` |\nMoE expert-batched decode bandwidth |\n`bench_sme` |\ndirect SME2 matrix-unit kernels + precision sweep |\n\n`scripts/bench_ollama.sh <model> <cpu|gpu>`\n\nproduces the llama.cpp baseline **and reads\nOllama's server log to confirm actual GPU/CPU placement** — it refuses to report a \"CPU\"\nnumber if any layers ran on the GPU.\n\nBandwidth-bound CPU benchmarks are easy to get wrong, and this project got them wrong three times before getting them right:\n\n**Contaminated baseline**— the first llama.cpp number was measured while a 65 GB download ate memory bandwidth in the background.** Unverified placement**— on macOS, Ollama defaults to full Metal GPU; some \"CPU\" runs were silently 49/49 layers on the GPU. Only the server log (`offloaded N/M layers to GPU`\n\n) reveals the truth.**Thermal throttling**— after days of load the*same binary*gave 66 → 42 → 5 tok/s on back-to-back runs.\n\ncpubrrr also saturates all 12 cores, so its peak needs a quiet machine; llama.cpp\ntolerates background load and heat better. Meta-lesson: **be more skeptical of\nbenchmarks that flatter you, not less.** Re-verify on your own hardware.\n\nThis is a **research engine**, not a production server (no cross-user batching, no\nserving hardening). Numbers are Apple M4; the techniques port to other ARM and (with\nAVX-512) x86, but those numbers must be measured, not assumed. A `training/kernel`\n\nresearch track programs the M4's SME/AMX matrix unit directly from Rust assembly\n(~4.2 TFLOPS fp32, measured to exceed Accelerate) — see the research log. On training\nscale: consumer CPUs can realistically train models up to ~1B params; a\ntrillion-parameter model is ~47,000 years of compute (physics, not pessimism).\n\nDual-licensed under [MIT](/arizqi/cpubrrr/blob/main/LICENSE-MIT) or [Apache 2.0](/arizqi/cpubrrr/blob/main/LICENSE-APACHE), at your option.\n\nNot affiliated with OpenAI, Apple, or Ollama. `gpt-oss`\n\nand `Qwen`\n\nare released by their\nauthors under their own licenses; this project neither includes nor redistributes model\nweights.", "url": "https://wpnews.pro/news/frontier-class-llm-inference-on-a-laptop-cpu", "canonical_source": "https://github.com/arizqi/cpubrrr", "published_at": "2026-07-23 19:10:04+00:00", "updated_at": "2026-07-23 19:25:23.404290+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure", "developer-tools"], "entities": ["cpubrrr", "llama.cpp", "Apple M4 Max", "gpt-oss:20b", "Qwen3-Coder-30B", "Ollama", "NEON", "gguf"], "alternates": {"html": "https://wpnews.pro/news/frontier-class-llm-inference-on-a-laptop-cpu", "markdown": "https://wpnews.pro/news/frontier-class-llm-inference-on-a-laptop-cpu.md", "text": "https://wpnews.pro/news/frontier-class-llm-inference-on-a-laptop-cpu.txt", "jsonld": "https://wpnews.pro/news/frontier-class-llm-inference-on-a-laptop-cpu.jsonld"}}