Frontier-class LLM inference on a laptop CPU A new CPU-only inference runtime called cpubrrr achieves up to 5× faster token generation than llama.cpp's CPU path on frontier-class mixture-of-experts models, running on an Apple M4 Max without GPU access. The research runtime, which links only the C standard library, delivers ~77 tok/s on gpt-oss:20b (MXFP4) versus llama.cpp's ~14 tok/s, and ~92 tok/s on Qwen3-Coder-30B (Q4_K/Q6_K) versus ~82 tok/s, using hand-written NEON integer kernels and worker-driven execution. The project's author documented how earlier over-optimistic claims were corrected through rigorous re-measurement, emphasizing that the most transferable part is the methodology for identifying and fixing performance bottlenecks. cpubrr low.mp4 From-scratch CPU-only LLM inference that beats llama.cpp's CPU path — on both quant formats it runs, no GPU. cpubrrr is a research runtime that runs frontier-class mixture-of-experts models on an Apple M4 Max CPU only — the binary links nothing but the C standard library, so it physically cannot touch the GPU otool -L target/release/engine to verify . It started as a one-model experiment gpt-oss:20b and now runs four MoE models across two architectures and two quantization formats from one config-driven engine — and it is faster than llama.cpp's CPU path on both formats , including Q4 K, llama.cpp's own hand-tuned home turf. | model | quant | cpubrrr | llama.cpp CPU | | |---|---|---|---|---| gpt-oss:20b | MXFP4 | ~77 tok/s | ~14 tok/s | ~5× | Qwen3-Coder-30B | Q4 K/Q6 K | ~92 tok/s | ~82 tok/s | ~1.1–1.2× | Decode throughput, several runs each. Output verified correct in both cases. llama.cpp placement confirmed CPU-only from Ollama's own server logs it defaults to Metal GPU on macOS — num gpu:0 is a request , not a fact; see the benchmark-integrity note below . These numbers replace this repo's earlier, over-optimistic claims a "7.5× / 110 tok/s" headline that did not survive rigorous re-measurement — the reproducible figure is ~77, recovered from a 52 tok/s regression by fixing dispatch overhead the hard way . The story of how the early numbers were wrong — a contaminated baseline, unverified GPU/CPU placement, thermal throttling, and a thread-pool whose condvar wakeups silently cost 7.5 ms/token — is documented in full, in order, with evidence. We consider that the most transferable part of the project. Token generation for an MoE model is memory-bandwidth bound , not compute bound: a 21B model activates only ~3.6B params per token, so decode speed ≈ memory bandwidth ÷ active-bytes-per-token. llama.cpp's MXFP4 MoE CPU path uses a small fraction of the available bandwidth; cpubrrr streams expert weights at close to what the cores can sustain. Core techniques: - Hand-written NEON ARM SIMD integer kernels using sdot / tbl for exact 4-bit arithmetic on integer hardware — no float dequant in the inner loop. Quad-interleaved weight layout so each core reads one sequential stream the single biggest MXFP4 win — a byte-reordering, not a code change . Integer-accumulation Q8 K kernel for Q4 K/Q6 K — llama.cpp's own algorithm, found by reading its ARM source: quantize activations to Q8 K, accumulate sub-block integer dot products weighted by the 6-bit scales in int32, and convert to float once per 256-value superblock . This is what took Q4 K from losing ~71 vs ~86 to winning ~92 vs ~82 . Kernels verified bit-exact against a dequant reference. Worker-driven execution : 12 persistent workers run the whole forward pass with a yielding spin-barrier spin briefly, then let the OS in — saturates all cores without the collapse-under-jitter that a pure spin-barrier suffers.- MoE-aware scheduling, block-wise Q8 activation quantization, mmap 'd weights so a 117B model pages in under memory pressure instead of OOM-killing . The lesson behind the Q4 K win: to beat a mature kernel, don't out-clever it with micro-tweaks — read its source and the research, find the algorithmic edge, adopt it, then out-schedule it. One config-driven engine dimensions read from the model at setup — same-family models run with zero code changes : gpt-oss:20b MXFP4 — the original target, verified end-to-end. gpt-oss-120b 117B / ~5.1B active, MXFP4 — 6× bigger, same family; runs on the laptop CPU via mmap 'd weights. Qwen3-Coder-30B qwen3moe , Q4 K/Q6 K — a different architecture and a different quant format; writes correct code. Qwen3-30B general — same arch, drop-in. Architecture math was recovered from llama.cpp source and the 4-bit unpacking verified bit-for-bit against the official gguf library before writing the forward pass — so the new architecture produced correct output on the first run. — chronological lab log: every measured number, every wrong number, every correction, in order. docs/RESEARCH LOG V2.md /arizqi/cpubrrr/blob/main/docs/RESEARCH LOG V2.md — the original single-model log. docs/RESEARCH LOG.md /arizqi/cpubrrr/blob/main/docs/RESEARCH LOG.md — first-principles ceilings and the experiment ladder. docs/PLAN.md /arizqi/cpubrrr/blob/main/docs/PLAN.md - Apple silicon Mac M-series; developed on M4 Max , ARMv9 + SME . Other ARM/x86 targets need a port — see the honest-limits section below. Rust https://rustup.rs/ stable , Python 3, and Ollama https://ollama.com/ with a model pulled e.g. ollama pull gpt-oss:20b or ollama pull qwen3-coder:30b . 1. build uses target-cpu=native cargo build --release 2. prepare runtime data from your local Ollama copy no weights are copied ./scripts/setup model.sh gpt-oss:20b or qwen3-coder:30b, etc. writes data-