{"slug": "ran-moonshot-s-2-8t-parameter-kimi-k3-on-a-gpu-less-mini-pc", "title": "Ran Moonshot's 2.8T-parameter Kimi K3 on a GPU-less mini-PC", "summary": "Moonshot AI published Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model, on July 27, 2026, and rabbit ran it the next day on a GPU-less mini-PC using a Rust engine that streams routed experts from disk on demand, achieving correct output for a sample prompt in 2698.1 seconds for 40 tokens with no performance tuning yet.", "body_md": "**Small hops, immense models.** A Rust engine that runs frontier open-weight MoE models on a\nsingle machine. The dense part stays resident in RAM; the routed experts stream from disk on\ndemand.\n\nMoonshot AI published **Kimi K3** on 2026-07-27, a 2.8-trillion-parameter Mixture-of-Experts\nmodel and one of the largest open-weight releases so far. rabbit runs it the next day, straight\noff Moonshot's published checkpoint. No conversion step.\n\n``` bash\n$ rabbit --model /mnt/data/kimi-k3 --prompt \"What is the capital of France?\" --max-tokens 40\nloading model (dbits=4, ebits=4)...\nmodel loaded in 610.0s (93 layers, 896 experts/layer)\nprefill (7 tokens)...\n  prefill done in 412.8s\n  ...\n...response[\"answer\"] == \"Paris\"...\n\n40 tokens in 2698.1s\n```\n\nCorrectness validated three ways before that run. Bit-exact against a random-weight instance of\nMoonshot's own PyTorch reference (teacher-forcing, every position, `tests/teacher_forcing_k3.rs`\n\n).\nA structural smoke test against the real 1.56TB checkpoint (`examples/k3_smoke.rs`\n\n). And the real\nprompt above, answered correctly. No performance work has landed for K3 yet; the numbers above\nare correctness-first, not tuned. rabbit's other architecture, GLM-5.2, started from a similarly\nslow floor and is now 3.5× faster across eight measured versions. See\n[Honest numbers](#honest-numbers-ryzen-ai-9-hx-370-12-cores24-threads) and `PERFORMANCE.md`\n\n.\n\nK3 brings Kimi Delta Attention plus a Gated MLA hybrid, Stable LatentMoE (routed experts compute in a narrower latent width, not the full hidden size), Attention Residuals (a transient block-pooling mechanism across layers), and native OCP MXFP4 quantization for its 896 experts/layer. rabbit reads that MXFP4 straight off disk, byte for byte, no requantization.\n\nA large MoE model activates a small fraction of its parameters per token, and only the routed experts change token to token. So, across every architecture rabbit runs:\n\n- the\n**dense part**(attention, shared experts, embeddings) stays** resident in RAM**, quantized; - the\n**routed experts**, thousands of them, tens of MB each, live** on disk**and are** streamed on demand**, through a per-layer LRU cache, a persistent learned pin for the hottest ones, and the OS page cache as a free extra tier.\n\n| Kimi K3 | Kimi Linear 48B | GLM-5.2 | |\n|---|---|---|---|\n| total / active params | 2.8T / not yet characterized | 48B / ~3B | 744B / ~40B |\n| attention | KDA + Gated MLA, extra output gates | Kimi Delta Attention + Gated MLA | MLA + DSA sparse indexer |\n| MoE routing | Stable LatentMoE (narrower latent width), shared experts | grouped routing, shared experts | `noaux_tc` sigmoid, shared expert |\n| native quantization read | OCP MXFP4 (routed experts) | BF16 | FP8 (E4M3, block-scale) |\n| checkpoint | Moonshot's real release | Moonshot's real release | pre-converted by colibrì's tooling |\n| status | correctness-validated, perf work pending | tuned (`--session` , real chat) |\nfully tuned, 8 versions of perf work |\n\nOne `Model`\n\n/`KvState`\n\n/`ExpertCaches`\n\nfamily-dispatch enum in `src/model.rs`\n\nroutes to the right\narchitecture from `config.json`\n\n's `model_type`\n\n. `--chat`\n\n/`--serve`\n\n/`--prompt`\n\n/`--session`\n\nall\nwork the same way across the three.\n\n**Faithful forward pass for all three architectures.** Validated token-exact against a synthetic oracle built from each model family's own real reference code, plus real-checkpoint validation for each.**MLA attention** with**weight absorption** for decode (no per-token k/v reconstruction) and dense reconstruction for prefill, parallelized with`rayon`\n\nacross attention heads.**DSA sparse attention**(GLM-5.2's lightning indexer) and** Kimi Delta Attention**(KDA's chunked recurrence, short convolutions, per-channel decay gate). Real math, not approximated.** int4/int8/int2 quantization**, native** FP8**(E4M3, block-scale) and** OCP MXFP4**checkpoint loading, grouped-scale int4, and apre-quantized fast path.`.qs`\n\n**AVX2 + AVX-512/VNNI kernels**, runtime-selected, plus`rayon`\n\nparallelization across CPU cores for every matmul and the absorbed-attention decode path., with a sequential-`io_uring`\n\n-batched expert streaming`pread`\n\nfallback. K3's MXFP4 experts use the fallback today; batching that path is open work (see`ROADMAP.md`\n\n).**Persistent expert usage cache**(`.rabbit_usage`\n\n). Learns which experts your usage routes to and pins them, lazily, once a candidate is actually loaded through normal use.**KV-cache persistence**(`--session`\n\n). Conversations reopen warm across restarts.**A standalone checkpoint converter**(`bin/convert.rs`\n\n). Architecture-agnostic tensor classification, per-bucket bit-depth control, a`--report`\n\nquality pass. No dependency on colibrì's own tooling except for the pre-converted GLM-5.2 checkpoint above.**OpenAI-compatible HTTP server**(`--serve`\n\n). Streaming and non-streaming`/v1/chat/completions`\n\n,`/v1/models`\n\n, plus`/profile`\n\n, a rolling per-turn phase-timing window.**Multi-turn chat**(`--chat`\n\n) with each model's own real chat template.\n\nNot yet built: `io_uring`\n\n-batched MXFP4 expert loading, a SIMD tier for the MXFP4 matmul kernel\n(scalar only today), live expert re-pinning, GPU/CUDA, MTP speculative decoding, ARM NEON,\ngrammar-constrained decoding, and a web UI (`/profile`\n\nis a JSON endpoint, no page serves it yet;\nsee `DASHBOARD_BRIEF.md`\n\n).\n\n| metric | value |\n|---|---|\n| model load | 610.0s (93 layers, 896 experts/layer) |\n| prefill (7 tokens) | 412.8s |\n| decode, steady state | 50-70s/token |\n| decode I/O share | 30-40% disk I/O, 60-70% compute |\nexpert-cache hit rate (40-token run, `--expert-cache 64` ) |\ngrows to ~52% cumulative, still ~43% miss rate late in the run |\n\nThis is the correctness-first floor, not a tuned number. Same starting point GLM-5.2 was at\nbefore its own eight versions of `rayon`\n\n/SIMD/`io_uring`\n\nwork. The single biggest known lever:\nthe MXFP4 matmul kernel is scalar only, and compute, not disk, is most of each token's time\nhere, the opposite of GLM-5.2's I/O-bound steady state below.\n\n| metric | value |\n|---|---|\n| checkpoint | 378 GB (`jlnsrk/GLM-5.2-colibri-int4` ) |\n`rayon` matmul parallelization |\n128.9s → 36.3s for 5 tokens (3.5×), bit-exact output |\n`rayon` absorbed-attention parallelization |\n224.3s → 158.4s for 70 decode tokens (~29% faster) |\n| decode I/O share, steady state (warm cache) | 30-35% disk I/O, 65-70% compute |\n| prefill I/O share (cold cache) | ~75% disk I/O |\n| expert-cache hit rate, steady-state decode | 70-77% (miss floor 23-30%) |\n| usage-cache auto-pin | 150 experts (2/layer × 75 MoE layers): prefill hits 0 → 136 |\n| decode speed, current (v0.22.0) | 1.02 words/sec, up from 0.29 across eight measured versions |\n\nAll measured against the real checkpoints, not estimated. See `PERFORMANCE.md`\n\nfor the full\nchronological log, including techniques that were tried and reverted.\n\n```\ncargo build --release\ncargo test\n./target/release/rabbit --model /mnt/data/kimi-k3 --prompt \"What is the capital of France?\"\n./target/release/rabbit --model /mnt/data/kimi-k3 --chat --session ~/.rabbit_session\n./target/release/rabbit --model /mnt/data/kimi-k3 --serve --port 8000\n```\n\n`--model`\n\nauto-detects the architecture from the checkpoint's `config.json`\n\n. Kimi K3 and Kimi\nLinear read Moonshot's own published `safetensors`\n\nshards directly: download\n[ moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) and point\n\n`--model`\n\nat it, no\nconversion step. GLM-5.2 needs a directory in colibrì's converted layout; a pre-converted\ncheckpoint such as [works directly. See](https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4)\n\n`jlnsrk/GLM-5.2-colibri-int4`\n\n`--help`\n\nfor the full flag list (`--max-tokens`\n\n, `--temperature`\n\n,\n`--nucleus`\n\n, `--expert-cache`\n\n, `--dbits`\n\n, `--ebits`\n\n, `--shard-dirs`\n\n, `--no-usage-cache`\n\n, ...).\n\n```\nsrc/\n├── kimi_k3/                                  Kimi K3: SituAndMul, LatentMoE, Attention Residuals, MXFP4\n├── kimi_linear/                              Kimi Linear 48B: KDA, short convs, tokenizer, chat template\n├── glm52/                                    GLM-5.2: MLA+DSA attention, MoE router, checkpoint converter\n├── model.rs                                  family-dispatch enum: Model/KvState/ExpertCaches/Tokenizer\n├── safetensors.rs, quant.rs, kernels.rs      shard index, quantization, scalar/AVX2/AVX-512/MXFP4 kernels\n├── expert_cache.rs, usage_cache.rs           LRU expert streaming + persistent usage learning\n├── generate.rs, kv_session.rs                shared generation loop + KV-cache persistence\n├── chat.rs, server.rs, main.rs               chat templates, HTTP server, CLI entrypoint\ntests/oracle/     per-architecture oracle generators (real reference code, vendored) + fixtures\ntools/            real-tokenizer validation fixtures (dev-only, not a runtime dependency)\nbenches/          criterion benchmarks (kernels, expert loading)\n```\n\ncolibrì is C, hand-written, effectively zero-dependency, and GLM-5.2-only. rabbit ports the same\nalgorithms to Rust with a short list of well-justified dependencies instead of a zero-dep stance,\nadds its own performance work (the `rayon`\n\nparallelization above, KV-session and expert-usage\npersistence), and generalizes the whole engine into a family-dispatch design that now runs two\nmore architectures colibrì doesn't. Every architecture is validated the same way colibrì\nvalidates itself: token-exact teacher-forcing against a tiny synthetic model built from that\narchitecture's own real reference code.\n\nPart of the [ferrumox](/ferrumox/rabbit/blob) AI lab, alongside [fox](/ferrumox/rabbit/blob/fox) (a production local-LLM server\nwrapping llama.cpp). rabbit is the opposite kind of project: a research engine for models that\ndon't fit in memory even offloaded, built by hand instead of wrapping an existing runtime.\n\nThe name is a nod to [RabbitLLM](https://github.com/ManuelSLemos/RabbitLLM), an earlier,\nunrelated project of mine (a fork of AirLLM that streams full model layers through limited GPU\nVRAM). Same interest in running large models on constrained hardware, different problem, a\ncompletely different technique. Nothing in this codebase is derived from that one.\n\nPre-1.0, `0.MINOR.PATCH`\n\n, tracked via git tags and `release/vX.Y.Z`\n\nbranches rather than\n`Cargo.toml`\n\n's version field. `MINOR`\n\nbumps at the end of each development phase, `PATCH`\n\nfor\nfixes/polish within one. `rabbit-plan.md`\n\nhas the full phase-by-phase history through GLM-5.2 and\nKimi Linear's bring-up.\n\nTBD.", "url": "https://wpnews.pro/news/ran-moonshot-s-2-8t-parameter-kimi-k3-on-a-gpu-less-mini-pc", "canonical_source": "https://github.com/ferrumox/rabbit", "published_at": "2026-07-29 06:05:34+00:00", "updated_at": "2026-07-29 06:22:17.646196+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["Moonshot AI", "Kimi K3", "rabbit", "GLM-5.2", "Kimi Linear 48B", "OCP MXFP4", "Ryzen AI 9 HX 370", "colibrì"], "alternates": {"html": "https://wpnews.pro/news/ran-moonshot-s-2-8t-parameter-kimi-k3-on-a-gpu-less-mini-pc", "markdown": "https://wpnews.pro/news/ran-moonshot-s-2-8t-parameter-kimi-k3-on-a-gpu-less-mini-pc.md", "text": "https://wpnews.pro/news/ran-moonshot-s-2-8t-parameter-kimi-k3-on-a-gpu-less-mini-pc.txt", "jsonld": "https://wpnews.pro/news/ran-moonshot-s-2-8t-parameter-kimi-k3-on-a-gpu-less-mini-pc.jsonld"}}