{"slug": "show-hn-pulsarforge-run-a-744b-moe-model-on-32gb-ram-with-zero-gpu-pure-c", "title": "Show HN: PulsarForge – run a 744B MoE model on 32GB RAM with zero GPU (pure C)", "summary": "PulsarForge, a CPU-only LLM inference engine written in C11, runs a 744-billion-parameter GLM-5.2 MoE model on a 2018 laptop with 32GB RAM and a USB SSD, achieving 9.3 seconds per token interactive speed after optimization, a 21x improvement over the naive run. The engine, validated bit-exact against independent oracles for five architectures, is claimed to be the measured state of the art at this RAM class, with no competing published results for 700B-class models on such hardware.", "body_md": "A from-scratch, **CPU-only LLM inference engine in C11** that pushes a\n**744-billion-parameter model** through a 2018 laptop with 32GB of RAM\nand a USB SSD — five architectures validated bit-exact against an\nindependent oracle, cross-platform bit-exactness on Windows and Linux,\nand a public engineering retrospective covering every measurement,\nincluding the ones that failed.\n\nStatus: research engine.Windows is the primary, most-measured path; Linux (native or WSL) is a validated second target — same source, byte-identical output on the same model. Want to run it? See[GETTING_STARTED.md]— build, tokenizer bootstrap, one-time conversion, chat and API in five steps.\n\n*Questo README è in inglese per il pubblico open source; il README\noperativo ( README.it.md) è in italiano — la sua\ntraduzione aggiornata in inglese è OPERATIONS.md.*\n\nPulsarForge started as a didactic engine for Qwen3-0.6B: understand inference by building it, validating every milestone against the HuggingFace reference. It grew into something rarer:\n\n**Five architectures**, each validated bit-exact or argmax-exact against an independent oracle:`qwen3`\n\n,`qwen3moe`\n\n,`deepseek2`\n\n(MLA),`olmoe`\n\n, and`glm-dsa`\n\n— GLM-5.2, a 744B MoE with compressed-Q MLA, a sparse DSA indexer, and 256 experts per layer.**Expert streaming on pure CPU**: the 202GB GGUF lives on a USB SSD; experts are fetched per-token through an explicit`NO_BUFFERING`\n\nreader with an LRU cache, cross-layer predictive prefetch (80.8% recall, measured offline*before*the code was written), arrival-order compute/IO overlap, and a priority queue where demand fetches overtake speculation.**A measurement discipline** behind every number below: baseline before building, interleaved A/B/A runs against thermal drift, counter-proofs, and honest negative verdicts — see[RETROSPECTIVE.md](/siris9476/pulsarforge/blob/main/RETROSPECTIVE.md)for the story, including the experiments that didn't work.\n\n| What | Result |\n|---|---|\n| GLM-5.2 744B (202GB GGUF), first naive run | 196 s/token |\n| Same model, same laptop, after the campaign | 9.3 s/token interactive (21×) |\nSame model in PulsarForge's own `.forge` container |\n~9 s/token, clean termination (see below) |\n| Batch mode (8 correlated streams, layer-major sweep) | ~2.9 s/token aggregate (68×) |\n| Qwen3-30B-A3B MoE chat | ~6.8 tok/s from the first token |\n| Qwen3-4B | ~2.6 tok/s — the machine's sweet spot |\n| Qwen3-0.6B (the didactic starting point) | ~13 tok/s Q6_K, ~15 tok/s Q4_K_M (`--no-think` only) |\n\nThe remaining walls are named and measured, not guessed: the USB chain tops out at 580–590MB/s real (Gen2 confirmed by probing cold regions), and the 15W PL1 caps the CPU. The diary closes the fetch axis with a measured verdict on every possible move — don't fetch (cache is at the routing-locality ceiling, flat from 8GB to 20GB, above simulated Bélády), fetch less (lossless compression: 0.1%; sub-expert grain loses to 26MB/s 4K-random reads), fetch earlier (prefetch is capacity-bound), fetch elsewhere (striping and cross-token prefetch: built, falsified).\n\n**The router is already a near-oracle on GLM-5.2**: correlation ρ=0.859 between router weights and true expert importance — contradicting published expectations for the DeepSeek family (weak correlation reported in arXiv 2406.18219 and follow-ups). This is what makes dynamic-k truncation safe here.**GLM-5.2 is \"dense in use\"**: no near-twin experts (max 1.2% similarity), no lazy layers, no dead positions, experts incompressible (ratio 1.000). The whole \"skip/deduplicate parts of the model\" family of optimizations is closed by how the model is, with citable numbers.**Nobody publishes numbers in this regime**(700B-class model, ≤32GB RAM, USB storage). The community results we could find need 96GB RAM + a 24GB GPU, or 512GB EPYC machines. At this RAM class, PulsarForge is the measured state of the art — partly for lack of competitors, which is exactly why the numbers are published here.\n\nThe head-to-head below exposed a trade nobody should have to make: our\n2.17-bit GGUF was 7× faster than colibrì but often failed to emit a\nstop token (an artifact-level fragility their deprecated per-row int4\nalso had — it lives in the quantization, not the engine); colibrì's\nint4 terminated cleanly but crawled. So PulsarForge grew what colibrì\nhas: **its own container**, converted directly from the official FP8\nweights by a streaming converter (141 shards, one at a time, ~6GB of\ntemp space, resumable, numerically validated), with the bit budget\nspent where this project's *measurements* say it matters — experts at\nint4 group-64 (the quality point colibrì's fix proved), router and\ncorrection biases at f32 (that's where the measured near-ties live),\none contiguous 20MB record per expert (one `pread`\n\ninstead of three).\n\nResult, measured A/B/A interleaved on the same laptop, byte-identical across repeat runs:\n\n| Stack | Terminates cleanly? | s/token |\n|---|---|---|\n| 2.17-bit GGUF + PulsarForge (retired) | no | ~6–9.3 |\n`.forge` int4 + PulsarForge |\nyes |\n~9 |\n| int4 + colibrì | yes | 63–66 |\n\n**Later update:** a two-day, measurement-led marathon took the\nsame scenario from 682s to **~300s** (several levers, every one bit-gated:\nprefetch diet, a Huffman-compressed container `.forgezh`\n\nat 351GB with\nparallel block decode, batched prefill with fetch/compute overlap and\nbatched attention, fused multi-activation kernels, and a novel\n**cache-aware router** — residency bonus on near-tie ranks ≥4, factual\nquality 3/3 at both tested strengths). Decode now runs at **~4.5 s/token**\n(~14× colibrì per token), with k=6 expert truncation and a cache-aware\nrouter both bracketed by measured quality cliffs on BOTH sides; a full 24-token run *including model load*\nfinishes before colibrì's 8-token decode alone.\n\nEven at that earlier `.forge`\n\nmilestone (before the marathon above),\nPulsarForge already matched colibrì-class quality at ~7× colibrì's\nspeed — and the same price the fragile GGUF used to cost. The last\n1.6× of *that* stretch came from a measurement-led\nrefinement worth retelling: dense weights at int8 cost ~17GB of RAM\nresidency, squeezing the expert cache below one token's working set\n(the cache *cycled*); requantizing them to int4 **in place** (687\ntensors, 264 seconds, no re-download — the JSON kind strings patched\nbyte-for-byte) halved residency, freed the cache to ~750 slots, and\ndropped read traffic to exactly the artifact-size ratio.\n\n```\n# convert once (resumable; ~a day of streaming on this line):\npython tools/forge_convert.py --dest glm52.forge\npython tools/forge_idx.py     glm52.forge\n\n# daily driving (auto-config included; the tokenizer is found\n# automatically at models/glm52-tokenizer.gguf):\nnf chat glm52.forge --session my.nfs\n```\n\n(That's the uncompressed `.forge`\n\ncontainer from before the later\nupdate above. For a new setup, use the compressed `.forgezh`\n\ncontainer\ninstead — see [GETTING_STARTED.md](/siris9476/pulsarforge/blob/main/GETTING_STARTED.md) for the\nconversion pipeline.)\n\nWhat was two artisans' territory a year ago (colibrì, DwarfStar) is now a category. The closest relatives, verified at the source:\n\n| Project | Stack | Model / speed | Hardware floor |\n|---|---|---|---|\n|\n\n**17.5 GB/s** SSD[pulsar](https://github.com/giannisanni/pulsar)**GLM-5.2 744B @ 2.7 tok/s**[sepia](https://github.com/rinaldofesta/sepia)[moe-stream](https://github.com/GOBA-AI-Labs/moe-stream)[DwarfStar/ds4](https://github.com/antirez/ds4)[PR #25294](https://github.com/ggml-org/llama.cpp/pull/25294))**PulsarForge****C, CPU-only****GLM-5.2 744B @ ~0.22 tok/s****32GB RAM, 580-590MB/s USB SSD (real), zero GPU** Same technique family everywhere — experts streamed from disk, hot-set\ncaching, routing-aware I/O. The niche nobody else occupies is the\nhardware floor: pulsar runs the *same model* ~12× faster on ~30× the\nI/O+compute budget; flash-moe's SSD alone moves ~30× our bytes/second.\nPulsarForge is the existence proof for the bottom of the curve — a\n744B frontier model, bit-gated quality, on a laptop that predates the\nmodel by seven years.\n\n**vs colibrì** — the closest\nproject in spirit (CPU inference of GLM-class MoE with SSD streaming).\nHead-to-head on OLMoE-1B-7B (same model, same prompt, greedy, this\nlaptop): both engines — built with zero shared code — produce the\n\n*identical token-for-token continuation*, a strong mutual correctness validation. On speed PulsarForge decodes ~3 tok/s vs 0.14 for colibrì's\n\n`olmoe.c`\n\n; after we patched an AVX2 kernel *into their prototype*it reached 0.25, still ~12× slower — but that file is their validation prototype, not their optimized main engine.\n\nSo we ran the real one. **Head-to-head on GLM-5.2 itself** (August\n2026): colibrì v1.5.0 main engine, their recommended int4-gs64\ncontainer (~429GB), this same laptop, same USB SSD, same greedy\nprompt, interleaved runs. Result: colibrì **~64.5 s/token** (8 tokens\nin 502–530s, 3% expert residency, its own printout: 0.02 tok/s) vs\nPulsarForge **~9.3 s/token** — **~7× faster per token**; our entire\nrun *including model load* finished before their decode alone. Honest\ndecomposition: ~1.8× of that comes from bytes (their int4 artifact\nmoves ~11GB/token, our 2.17-bit moves ~6 — a format choice, not an\nengine one); the remaining ~4× is stack (8GB expert cache at 40–66%\nhit + 80.8% cross-layer prefetch + I/O overlap, vs their 3% residency\nat the cap=1 floor this machine's RAM forces on them). **Quality went\nthe other way**: their int4 answered cleanly and self-terminated at 8\ntokens, twice out of twice; our 2.17-bit opens with the right answer\nand then fails to stop — the price of extreme quantization, which\ntheir own deprecated per-row int4 container also paid (documented\nnon-termination loops, their issue #455): cross-engine evidence that\ntermination fragility lives in the artifact, not the engine. Fairness\nnotes on the record: colibrì ran RAM-starved (overcommit, cache floor;\nwith 128GB it declares ~1.8 tok/s), while PulsarForge ran a config\ntuned for weeks on this exact machine. Delightful convergence: their\nauto-tuner disabled MTP on this box with the same reasoning our\nmeasurements had already established (\"widens the expert union, adds\ndisk reads\").\n\n**vs DwarfStar (antirez)** — the project this one grew from, and the\ninventor of the SSD-streaming idea. A head-to-head is impossible *by\ndesign*, and that difference is the point: DwarfStar's streaming is\nGPU-only (`ds4.c`\n\nrefuses `--ssd-streaming`\n\non CPU; its overlap works\nbecause the GPU computes while a CPU thread reads). PulsarForge streams\nexperts on **pure CPU**, where the computing threads are the same ones\nthat would contend for I/O — the regime DwarfStar explicitly does not\nenter. The projects explore disjoint regimes of the same problem.\nInherited from DwarfStar: the streaming idea, the RoPE `theta_scale`\n\ntrick, the \"narrow bet\" discipline. Contributed back: a quantified\nKnown Issue (per-head cos/sin redundancy in its CPU reference path),\ndocumented in its AGENT.md.\n\n**vs llama.cpp** — on models that fit in RAM, measured on this machine\nwith official binaries, interleaved A/B/A: long-context decode (d4096)\n**won 2.1–2.4×**, 30B MoE decode **won ~1.3×**, MoE prefill **won\n1.25×**, long prefill **won 1.02×**, short decode **tied at 100.0%**\n(the memory-bandwidth wall both engines hit at the same point), short\ndense prefill **tied at 101.7%** (a different, compute-side tie, not\nthe same bandwidth ceiling). llama.cpp remains the right tool for\nalmost everyone; these numbers exist to show what a readable\nsingle-purpose engine can reach on its reference hardware.\n\n```\nbuild.bat              # Windows, MSVC (VS2022) — primary target\nsh build_posix.sh nf   # Linux, gcc — validated bit-exact against Windows\n```\n\nModel downloads, Python test venv, and the full milestone-by-milestone\ncommand tour are in [OPERATIONS.md](/siris9476/pulsarforge/blob/main/OPERATIONS.md) (English) /\n[README.it.md](/siris9476/pulsarforge/blob/main/README.it.md) (Italian original). The regression suite\nis one command: `venv-tools/Scripts/python.exe tests/run_all.py`\n\n.\n\nDaily driving a model that fits in RAM:\n\n```\nnf.exe chat models/Qwen3-4B-Q4_K_M.gguf --system \"You are a concise assistant.\"\n```\n\nDaily driving the 744B (yes, really — one command, ~4.5s/token) with the\ncompressed `.forgezh`\n\ncontainer (see\n[GETTING_STARTED.md](/siris9476/pulsarforge/blob/main/GETTING_STARTED.md) for the full conversion\npipeline, from the original FP8 weights):\n\n```\nnf.exe chat glm52.forgezh --session my.nfs\n```\n\nThe measured-best configuration (fifteen tuning knobs) applies itself\nautomatically on glm GGUFs (`NF_GLM_AUTO=0`\n\nto disable; any variable\nyou set yourself wins). `--session`\n\nmakes conversation history free on\nresume (~4.5 s/token of history *not* re-prefilled). A cost-aware\nper-turn cap (96 tokens ≈ 7 min worst case on this hardware) protects\nagainst the failure mode of extreme quants: at 2.17 bits/weight the\nmodel often answers correctly in the first sentence and then fails to\nemit a stop token — at every quality level, approximations exonerated\nby byte-identical A/B (it's the model, not the engine). Ask for\nbrevity *inside your question* (\"Answer in a few words\") — it works;\na system prompt at this bitrate destabilizes instead.\n\n- Tuned on one machine. The techniques are general; the constants (cache budget, thread counts, dynamic-k threshold) are this laptop's.\n- Windows/MSVC is the primary, most-measured path. Linux (\n`build_posix.sh`\n\n, gcc) is validated for correctness — cross-platform bit-exact output, functional`nf serve`\n\non native Ubuntu and WSL2 — but hasn't had the months of performance tuning Windows has. - This is a learning engine with real results, not a product. If you\nwant to\n*run*models, use llama.cpp. If you want to*understand*what running them costs and why, read[RETROSPECTIVE.md](/siris9476/pulsarforge/blob/main/RETROSPECTIVE.md).\n\n[DwarfStar](https://github.com/antirez) for the lessons and the\nstreaming idea; [colibrì](https://github.com/JustVugg/colibri) for the\nindependent cross-validation target and an honest rival worth\nmeasuring against; llama.cpp for being the reference that kept every\nclaim here disciplined.\n\nBuilt with assistance from Claude Fable 5 (Anthropic).\n\nMIT (see `LICENSE`\n\n). The vendored zstd decoder (`zstd/`\n\n) is BSD-3-Clause,\nCopyright Meta Platforms — see `zstd/LICENSE`\n\n.", "url": "https://wpnews.pro/news/show-hn-pulsarforge-run-a-744b-moe-model-on-32gb-ram-with-zero-gpu-pure-c", "canonical_source": "https://github.com/siris9476/pulsarforge", "published_at": "2026-09-02 12:12:07+00:00", "updated_at": "2026-09-02 12:22:52.735236+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["PulsarForge", "GLM-5.2", "Qwen3", "DeepSeek", "HuggingFace"], "alternates": {"html": "https://wpnews.pro/news/show-hn-pulsarforge-run-a-744b-moe-model-on-32gb-ram-with-zero-gpu-pure-c", "markdown": "https://wpnews.pro/news/show-hn-pulsarforge-run-a-744b-moe-model-on-32gb-ram-with-zero-gpu-pure-c.md", "text": "https://wpnews.pro/news/show-hn-pulsarforge-run-a-744b-moe-model-on-32gb-ram-with-zero-gpu-pure-c.txt", "jsonld": "https://wpnews.pro/news/show-hn-pulsarforge-run-a-744b-moe-model-on-32gb-ram-with-zero-gpu-pure-c.jsonld"}}