{"slug": "a-26b-model-in-2-gb-of-ram-courtesy-of-your-ssd", "title": "A 26B Model in 2 GB of RAM, Courtesy of Your SSD", "summary": "TurboFieldfare, a Swift-and-Metal runtime, runs Google's Gemma 4 26B-A4B model on Apple Silicon Macs with as little as 2 GB of RAM by streaming expert weights from SSD, achieving 5.1–6.3 tokens per second on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 Pro. The project exploits mixture-of-experts routing to keep only the shared core and KV cache resident, fetching activated experts from flash with a 16-slot cache that hits 67% of the time, effectively turning the SSD into a usable memory tier for local inference on low-RAM machines.", "body_md": "[AI](https://sourcefeed.dev/c/ai)Article\n\n# A 26B Model in 2 GB of RAM, Courtesy of Your SSD\n\nTurboFieldfare streams Gemma 4's experts from flash, and the trick will outlive the tool.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\nA 26-billion-parameter model in 2 GB of RAM sounds like a benchmark scam. It isn't — but the interesting part isn't the number. It's what the trick reveals about where local inference is headed: mixture-of-experts routing has quietly turned your SSD into a usable tier of the memory hierarchy, and [TurboFieldfare](https://github.com/drumih/turbo-fieldfare) is the first project I've seen that builds an entire engine around that bet.\n\nThe project, which hit the Hacker News front page this week, is a Swift-and-Metal runtime that runs Google's [Gemma 4](https://ai.google.dev/gemma) 26B-A4B — 4-bit quantized, 14.3 GB on disk — on any Apple Silicon Mac while holding roughly 2 GB resident. On a base M2 MacBook Air with 8 GB of RAM, the author reports 5.1–6.3 tokens per second. On an M5 Pro, 31–35. Those are self-reported numbers, but the HN thread stress-tested the design hard and it held up.\n\n## Why this works now and didn't before\n\nStreaming weights from disk is old news, and mostly bad news. llama.cpp has mmap'd weights since 2023, and for dense models the result on a RAM-starved machine is thrashing: every token touches every parameter, so the page cache churns through the whole file per forward pass. Apple's own researchers mapped this territory in the [\"LLM in a flash\" paper](https://arxiv.org/abs/2312.11514) back in December 2023, using activation sparsity to predict which FFN neurons to pull from flash. Clever, but it fought the architecture. Dense models simply weren't built to be partially loaded.\n\nMoE models are. Gemma 4 26B-A4B activates only about 4B of its 26B parameters per token — the rest sit idle in expert blocks the router didn't pick. TurboFieldfare exploits the obvious corollary: keep the shared 1.35 GB core (embeddings, attention, router) and the FP16 KV cache pinned in RAM, and fetch only the experts each token actually routes to from SSD.\n\nThe engineering details are where it gets good. The author found that mmap was the wrong tool — cold expert loads took 10 ms through the page-fault path versus 2.8 ms with explicit `pread`\n\n, which in their simulations was the difference between 0.5 and 4 tokens per second. And expert selection has real temporal locality: the full route changes almost every token, but around 40% of experts repeat on the next token and 57% recur within two, so a small 16-slot cache hits about 67% of the time. That locality is the load-bearing fact. Without it, you're just benchmarking your SSD.\n\nThis is the same insight the community applied to Mixtral with CPU-RAM expert offloading in 2024, and that [ktransformers](https://github.com/kvcache-ai/ktransformers) uses to run DeepSeek-scale MoEs on single GPUs. TurboFieldfare pushes it one tier down the hierarchy — past DRAM, onto flash — which only pencils out because Apple ships fast NVMe in everything.\n\n## What you'd actually do with it\n\nBe clear-eyed about the trade. The author is: fully loaded, the same model does 75 tok/s in 14 GB of RAM. Streaming trades away roughly 40 tok/s to save 12 GB. If you have a 24 GB or 32 GB machine, you should mostly just load the model — or note that on high-RAM Macs the macOS page cache ends up holding the hot experts anyway, which is part of why the M5 numbers look so much better than the M2's.\n\nThe real audience is the enormous installed base of 8 GB M1 and M2 MacBooks — machines Apple sold by the millions before bumping the base config to 16 GB, and which have been effectively locked out of local models beyond the 3B–7B class. For those users, 5–6 tok/s from a model that lands top-ten on open-model leaderboards is a genuine unlock. That's comfortable reading speed for chat and drafting. It is not enough for agentic loops or an inline coding assistant, where you'll feel every second of a 2,000-token response.\n\nAdoption is low-friction: a streaming installer, a CLI, a native Mac app, and an OpenAI-compatible server on loopback, so anything that speaks the Chat Completions API can point at it. The constraints are sharper: macOS 26+, Metal 4, text-only, and — this is the big one — the engine is hard-wired to exactly one model. There's no GGUF loading, no swapping in next quarter's checkpoint. Two practical footnotes from the HN thread: sustained SSD reads don't wear flash (wear comes from writes), and the 2 GB figure includes an FP16 KV cache, so long contexts will push the footprint past the headline number.\n\n## Point solution, durable idea\n\nMy read: TurboFieldfare the product is a proof of concept you shouldn't build on, and TurboFieldfare the existence proof matters a lot. Single-model, single-platform engines don't survive contact with the model release cycle — Gemma 5 or the next Qwen MoE ships, and a bespoke runtime is instantly a museum piece. The Apache-2.0 code's real value is as a reference implementation, and history says the technique gets absorbed: [llama.cpp](https://github.com/ggml-org/llama.cpp) and [MLX](https://github.com/ml-explore/mlx) already have MoE plumbing and mmap machinery; a pread-based expert cache with locality-aware eviction is a bounded, well-specified feature for either. I'd expect a generalized version within a couple of quarters, because the incentive is enormous — every open-weights lab has converged on sparse MoE precisely because of the active-parameter economics this exploits.\n\nThe larger shift is worth naming plainly. For the entire local-LLM era, RAM has been the wall, and Apple's RAM pricing made it an expensive one. Expert streaming re-prices the problem: total parameters now cost you disk (cheap, plentiful), while active parameters cost you RAM and compute. Once runtimes generalize this, the question when a model drops stops being \"do I have 64 GB?\" and becomes \"how fast is my SSD, and how sparse is the router?\" That's a much better question — and for once, the answer favors the hardware people already own.\n\n## Sources & further reading\n\n-\n[Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac](https://github.com/drumih/turbo-fieldfare)— github.com -\n[Show HN discussion: TurboFieldfare](https://news.ycombinator.com/item?id=49098510)— news.ycombinator.com -\n[Gemma 4: Byte for byte, the most capable open models](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/)— blog.google -\n[Google Gemma 4 26B A4B now available on Workers AI](https://developers.cloudflare.com/changelog/post/2026-04-04-gemma-4-26b-a4b-workers-ai/)— developers.cloudflare.com -\n[LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://arxiv.org/abs/2312.11514)— arxiv.org\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/a-26b-model-in-2-gb-of-ram-courtesy-of-your-ssd", "canonical_source": "https://sourcefeed.dev/a/a-26b-model-in-2-gb-of-ram-courtesy-of-your-ssd", "published_at": "2026-07-29 17:08:32+00:00", "updated_at": "2026-07-29 17:26:33.648580+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["TurboFieldfare", "Google", "Gemma 4", "Apple", "M2 MacBook Air", "M5 Pro", "llama.cpp", "ktransformers"], "alternates": {"html": "https://wpnews.pro/news/a-26b-model-in-2-gb-of-ram-courtesy-of-your-ssd", "markdown": "https://wpnews.pro/news/a-26b-model-in-2-gb-of-ram-courtesy-of-your-ssd.md", "text": "https://wpnews.pro/news/a-26b-model-in-2-gb-of-ram-courtesy-of-your-ssd.txt", "jsonld": "https://wpnews.pro/news/a-26b-model-in-2-gb-of-ram-courtesy-of-your-ssd.jsonld"}}