{"slug": "show-hn-open-source-engine-running-gemma-4-26b-in-2-gb-ram-on-any-m-series-mac", "title": "Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac", "summary": "A developer has released TurboFieldfare, an open-source Swift and Metal engine that runs Google's Gemma 4 26B-A4B instruction-tuned model in about 2 GB of RAM on any Apple Silicon Mac, including 8 GB models. The engine streams only the experts needed for each token from SSD, keeping a 1.35 GB core and FP16 KV cache in memory, and achieves 31-35 tok/s on a 24 GB M5 Pro. TurboFieldfare is model-specific and provides a native Mac app, CLI, and OpenAI-compatible server.", "body_md": "**Gemma 4 26B-A4B inference in about 2 GB of RAM**\n\nA custom Swift + Metal runtime for any Apple Silicon Mac, even the 8 GB ones.\n\n[Quick start](#try-it) ·\n[Local server](/drumih/turbo-fieldfare/blob/main/docs/OPENAI_SERVER.md) ·\n[Benchmarks](/drumih/turbo-fieldfare/blob/main/docs/BENCHMARKS.md) ·\n[Contribute results](/drumih/turbo-fieldfare/blob/main/docs/COMMUNITY_BENCHMARKS.md) ·\n[How it works](/drumih/turbo-fieldfare/blob/main/docs/SYSTEM_DESIGN.md) ·\n[Experiments](/drumih/turbo-fieldfare/blob/main/docs/OPTIMIZATION_JOURNEY.md) ·\n[References](/drumih/turbo-fieldfare/blob/main/docs/IMPLEMENTATION_REFERENCES.md)\n\nMemory got expensive. So I gave a 26-billion-parameter model a ~2 GB budget.\n\nTurboFieldfare runs the instruction-tuned\n** Gemma 4 26B-A4B**\nwithout loading the entire 14.3 GB model into memory. It keeps the shared\n1.35 GB core and FP16 KV cache in memory, then streams only the experts needed\nfor each token from SSD. This is what lets the model run on Macs with 8 GB of\nRAM.\n\nThe runtime, streaming installer, CLI, and native Mac app are written in Swift\nand Metal. TurboFieldfare is model-specific rather than a wrapper around MLX or\nllama.cpp. The curated [experiment record](/drumih/turbo-fieldfare/blob/main/docs/experiments/EXPERIMENT_INVENTORY.md)\nsummarizes 103 measured results across kernels, caching, I/O, prefill, and\ndecode.\n\n```\ngit clone https://github.com/drumih/turbo-fieldfare.git\ncd turbo-fieldfare\nswift build -c release\n.build/release/TurboFieldfareMac\n```\n\nOn the first run, Swift Package Manager downloads and builds the Swift packages required by the tokenizer. The complete release build includes the foreground Mac app and its sibling decode-service executable.\n\nWhen the app opens, choose **Download** and let TurboFieldfare fetch and repack\nthe pinned model (about 15 GB). Once it is ready, choose **Load Model**, type\nyour prompt, and press **Generate**.\n\n| Metric | Value |\n|---|---|\n| Model | Gemma 4 26B-A4B IT, 26B total parameters, about 3.88B active per token |\n| Weights | MLX affine 4-bit, group 64; 8-bit router; 4-bit shared and routed experts |\n| Memory | ~2 GB of weights and 4K KV cache |\n| Storage | About 14.3 GB for the installed text-only model |\n| Hardware | Apple Silicon Mac; 8 GB of RAM |\n| Platform | macOS 26, Metal 4, Swift 6.2 |\n| M2 measured decode |\n|\n\n[31-35 tok/s](/drumih/turbo-fieldfare/blob/main/docs/BENCHMARKS.md#m5-measured-decode)on a 24 GB M5 ProThe measured result is a reference point, not a performance ceiling. Prompt\nlength, generated length, page-cache state, and hardware all affect throughput.\nTo help measure another Apple Silicon Mac, follow the\n[community benchmark guide](/drumih/turbo-fieldfare/blob/main/docs/COMMUNITY_BENCHMARKS.md).\n\nTurboFieldfare provides a native Mac app, a command-line interface, and an\nexperimental loopback OpenAI-compatible server. They use the same `.gturbo`\n\nmodel directory, but only one model-owning product should run at a time.\n\nThe Swift package exposes six products:\n\n| Product | Purpose |\n|---|---|\n`TurboFieldfare` |\nSwift library containing the runtime and Metal kernels |\n`TurboFieldfareMac` |\nNative Mac app for installation and generation |\n`TurboFieldfareDecodeService` |\nOne-shot local model and Metal owner used by the Mac app |\n`TurboFieldfareCLI` |\nCommand-line instruction chat and raw completion |\n`TurboFieldfareServer` |\nLoopback OpenAI-compatible Chat Completions server |\n`TurboFieldfareRepack` |\nStreaming model installer and install verifier |\n\n- An Apple Silicon Mac; the validated target is an 8 GB M2 MacBook Air\n- macOS 26 with Metal 4\n- Xcode 26 and Swift 6.2 or newer\n- Enough free storage for the ~14.3 GB model installation\n- An internet connection for the first model install\n\nThe package is arm64-only. Older macOS and Metal versions are not supported.\n\nThe Mac app treats what you type as an instruction and handles Gemma's chat formatting automatically. Just describe the task and include any context the model needs.\n\nGeneration defaults to temperature `0.2`\n\n, Top-K `64`\n\n, and Top-P `0.95`\n\n. Set\ntemperature to `0`\n\nfor deterministic greedy output. The model can still repeat\nitself or give incorrect answers, so check important results.\n\nTurboFieldfare is text-only. The app and CLI support user and model messages plus optional system guidance; they do not expose or execute tools. The loopback server accepts function-tool declarations and returns model-produced tool calls for the client to authorize and execute. Images, audio, and video are not supported.\n\nClone the repository, then run the app from its root:\n\n```\nswift build -c release\n.build/release/TurboFieldfareMac\n```\n\nBuild the complete package so the app and its sibling decode service are both\navailable. When launched from this checkout, the app stores the model in\n`scratch/gemma4.gturbo`\n\n.\n\nOn first launch, the app checks the available storage and shows the download\nand installed sizes. Choose **Download** to begin.\n\nThe installer never materializes the full source checkpoint. It streams the\nrequired byte ranges from the pinned Hugging Face revision and repacks them\ndirectly into the `.gturbo`\n\nlayout as they arrive. This avoids a second full\ncheckpoint on disk and keeps scratch memory bounded.\n\nThe first installation transfers about 15 GB through bounded Hugging Face\nrange requests. Network speed and Hugging Face response times vary, so it can\ntake a while. The completed `.gturbo`\n\ninstallation occupies about 14.3 GB and\nis accepted only after its manifest and file hashes have been validated.\nInstallation does not load the model into memory.\n\nAfter installation:\n\n- Choose\n**Load Model**. - Enter a prompt in the composer.\n- Choose\n**Generate**, or press`Command`+` Return`. - Use the stop button or\n`Escape` to end generation early.\n\nThe status bar shows generation progress, decode speed, and memory use. Use the\nright pane to configure sampling, context length, expert-cache slots, and\nruntime options. See [Runtime controls](/drumih/turbo-fieldfare/blob/main/docs/RUNTIME_CONTROLS.md) for details\nand defaults.\n\nThe CLI uses an existing `.gturbo`\n\ninstallation. If you installed the model\nthrough the Mac app, it is already available at `scratch/gemma4.gturbo`\n\n.\nOtherwise, install it from the command line:\n\n```\nswift run -c release TurboFieldfareRepack \\\n  --output scratch/gemma4.gturbo \\\n  --overwrite\n```\n\nContinue a cancelled or interrupted download:\n\n```\nswift run -c release TurboFieldfareRepack \\\n  --output scratch/gemma4.gturbo \\\n  --overwrite \\\n  --resume\n```\n\nRemove saved download state:\n\n```\nswift run -c release TurboFieldfareRepack \\\n  --discard-partial \\\n  --output scratch/gemma4.gturbo\n```\n\nThe runtime accepts only a completed `.gturbo`\n\ndirectory with a final\n`manifest.json`\n\n.\n\nVerify an existing installation without loading the model:\n\n```\nswift run -c release TurboFieldfareRepack \\\n  --verify-install \\\n  --input-gturbo scratch/gemma4.gturbo\n```\n\nPut chat messages in a JSON array and pass it with `--messages-file`\n\n:\n\n```\n[\n  {\"role\": \"user\", \"content\": \"Explain why chunked prefill reduces time to first token while keeping memory bounded.\"}\n]\nswift run -c release TurboFieldfareCLI \\\n  --model scratch/gemma4.gturbo \\\n  --messages-file messages.json\n```\n\nThis formats messages in the same way as the Mac app. The CLI response limit\nis set with `--max-new`\n\n, which defaults to 1,024 tokens. The Mac app can\ngenerate until the selected context window is full.\n\n`--prompt`\n\nis available for raw completion and reproducible comparisons. It\npasses the text directly to the model without chat formatting. Use\n`--messages-file`\n\nfor instruction-response conversations.\n\n```\nswift run -c release TurboFieldfareCLI \\\n  --model scratch/gemma4.gturbo \\\n  --prompt \"The capital of France is\" \\\n  --max-new 64 \\\n  --temperature 0\n```\n\nThis example deliberately requests a short greedy completion.\n\nCommon generation options include `--max-context`\n\n, `--temperature`\n\n, `--top-k`\n\n,\n`--top-p`\n\n, `--repetition-penalty`\n\n, `--seed`\n\n, and repeatable `--stop`\n\nstrings.\nThe public CLI uses production runtime defaults. Run the following command for\nthe complete option list:\n\n```\nswift run -c release TurboFieldfareCLI --help\n```\n\nGenerated text goes to standard output. Timing statistics go to standard error;\nadd `--quiet`\n\nto suppress that footer in scripts.\n\nBuild the server and point it at an installed model:\n\n```\nswift build -c release --product TurboFieldfareServer\n.build/release/TurboFieldfareServer \\\n  --model scratch/gemma4.gturbo\n```\n\nIt listens on `http://127.0.0.1:8080/v1`\n\nand supports Chat Completions,\nstreaming, function tools, and single-prefix prompt reuse. The client must\nauthorize and run every tool call. Keep the server on loopback; it has no\nremote authentication or TLS.\n\nSee [Local server](/drumih/turbo-fieldfare/blob/main/docs/OPENAI_SERVER.md) for a test request, Python and\nOpenCode setup, prompt reuse, tool handling, and the supported API subset.\n\nRun the public test suite serially:\n\n```\nScripts/test.sh\n```\n\nBefore starting a model run, close memory-heavy apps and check\n`memory_pressure -Q`\n\n. If it reports little free memory, postpone the run. Run\nonly one TurboFieldfare app, decode service, CLI, server, test, or other\nlocal-model process at a time.\n\nTo contribute a comparable performance result, follow the\n[community benchmark guide](/drumih/turbo-fieldfare/blob/main/docs/COMMUNITY_BENCHMARKS.md).\n\nAt each transformer layer, Metal computes attention and the router from\nresident weights. The CPU uses the router's top-8 expert IDs to plan against\nthe layer's 16-slot LFU cache, then fills misses with bounded parallel `pread`\n\ncalls into Metal-visible buffers. Metal computes the resident shared-expert\nbranch while those reads run, then combines the shared and routed outputs.\n\nPrompt prefill uses chunks of up to 128 tokens so one fetched expert can serve\nmultiple rows. Generation repeats the routed layer loop one token at a time.\nThe installer applies the same bounded-memory rule: it repacks remote ranges\ndirectly into `.gturbo`\n\nwithout staging a full shard or tensor.\n\nFor a visual introduction to the model architecture, see Maarten Grootendorst's\n[A Visual Guide to Gemma 4](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-gemma-4).\n\n[System design](/drumih/turbo-fieldfare/blob/main/docs/SYSTEM_DESIGN.md) explains the `.gturbo`\n\nlayout, memory\nownership, prefill, router handoff, `cb1`\n\n/`io`\n\n/`cb2`\n\nphases, Metal kernels, and\ncorrectness invariants.\n\nTurboFieldfare currently includes:\n\n- Remote streaming repack into the\n`.gturbo`\n\nmodel format - Instruction-tuned Gemma 4 26B-A4B with verified text-only chat formatting\n- 4-bit MLX affine embedding, attention, shared-expert, and routed-expert weights, with an 8-bit router\n- Custom Metal kernels for quantized GEMV, attention, MoE, normalization, RoPE, sampling, and production fusions\n- SSD-backed routed-expert streaming with a bounded expert cache\n- Chunked single-prompt prefill and token-by-token generation\n- FP16 KV storage with bounded circular storage for 25 sliding-window layers and linear storage for 5 full-attention layers\n- Exact split-K/V decode attention with distinct normalized K and V paths\n- A Swift library, streaming installer, command-line interface, loopback OpenAI-compatible server, and native SwiftUI/AppKit Mac app with a one-shot local decode service\n\nCurrent scope is text-only inference from the pinned Gemma 4 26B-A4B instruction checkpoint on Apple Silicon Macs with at least 8 GB of RAM.\n\n- Build iPhone and iPad apps, then measure inference speed and memory use on mobile hardware.\n- Benchmark more Apple Silicon Macs, especially the base 16 GB M4 Mac mini and other 8 GB models.\n\nThe [experiments that shaped TurboFieldfare](/drumih/turbo-fieldfare/blob/main/docs/OPTIMIZATION_JOURNEY.md)\nexplain the largest wins, the plausible ideas that failed, and the early\nresults that reversed under stronger validation. The detailed\n[experiment record](/drumih/turbo-fieldfare/blob/main/docs/experiments/EXPERIMENT_INVENTORY.md) keeps all 103\naudited entries as optional evidence.\n\nUseful entry points:\n\n[Local OpenAI-compatible server](/drumih/turbo-fieldfare/blob/main/docs/OPENAI_SERVER.md)[System design](/drumih/turbo-fieldfare/blob/main/docs/SYSTEM_DESIGN.md)[Benchmarks](/drumih/turbo-fieldfare/blob/main/docs/BENCHMARKS.md)[The experiments that shaped TurboFieldfare](/drumih/turbo-fieldfare/blob/main/docs/OPTIMIZATION_JOURNEY.md)[Experiment inventory and summaries](/drumih/turbo-fieldfare/blob/main/docs/experiments/EXPERIMENT_INVENTORY.md)[Implementation references](/drumih/turbo-fieldfare/blob/main/docs/IMPLEMENTATION_REFERENCES.md)\n\nTurboFieldfare's source and documentation are licensed under the\n[Apache License 2.0](/drumih/turbo-fieldfare/blob/main/LICENSE).\n\nModel weights are not included. The installer downloads them separately from\nthe pinned Hugging Face checkpoint, and the weights remain governed by their\nsource terms. See [THIRD_PARTY_NOTICES.md](/drumih/turbo-fieldfare/blob/main/THIRD_PARTY_NOTICES.md) for the model\nand Swift package license review.\n\nTurboFieldfare is an independent research project. It is not affiliated with, sponsored by, or endorsed by Google.\n\nThanks for checking out this project!\n\nMy name is Andrey Mikhaylov. You can find me on\n[LinkedIn](https://www.linkedin.com/in/andrey-mikhaylov-ios-dev/).\nI am the author of TurboFieldfare and an iOS and Metal engineer. Most of my\nwork is with images, video, and on-device AI.\n\nI dedicate this project to my wife, Sasha, the most supportive person I know. She stands by me even through the hardest times. She loves wildlife, goes birdwatching, and volunteers with our local birding community. Because of her, I have also grown closer to birds and nature.\n\nTurboFieldfare is named after the fieldfare, a member of the thrush family and my favourite bird. It is not the most noticeable or brightly coloured bird, but it definitely has a character and unique features of its own. I think the same is true of this project: it may not be the most practical, but I built it with my favourite tools, especially Metal, in my favourite field, on-device ML inference. It definitely has its own character and unique features.\n\nNext time you are outside, touch the grass and listen to the birds. Sometimes it is the most beautiful thing you can do. And if you can, support your local wildlife community. They do important work.\n\nThank you!", "url": "https://wpnews.pro/news/show-hn-open-source-engine-running-gemma-4-26b-in-2-gb-ram-on-any-m-series-mac", "canonical_source": "https://github.com/drumih/turbo-fieldfare", "published_at": "2026-07-29 15:05:43+00:00", "updated_at": "2026-07-29 16:10:40.831017+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["TurboFieldfare", "Gemma 4 26B-A4B", "Apple Silicon", "Metal", "Swift", "Google"], "alternates": {"html": "https://wpnews.pro/news/show-hn-open-source-engine-running-gemma-4-26b-in-2-gb-ram-on-any-m-series-mac", "markdown": "https://wpnews.pro/news/show-hn-open-source-engine-running-gemma-4-26b-in-2-gb-ram-on-any-m-series-mac.md", "text": "https://wpnews.pro/news/show-hn-open-source-engine-running-gemma-4-26b-in-2-gb-ram-on-any-m-series-mac.txt", "jsonld": "https://wpnews.pro/news/show-hn-open-source-engine-running-gemma-4-26b-in-2-gb-ram-on-any-m-series-mac.jsonld"}}