{"slug": "show-hn-run-faster-moe-inference-on-mac-with-less-ram", "title": "Show HN: Run faster MoE inference on Mac with less RAM", "summary": "MoEShift, an alpha-release local inference runtime from developer Davide Troiani, runs a 125-billion-parameter Qwen 3.8 Flash Next model on Apple Silicon at up to 11 tokens per second with under 10 GB of peak process memory. On a Mac mini M4 Pro with 24 GB of unified memory, MoEShift used 6.4 GB versus Strata 0.2.7's 14 GB on a 66-token prompt and 7.5 GB versus 14 GB on a 6,009-token prompt, while processing that long prompt at 160.8 tokens per second against Strata's 61.8. The runtime, installable via Homebrew, streams MoE experts from SSD and requires at least 10 GB of RAM and roughly 120 GB of SSD space for the Flash Next setup.", "body_md": "**Large AI models on your Mac. Starting with 10 GB of RAM.**\n\n**125 billion parameters. Up to 11 tokens per second. Under 10 GB of\npeak process memory.** MoEShift is optimized for Apple Silicon: on a Mac mini\nM4 Pro with 24 GB of unified memory, it runs Qwen 3.8 Flash Next with roughly\nhalf the peak process memory of Strata and processes long prompts 2.6× faster.\n\nMoEShift leaves room for your other apps, helping your Mac stay responsive during inference instead of grinding to a halt under memory pressure. Keep browsing, writing and coding while the model works—all locally, with no cloud subscription.\n\n*Qwen 3.8 Flash Next running locally with MoEShift, connected to OpenCode.*\n\n**Alpha release.** The core runtime is working, and the project is evolving\nquickly. You may encounter rough edges; your feedback helps shape the next release.\nTested models and quantizations are listed below.\n\n| Model | GGUF architecture | Tested quantization | \n|---|---|---|\n| Qwen 3.8 Flash Next (125B) | `qwen4exp` | ISTA-DASLab GSQ-RCO IQ3_XXS | \n| Qwen3.6-35B-A3B | `qwen35moe` | UD Q6_K XL (Q6_K/Q8_0 experts) | \n\nBoth use the shared SSD expert runtime on Apple Silicon. See\n[supported architectures](https://github.com/davide-troiani/moeshift/blob/main/docs/architecture.md#supported-llm-architectures).\n\nCompared with **Strata**, a popular choice for local MoE inference, on a\n**Mac mini M4 Pro with 24 GB unified memory**. Both ran Qwen 3.8 Flash Next\n(125B), ISTA-DASLab GSQ-RCO IQ3_XXS, with an 8,192-token context, no cached prompt\ntokens and 256 output tokens. Strata: 0.2.7, build 0.1.52.\n\n| Measurement | Short prompt: MoEShift / Strata | Long prompt: MoEShift / Strata | \n|---|---|---|\n| Input tokens | 66 / 66 | 6,009 / 6,009 | \n| Inference time ↓ | **29.6 / 42.1 s** | **63.3 / 129.9 s** | \n| Prompt processing ↑ | — | **160.8 / 61.8 tok/s** | \n| Text generation ↑ | **10.44 / 8.4 tok/s** | **9.9 / 7.8 tok/s** | \n| Peak process memory ↓ | **6.4 / 14 GB** | **7.5 / 14 GB** | \n\nTime includes server-measured prompt processing and generation, excluding\nstartup. Memory is macOS `footprint -p` peak physical process footprint.\n\nUD Q6_K XL on the same Mac mini M4 Pro with 24 GB unified memory.\nMedians of three measured runs after warmup, with no cached prompt tokens and\n512 output tokens, using the [benchmark runner](https://github.com/davide-troiani/moeshift/blob/main/benchmarks/README.md).\nThese runs use different output lengths from the Flash Next comparison above.\n\n| Measurement | Short prompt | Long prompt | \n|---|---|---|\n| Input tokens | 72 | 6,060 | \n| Inference time ↓ | **34.25 s** | **49.45 s** | \n| Prompt processing ↑ | - | **417.24 tok/s** | \n| Text generation ↑ | **16.50 tok/s** | **14.66 tok/s** | \n| Peak process memory ↓ | **5.4 GB** | **5.6 GB** | \n\n- A **Mac with Apple Silicon** (M-series).\n- At least **10 GB of RAM** .\n- A fast SSD with space for both the GGUF and prepared expert files: at least\n**120 GB** for the Flash Next setup below, or about**60 GB** for Qwen3.6 UD Q6_K XL.\n- [Homebrew](https://brew.sh/) , the package manager for macOS. If you have never\nused it, follow its installation instructions, then return here.\n\nOpen **Terminal** and run:\n\n```\nbrew install davide-troiani/moeshift/moeshift\n```\n\nThe installation includes model preparation tools. MoEShift creates\n`~/.config/moeshift/config.json` automatically on first launch and preserves\nyour settings on subsequent launches and upgrades.\n\nChoose a model and quantization from its download page:\n\n| Model | Download | \n|---|---|\n| Qwen 3.8 Flash Next | [ISTA-DASLab GGUF files](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/tree/main) | \n| Qwen3.6-35B-A3B | [Unsloth GGUF files](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) | \n\nQuantization is your choice: smaller files reduce storage and expert-cache memory; higher precision requires more space and memory. The benchmarks above apply only to the tested quantizations listed above.\n\nDownload all shards of your chosen quantization into the same folder, keeping\ntheir names. Both models use the same preparation steps and the same\n`~/.config/moeshift/config.json`; each has its own entry under `models`.\n\nThe commands below use Flash Next **GSQ-RCO IQ3_XXS** as an example:\n\n- [Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/main/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf?download=true)\n- [Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/main/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf?download=true)\n\nThese two files total about **76 GB**. Reuse them if already downloaded.\nMoEShift does not distribute or requantize models.\n\nRun this once to copy the model's specialized parts (“experts”) into files\nMoEShift can read directly from SSD. Keep all original GGUF files.\nSet `MODEL` to your GGUF (the first shard for split files) and `EXPERTS` to a\nseparate output folder for that model and quantization:\n\n```\nMODEL=\"$HOME/Downloads/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf\"\nEXPERTS=\"$HOME/Models/moeshift/qwen-iq3-experts\"\nMOESHIFT_PREFIX=\"$(brew --prefix moeshift)\"\n\"$MOESHIFT_PREFIX/libexec/bin/python\" \\\n  \"$MOESHIFT_PREFIX/share/moeshift/tools/moe/experts.py\" \\\n  extract --model \"$MODEL\" --out-dir \"$EXPERTS\"\n```\n\nThen check that the new files match the downloaded model:\n\n```\n\"$MOESHIFT_PREFIX/libexec/bin/python\" \\\n  \"$MOESHIFT_PREFIX/share/moeshift/tools/moe/experts.py\" \\\n  verify --model \"$MODEL\" --experts \"$EXPERTS\"\n```\n\nWait for both commands to finish successfully. Verification compares the copied\nweights with the originals. Keep the output folder intact (about **43 GB** for\nthe Flash Next IQ3_XXS example).\nIf interrupted, retry into a new empty folder and verify again.\n\nRun `moeshift` once to create the configuration, then stop it with Ctrl-C.\nOpen the configuration:\n\n```\nopen -e \"$HOME/.config/moeshift/config.json\"\n```\n\nInside `models` → your chosen model ID → `args`, replace these two values:\n\n| Setting | What to enter | \n|---|---|\n| `model` | The full path to the **first** downloaded`.gguf` file. | \n| `moe-experts` | The full path to the folder created in step 3. | \n\nUse full paths in JSON (not `~` or `$HOME`), keep the quotes and commas, and save.\nBoth model profiles are included in the template. Existing user configurations\nare preserved; copy the desired profile from [the template](https://github.com/davide-troiani/moeshift/blob/main/configs/moeshift.json)\nif it is missing. Use its model ID in API requests and clients. Leave the other\nsettings at their defaults for your first run.\n\n```\nmoeshift\n```\n\nLeave this Terminal open. MoEShift starts a local server at\n`http://127.0.0.1:8080`. Stop it with **Ctrl-C**. Next time, just run the same\ncommand; you do not need to repeat the model preparation.\n\nOpen another Terminal and run:\n\n```\ncurl http://127.0.0.1:8080/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"model\":\"qwen-3.8-flash-next-rco-gsq-iq3-xxs\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hello in one sentence.\"}],\"chat_template_kwargs\":{\"enable_thinking\":false},\"max_tokens\":128,\"stream\":false}'\n```\n\nIn any client that supports an OpenAI-compatible API, enter:\n\n| Field | Value | \n|---|---|\n| Base URL | `http://127.0.0.1:8080/v1` | \n| Model | `qwen-3.8-flash-next-rco-gsq-iq3-xxs` | \n\nMoEShift handles one inference request at a time and listens only on localhost.\n\nRead the [MoEShift architecture](https://github.com/davide-troiani/moeshift/blob/main/docs/architecture.md) for the project’s design:\nLES (Lookahead Expert Scheduling), expert loading and Metal execution, cache\npolicy, memory management, and server/worker lifetimes.\n\nYou can adjust these settings in the configuration file to suit your Mac.\nWith more RAM, you can increase `moe-prefill-bank` and\n`LLAMA_MOE_PROTECTED_SLOTS` to keep more model data in memory and reduce SSD\nreads. Lower them to use less RAM. For better control, change one setting at a time and restart\nMoEShift to apply it.\n\n| Setting | Default | Direct effect | \n|---|---|---|\n| `args.ctx-size` | `65536` | Maximum conversation length in tokens. Lower it to limit memory growth during long conversations; raise it to allow longer conversations, with more memory use as they grow. | \n| `args.moe-prefill-bank` | `512` /`256` ¹ | Expert cache used when reading your prompt. More slots can reduce SSD reads and speed up prompt processing, using more RAM; fewer slots save RAM. | \n| `args.prefill-max-tokens` | `3072` | Amount of prompt text processed per batch. Smaller batches reduce peak working memory; larger batches can improve prompt-processing speed. | \n| `env.LLAMA_MOE_PROTECTED_SLOTS` | `24` | Extra experts kept in memory while generating replies. More slots can reduce SSD reads and improve speed, using more RAM; fewer slots save RAM. | \n| `args.threads` | `10` | CPU threads available to inference. More threads can help CPU work but compete with other apps; more is not always faster. | \n| `env.LLAMA_GPU_KEEPALIVE` | `1` | Keeps the GPU active to reduce wake-up delays. Set to `0` to reduce idle GPU activity and power use, but expect lower speeds. | \n| `env.LLAMA_MOE_QUIET` | `1` | Keeps expert diagnostics out of Terminal. Set to `0` for more detail when investigating a problem. | \n| `port` | `8080` | Change this if another application already uses the port; use the same port in your client. | \n\n¹ Flash Next / Qwen3.6. Slot sizes depend on the model and quantization;\nequal slot counts do not imply equal memory use. The Flash Next defaults were\ntuned on a Mac mini M4 Pro with 24 GB unified memory. The\n[full configuration reference](https://github.com/davide-troiani/moeshift/blob/main/configs/README.md) covers the remaining options,\nvalid combinations and command-line overrides. Settings under `env` are edited\nin this file, rather than exported in Terminal.\n\nFound a problem or have an idea?\n[Open an issue](https://github.com/davide-troiani/moeshift/issues/new/choose),\nafter checking whether someone has already reported it.\n\nChoose **Bug report** for installation, startup or inference problems, or\n**Feature request** for suggestions, or **Benchmark result** to share measurements\nfrom your Mac. The form guides you through the useful\ninformation: your Mac chip and RAM, macOS version, MoEShift version, model files,\nsteps to reproduce and relevant Terminal output. Remove private prompts,\ncredentials and personal paths before posting.\n\nSee [CONTRIBUTING.md](https://github.com/davide-troiani/moeshift/blob/main/CONTRIBUTING.md) for fixes, suggestions and pull requests.\n\n**Help someone with a Mac like yours decide whether MoEShift fits their day.**\nCan you keep browsing in Safari while it answers? How does it feel on battery,\nor with your usual apps open? Share a benchmark from your MacBook, Mac mini or\nMac Studio—every chip and memory size adds a useful piece of the picture.\n\nYou do not need a perfectly idle machine. Keep your normal setup and tell us\nwhat you were doing. The [benchmark runner](https://github.com/davide-troiani/moeshift/blob/main/benchmarks/README.md) handles the\nfixed short and long prompts, repeats the measurements and creates a JSON report\nfor you. It collects benchmark metrics only; you describe your Mac and working\nconditions yourself.\n\nThen [share your results](https://github.com/davide-troiani/moeshift/issues/new?template=benchmark_result.yml)\nand attach the report. Screenshots and a few words about how responsive your\nother apps felt are welcome too. Your experience can help the next user find\nout what to expect—and help us see where MoEShift needs to improve.\n\nThe alpha targets Apple Silicon Macs. Some remaining documents describe upstream\ncomponents and may mention platforms or features outside MoEShift's scope; they\nwill be cleaned up over time. Start with this README and the\n[configuration reference](https://github.com/davide-troiani/moeshift/blob/main/configs/README.md). Runtime details:\n[adaptive prefill](https://github.com/davide-troiani/moeshift/blob/main/docs/les/adaptive-prefill.md),\n[protected cache](https://github.com/davide-troiani/moeshift/blob/main/docs/les/protected-decode-cache.md), and\n[GPU keepalive](https://github.com/davide-troiani/moeshift/blob/main/docs/les/gpu-keepalive.md).\n\nMoEShift builds on [tc-mb/llama.cpp-omni](https://github.com/tc-mb/llama.cpp-omni),\n[llama.cpp](https://github.com/ggml-org/llama.cpp) and\n[ggml](https://github.com/ggml-org/ggml). The inherited MoE-SSD/LES foundations\nprecede MoEShift's Qwen runtime, Metal scheduling and cache changes. The protected\ncache adapts the demand-weighted decay idea from\n[LRFU](https://www.openu.ac.il/home/wiseman/2os/lru/lrfu.pdf).\n\nModel and quantization credits belong to their respective authors; the\nrecommended GGUF files are published by\n[ISTA-DASLab](https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF).\n\nOriginal Git history and copyright notices are retained. See [LICENSE](https://github.com/davide-troiani/moeshift/blob/main/LICENSE)\nand the bundled third-party licenses. Model weights retain their own licenses.", "url": "https://wpnews.pro/news/show-hn-run-faster-moe-inference-on-mac-with-less-ram", "canonical_source": "https://github.com/davide-troiani/moeshift", "published_at": "2026-11-03 21:17:06+00:00", "updated_at": "2026-11-03 21:20:16.920405+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["MoEShift", "Davide Troiani", "Apple Silicon", "Qwen 3.8 Flash Next", "Strata", "Mac mini M4 Pro", "Qwen3.6-35B-A3B", "Homebrew"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-run-faster-moe-inference-on-mac-with-less-ram", "markdown": "https://wpnews.pro/news/show-hn-run-faster-moe-inference-on-mac-with-less-ram.md", "text": "https://wpnews.pro/news/show-hn-run-faster-moe-inference-on-mac-with-less-ram.txt", "jsonld": "https://wpnews.pro/news/show-hn-run-faster-moe-inference-on-mac-with-less-ram.jsonld"}}