Show HN: Run faster MoE inference on Mac with less RAM

MoEShift, an alpha-release local inference runtime from developer Davide Troiani, runs a 125-billion-parameter Qwen 3.8 Flash Next model on Apple Silicon at up to 11 tokens per second with under 10 GB of peak process memory. On a Mac mini M4 Pro with 24 GB of unified memory, MoEShift used 6.4 GB versus Strata 0.2.7's 14 GB on a 66-token prompt and 7.5 GB versus 14 GB on a 6,009-token prompt, while processing that long prompt at 160.8 tokens per second against Strata's 61.8. The runtime, installable via Homebrew, streams MoE experts from SSD and requires at least 10 GB of RAM and roughly 120 GB of SSD space for the Flash Next setup.

Large AI models on your Mac. Starting with 10 GB of RAM. 125 billion parameters. Up to 11 tokens per second. Under 10 GB of peak process memory. MoEShift is optimized for Apple Silicon: on a Mac mini M4 Pro with 24 GB of unified memory, it runs Qwen 3.8 Flash Next with roughly half the peak process memory of Strata and processes long prompts 2.6× faster. MoEShift leaves room for your other apps, helping your Mac stay responsive during inference instead of grinding to a halt under memory pressure. Keep browsing, writing and coding while the model works—all locally, with no cloud subscription. Qwen 3.8 Flash Next running locally with MoEShift, connected to OpenCode. Alpha release. The core runtime is working, and the project is evolving quickly. You may encounter rough edges; your feedback helps shape the next release. Tested models and quantizations are listed below. | Model | GGUF architecture | Tested quantization | |---|---|---| | Qwen 3.8 Flash Next 125B | qwen4exp | ISTA-DASLab GSQ-RCO IQ3 XXS | | Qwen3.6-35B-A3B | qwen35moe | UD Q6 K XL Q6 K/Q8 0 experts | Both use the shared SSD expert runtime on Apple Silicon. See supported architectures https://github.com/davide-troiani/moeshift/blob/main/docs/architecture.md supported-llm-architectures . Compared with Strata , a popular choice for local MoE inference, on a Mac mini M4 Pro with 24 GB unified memory . Both ran Qwen 3.8 Flash Next 125B , ISTA-DASLab GSQ-RCO IQ3 XXS, with an 8,192-token context, no cached prompt tokens and 256 output tokens. Strata: 0.2.7, build 0.1.52. | Measurement | Short prompt: MoEShift / Strata | Long prompt: MoEShift / Strata | |---|---|---| | Input tokens | 66 / 66 | 6,009 / 6,009 | | Inference time ↓ | 29.6 / 42.1 s | 63.3 / 129.9 s | | Prompt processing ↑ | — | 160.8 / 61.8 tok/s | | Text generation ↑ | 10.44 / 8.4 tok/s | 9.9 / 7.8 tok/s | | Peak process memory ↓ | 6.4 / 14 GB | 7.5 / 14 GB | Time includes server-measured prompt processing and generation, excluding startup. Memory is macOS footprint -p peak physical process footprint. UD Q6 K XL on the same Mac mini M4 Pro with 24 GB unified memory. Medians of three measured runs after warmup, with no cached prompt tokens and 512 output tokens, using the benchmark runner https://github.com/davide-troiani/moeshift/blob/main/benchmarks/README.md . These runs use different output lengths from the Flash Next comparison above. | Measurement | Short prompt | Long prompt | |---|---|---| | Input tokens | 72 | 6,060 | | Inference time ↓ | 34.25 s | 49.45 s | | Prompt processing ↑ | - | 417.24 tok/s | | Text generation ↑ | 16.50 tok/s | 14.66 tok/s | | Peak process memory ↓ | 5.4 GB | 5.6 GB | - A Mac with Apple Silicon M-series . - At least 10 GB of RAM . - A fast SSD with space for both the GGUF and prepared expert files: at least 120 GB for the Flash Next setup below, or about 60 GB for Qwen3.6 UD Q6 K XL. - Homebrew https://brew.sh/ , the package manager for macOS. If you have never used it, follow its installation instructions, then return here. Open Terminal and run: brew install davide-troiani/moeshift/moeshift The installation includes model preparation tools. MoEShift creates ~/.config/moeshift/config.json automatically on first launch and preserves your settings on subsequent launches and upgrades. Choose a model and quantization from its download page: | Model | Download | |---|---| | Qwen 3.8 Flash Next | ISTA-DASLab GGUF files https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/tree/main | | Qwen3.6-35B-A3B | Unsloth GGUF files https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF | Quantization is your choice: smaller files reduce storage and expert-cache memory; higher precision requires more space and memory. The benchmarks above apply only to the tested quantizations listed above. Download all shards of your chosen quantization into the same folder, keeping their names. Both models use the same preparation steps and the same ~/.config/moeshift/config.json ; each has its own entry under models . The commands below use Flash Next GSQ-RCO IQ3 XXS as an example: - Qwen3.8-Flash-Next-GSQ-RCO-IQ3 XXS-00001-of-00002.gguf https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/main/IQ3 XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3 XXS-00001-of-00002.gguf?download=true - Qwen3.8-Flash-Next-GSQ-RCO-IQ3 XXS-00002-of-00002.gguf https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/main/IQ3 XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3 XXS-00002-of-00002.gguf?download=true These two files total about 76 GB . Reuse them if already downloaded. MoEShift does not distribute or requantize models. Run this once to copy the model's specialized parts “experts” into files MoEShift can read directly from SSD. Keep all original GGUF files. Set MODEL to your GGUF the first shard for split files and EXPERTS to a separate output folder for that model and quantization: MODEL="$HOME/Downloads/Qwen3.8-Flash-Next-GSQ-RCO-IQ3 XXS-00001-of-00002.gguf" EXPERTS="$HOME/Models/moeshift/qwen-iq3-experts" MOESHIFT PREFIX="$ brew --prefix moeshift " "$MOESHIFT PREFIX/libexec/bin/python" \ "$MOESHIFT PREFIX/share/moeshift/tools/moe/experts.py" \ extract --model "$MODEL" --out-dir "$EXPERTS" Then check that the new files match the downloaded model: "$MOESHIFT PREFIX/libexec/bin/python" \ "$MOESHIFT PREFIX/share/moeshift/tools/moe/experts.py" \ verify --model "$MODEL" --experts "$EXPERTS" Wait for both commands to finish successfully. Verification compares the copied weights with the originals. Keep the output folder intact about 43 GB for the Flash Next IQ3 XXS example . If interrupted, retry into a new empty folder and verify again. Run moeshift once to create the configuration, then stop it with Ctrl-C. Open the configuration: open -e "$HOME/.config/moeshift/config.json" Inside models → your chosen model ID → args , replace these two values: | Setting | What to enter | |---|---| | model | The full path to the first downloaded .gguf file. | | moe-experts | The full path to the folder created in step 3. | Use full paths in JSON not ~ or $HOME , keep the quotes and commas, and save. Both model profiles are included in the template. Existing user configurations are preserved; copy the desired profile from the template https://github.com/davide-troiani/moeshift/blob/main/configs/moeshift.json if it is missing. Use its model ID in API requests and clients. Leave the other settings at their defaults for your first run. moeshift Leave this Terminal open. MoEShift starts a local server at http://127.0.0.1:8080 . Stop it with Ctrl-C . Next time, just run the same command; you do not need to repeat the model preparation. Open another Terminal and run: curl http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"qwen-3.8-flash-next-rco-gsq-iq3-xxs","messages": {"role":"user","content":"Say hello in one sentence."} ,"chat template kwargs":{"enable thinking":false},"max tokens":128,"stream":false}' In any client that supports an OpenAI-compatible API, enter: | Field | Value | |---|---| | Base URL | http://127.0.0.1:8080/v1 | | Model | qwen-3.8-flash-next-rco-gsq-iq3-xxs | MoEShift handles one inference request at a time and listens only on localhost. Read the MoEShift architecture https://github.com/davide-troiani/moeshift/blob/main/docs/architecture.md for the project’s design: LES Lookahead Expert Scheduling , expert loading and Metal execution, cache policy, memory management, and server/worker lifetimes. You can adjust these settings in the configuration file to suit your Mac. With more RAM, you can increase moe-prefill-bank and LLAMA MOE PROTECTED SLOTS to keep more model data in memory and reduce SSD reads. Lower them to use less RAM. For better control, change one setting at a time and restart MoEShift to apply it. | Setting | Default | Direct effect | |---|---|---| | args.ctx-size | 65536 | Maximum conversation length in tokens. Lower it to limit memory growth during long conversations; raise it to allow longer conversations, with more memory use as they grow. | | args.moe-prefill-bank | 512 / 256 ¹ | Expert cache used when reading your prompt. More slots can reduce SSD reads and speed up prompt processing, using more RAM; fewer slots save RAM. | | args.prefill-max-tokens | 3072 | Amount of prompt text processed per batch. Smaller batches reduce peak working memory; larger batches can improve prompt-processing speed. | | env.LLAMA MOE PROTECTED SLOTS | 24 | Extra experts kept in memory while generating replies. More slots can reduce SSD reads and improve speed, using more RAM; fewer slots save RAM. | | args.threads | 10 | CPU threads available to inference. More threads can help CPU work but compete with other apps; more is not always faster. | | env.LLAMA GPU KEEPALIVE | 1 | Keeps the GPU active to reduce wake-up delays. Set to 0 to reduce idle GPU activity and power use, but expect lower speeds. | | env.LLAMA MOE QUIET | 1 | Keeps expert diagnostics out of Terminal. Set to 0 for more detail when investigating a problem. | | port | 8080 | Change this if another application already uses the port; use the same port in your client. | ¹ Flash Next / Qwen3.6. Slot sizes depend on the model and quantization; equal slot counts do not imply equal memory use. The Flash Next defaults were tuned on a Mac mini M4 Pro with 24 GB unified memory. The full configuration reference https://github.com/davide-troiani/moeshift/blob/main/configs/README.md covers the remaining options, valid combinations and command-line overrides. Settings under env are edited in this file, rather than exported in Terminal. Found a problem or have an idea? Open an issue https://github.com/davide-troiani/moeshift/issues/new/choose , after checking whether someone has already reported it. Choose Bug report for installation, startup or inference problems, or Feature request for suggestions, or Benchmark result to share measurements from your Mac. The form guides you through the useful information: your Mac chip and RAM, macOS version, MoEShift version, model files, steps to reproduce and relevant Terminal output. Remove private prompts, credentials and personal paths before posting. See CONTRIBUTING.md https://github.com/davide-troiani/moeshift/blob/main/CONTRIBUTING.md for fixes, suggestions and pull requests. Help someone with a Mac like yours decide whether MoEShift fits their day. Can you keep browsing in Safari while it answers? How does it feel on battery, or with your usual apps open? Share a benchmark from your MacBook, Mac mini or Mac Studio—every chip and memory size adds a useful piece of the picture. You do not need a perfectly idle machine. Keep your normal setup and tell us what you were doing. The benchmark runner https://github.com/davide-troiani/moeshift/blob/main/benchmarks/README.md handles the fixed short and long prompts, repeats the measurements and creates a JSON report for you. It collects benchmark metrics only; you describe your Mac and working conditions yourself. Then share your results https://github.com/davide-troiani/moeshift/issues/new?template=benchmark result.yml and attach the report. Screenshots and a few words about how responsive your other apps felt are welcome too. Your experience can help the next user find out what to expect—and help us see where MoEShift needs to improve. The alpha targets Apple Silicon Macs. Some remaining documents describe upstream components and may mention platforms or features outside MoEShift's scope; they will be cleaned up over time. Start with this README and the configuration reference https://github.com/davide-troiani/moeshift/blob/main/configs/README.md . Runtime details: adaptive prefill https://github.com/davide-troiani/moeshift/blob/main/docs/les/adaptive-prefill.md , protected cache https://github.com/davide-troiani/moeshift/blob/main/docs/les/protected-decode-cache.md , and GPU keepalive https://github.com/davide-troiani/moeshift/blob/main/docs/les/gpu-keepalive.md . MoEShift builds on tc-mb/llama.cpp-omni https://github.com/tc-mb/llama.cpp-omni , llama.cpp https://github.com/ggml-org/llama.cpp and ggml https://github.com/ggml-org/ggml . The inherited MoE-SSD/LES foundations precede MoEShift's Qwen runtime, Metal scheduling and cache changes. The protected cache adapts the demand-weighted decay idea from LRFU https://www.openu.ac.il/home/wiseman/2os/lru/lrfu.pdf . Model and quantization credits belong to their respective authors; the recommended GGUF files are published by ISTA-DASLab https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF . Original Git history and copyright notices are retained. See LICENSE https://github.com/davide-troiani/moeshift/blob/main/LICENSE and the bundled third-party licenses. Model weights retain their own licenses.