{"slug": "run-mixtral-8x7b-locally-with-llama-cpp-and-benchmark-moe-vs-dense", "title": "Run Mixtral 8x7B Locally with llama.cpp and Benchmark MoE vs. Dense", "summary": "Mistral AI's Mixtral 8x7B mixture-of-experts model can be run locally using llama.cpp build b10261, with generation speed tracking its ~12.9B active parameters while memory usage tracks all 46.7B parameters, according to a tutorial by Mariana Souza. The guide compiles llama.cpp from source, downloads quantized GGUF files (Mixtral Q4_K_M at 26.49 GiB and Qwen3-14B Q4_K_M at 9.0 GB as a dense baseline), and uses llama-bench to measure the trade-off. The tutorial requires ~40 GB disk space, 32 GB+ RAM or a 24 GB GPU with CPU offload, and verifies that the MoE architecture offers faster generation per active parameter but higher memory cost than dense models.", "body_md": "# Run Mixtral 8x7B Locally with llama.cpp and Benchmark MoE vs. Dense\n\nCompile llama.cpp, run quantized Mixtral 8x7B, and measure the mixture-of-experts speed-versus-memory trade-off yourself.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\n## What you'll build / learn\n\nYou'll compile [llama.cpp](https://github.com/ggml-org/llama.cpp) from source, run a 4-bit quantized [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) locally, and use `llama-bench`\n\nto prove the mixture-of-experts (MoE) trade-off with your own numbers: generation speed tracks the ~12.9B *active* parameters per token, while memory cost tracks all 46.7B.\n\n## Prerequisites\n\nVerified against llama.cpp build **b10261** (August 4, 2026) and the current Hugging Face `hf`\n\nCLI. Commands are for macOS/Linux; Windows users should work inside WSL2.\n\n**Disk:**~40 GB free (both models).** Memory:**the Q4_K_M Mixtral weighs 26.49 GiB. You need one of: 32 GB+ system RAM (CPU-only), a GPU with 32 GB+ VRAM for full offload, or a 24 GB GPU plus ~16 GB RAM using the MoE CPU-offload flag covered in step 5. On Apple Silicon, 48 GB unified memory is comfortable; on 32/36 GB machines use the Q3_K_M file (21.0 GiB) instead.**Toolchain:**`git`\n\n, CMake ≥ 3.14, and a C++17 compiler (Xcode Command Line Tools on macOS,`build-essential`\n\non Debian/Ubuntu).**NVIDIA only:**[CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit)12.x installed, with`nvcc`\n\non your PATH.\n\nllama.cpp ships new tagged builds daily and flags do change — everything below is checked against b10261.\n\n## 1. Build llama.cpp\n\n```\ngit clone https://github.com/ggml-org/llama.cpp\ncd llama.cpp\n```\n\nOn macOS, Metal GPU support is enabled by default, so a plain build is all you need:\n\n```\ncmake -B build\ncmake --build build --config Release -j 8\n```\n\nOn Linux with an NVIDIA GPU, turn on the CUDA backend:\n\n```\ncmake -B build -DGGML_CUDA=ON\ncmake --build build --config Release -j 8\n```\n\nBinaries land in `build/bin/`\n\n. Confirm the two you'll use exist:\n\n```\nls build/bin/llama-cli build/bin/llama-bench\n```\n\n## 2. Download the models\n\nInstall the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli) and pull single files — never clone a GGUF repo, it holds every quantization level:\n\n```\ncurl -LsSf https://hf.co/cli/install.sh | bash\n\nhf download mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF \\\n  Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf --local-dir models\n\nhf download Qwen/Qwen3-14B-GGUF \\\n  Qwen3-14B-Q4_K_M.gguf --local-dir models\n```\n\nThat's 28.4 GB for Mixtral and 9.0 GB for [Qwen3-14B](https://huggingface.co/Qwen/Qwen3-14B-GGUF), our dense baseline. Qwen3-14B is the fair fight: its 14.8B parameters sit right next to Mixtral's 12.9B active parameters, and both files use the same Q4_K_M quantization, so any throughput gap comes from architecture, not quant format.\n\nSkip the famous TheBloke Mixtral quants from December 2023 — they predate llama.cpp's 2024 switch to merged expert tensors and the GGUF chat-template metadata. The mradermacher conversion above uses the modern format.\n\n## 3. Smoke-test Mixtral\n\n```\n./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \\\n  -c 8192 -ngl 99\n```\n\n`-ngl 99`\n\noffloads every layer to the GPU (recent builds also accept `-ngl auto`\n\n); `-c 8192`\n\ncaps the context so the KV cache doesn't inflate memory use. Because the GGUF carries a chat template, `llama-cli`\n\ndrops you straight into interactive conversation mode. Ask it something, confirm you get coherent output, then exit with `Ctrl+C`\n\n— it prints token-throughput timings on the way out. Watch the load logs: a line per layer ending in `offloaded 33/33 layers to GPU`\n\n(32 blocks plus the output layer) means the whole model is resident on the GPU.\n\nFor a scripted one-shot run instead of chat, use `-no-cnv`\n\nwith Mixtral's `[INST]`\n\ntemplate:\n\n```\n./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \\\n  -c 4096 -ngl 99 -no-cnv -n 128 \\\n  -p \"[INST] Explain mixture-of-experts routing in two sentences. [/INST]\"\n```\n\n## 4. Benchmark MoE vs. dense\n\n`llama-bench`\n\nmeasures two phases: `pp`\n\n(prompt processing, compute-bound) and `tg`\n\n(token generation, memory-bandwidth-bound). Generation is where MoE shines, because each token only reads the router plus 2 of the 8 experts per layer.\n\n```\n./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \\\n  -m models/Qwen3-14B-Q4_K_M.gguf -p 512 -n 128 -ngl 99\n```\n\nEach test runs 5 repetitions by default and reports mean ± stddev; add `-r 3`\n\nif you're impatient.\n\n## 5. Fit it on a smaller GPU with --n-cpu-moe\n\nIf Mixtral doesn't fit your VRAM, don't drop whole layers with a lower `-ngl`\n\n. Use `--n-cpu-moe N`\n\n, which keeps the MoE expert weights of the first N layers in system RAM while attention, norms, and the router stay on the GPU. Experts are ~96% of Mixtral's weights but only a quarter of them fire per token, so this costs far less speed than evicting entire layers:\n\n```\n./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \\\n  -c 8192 -ngl 99 --n-cpu-moe 12\n```\n\nOn a 24 GB card, start around `--n-cpu-moe 12`\n\n(Mixtral has 32 layers) and lower it until you hit an out-of-memory error, then back off. `llama-bench`\n\ntakes the same flag as `-ncmoe`\n\nand accepts a comma-separated sweep, so you can find the knee of the curve in one command:\n\n```\n./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \\\n  -ngl 99 -ncmoe 12,16,24,32 -p 512 -n 128\n```\n\n## Verify it works\n\nSuccess looks like a Markdown table from step 4 in this shape (these reference numbers are from a 64 GB M3 Max — your absolute values will differ by hardware):\n\n```\n| model                | size      | params  | backend    | ngl | test   | t/s            |\n| -------------------- | --------: | ------: | ---------- | --: | ------ | -------------: |\n| llama 8x7B Q4_K_M    | 26.49 GiB | 46.70 B | Metal,BLAS |  99 | pp 512 | 235.11 ± 1.82  |\n| llama 8x7B Q4_K_M    | 26.49 GiB | 46.70 B | Metal,BLAS |  99 | tg 128 | 32.40 ± 0.21   |\n| qwen3 14B Q4_K_M     |  8.38 GiB | 14.77 B | Metal,BLAS |  99 | pp 512 | 341.27 ± 2.05  |\n| qwen3 14B Q4_K_M     |  8.38 GiB | 14.77 B | Metal,BLAS |  99 | tg 128 | 35.87 ± 0.33   |\n```\n\nThe result that matters is the ratio, not the absolute numbers: Mixtral's `tg 128`\n\nshould land within roughly 25% of the dense 14B's, despite carrying 3.2× the parameters. That's expert routing working — per token it reads about the same number of bytes as a 13B dense model. For contrast, a *dense* 46.7B model at Q4_K_M would stream all 26.49 GiB through memory for every token and generate roughly 3× slower. The MoE bill arrives in the `size`\n\ncolumn instead: 26.49 GiB resident versus 8.38 GiB.\n\n## Troubleshooting\n\n** ggml_backend_cuda_buffer_type_alloc_buffer: allocating 26550.00 MiB on device 0: cudaMalloc failed: out of memory** — the full Q4_K_M doesn't fit your card. Add\n\n`--n-cpu-moe 12`\n\n(raise N until it loads) rather than cutting `-ngl`\n\n, or use the Q3_K_M file.** CMake Error at ggml/src/ggml-cuda/CMakeLists.txt (message): CUDA Toolkit not found** — CMake can't see\n\n`nvcc`\n\n. Install CUDA Toolkit 12.x, then reconfigure with the compiler pinned: `CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build -DGGML_CUDA=ON`\n\n.**Mac loads the model but generation crawls, with a log warning that allocated size exceeds recommendedMaxWorkingSetSize** — macOS caps GPU-wired memory at roughly 70% of unified RAM, so a 26.49 GiB model thrashes on 32/36 GB machines. Raise the cap (resets on reboot) with\n\n`sudo sysctl iogpu.wired_limit_mb=30720`\n\n, or switch to Q3_K_M.** hf: command not found** — the installer put the binary in\n\n`~/.local/bin`\n\n, which isn't on your PATH. Run `export PATH=\"$HOME/.local/bin:$PATH\"`\n\n(add it to your shell rc), or `pip install -U huggingface_hub`\n\nto get the same CLI via pip.## Next steps\n\nSwap `llama-cli`\n\nfor `llama-server`\n\nand you get an OpenAI-compatible API on `localhost:8080`\n\nwith the same flags, including `--n-cpu-moe`\n\n. Run llama.cpp's `llama-perplexity`\n\ntool over both models to see the quality side of the trade — speed parity means little if the 47B MoE doesn't beat the dense 14B on your workload. Then point the same benchmark harness at newer MoE models like Qwen3-30B-A3B or OpenAI's gpt-oss-20b, where sparser routing (3B–4B active) makes the effect even more dramatic. When you outgrow static Q4_K_M files, look into imatrix-calibrated quants, which measurably cut quantization loss at the same size.\n\n## Sources & further reading\n\n-\n[llama.cpp build documentation](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)— github.com -\n[llama-bench README](https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md)— github.com -\n[llama-cli README](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md)— github.com -\n[Mixtral-8x7B-Instruct-v0.1-GGUF model card](https://huggingface.co/mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF)— huggingface.co -\n[Qwen3-14B-GGUF model card](https://huggingface.co/Qwen/Qwen3-14B-GGUF)— huggingface.co -\n[Hugging Face CLI guide](https://huggingface.co/docs/huggingface_hub/en/guides/cli)— huggingface.co\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/run-mixtral-8x7b-locally-with-llama-cpp-and-benchmark-moe-vs-dense", "canonical_source": "https://sourcefeed.dev/a/run-mixtral-8x7b-locally-with-llamacpp-and-benchmark-moe-vs-dense", "published_at": "2026-08-05 11:43:42+00:00", "updated_at": "2026-08-05 12:28:34.363964+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Mistral AI", "llama.cpp", "Mixtral 8x7B", "Qwen3-14B", "Hugging Face", "Mariana Souza"], "alternates": {"html": "https://wpnews.pro/news/run-mixtral-8x7b-locally-with-llama-cpp-and-benchmark-moe-vs-dense", "markdown": "https://wpnews.pro/news/run-mixtral-8x7b-locally-with-llama-cpp-and-benchmark-moe-vs-dense.md", "text": "https://wpnews.pro/news/run-mixtral-8x7b-locally-with-llama-cpp-and-benchmark-moe-vs-dense.txt", "jsonld": "https://wpnews.pro/news/run-mixtral-8x7b-locally-with-llama-cpp-and-benchmark-moe-vs-dense.jsonld"}}