# Run Mixtral 8x7B Locally with llama.cpp and Benchmark MoE vs. Dense > Source: > Published: 2026-08-05 11:43:42+00:00 # Run Mixtral 8x7B Locally with llama.cpp and Benchmark MoE vs. Dense Compile llama.cpp, run quantized Mixtral 8x7B, and measure the mixture-of-experts speed-versus-memory trade-off yourself. [Mariana Souza](https://sourcefeed.dev/u/mariana_souza) ## What you'll build / learn You'll compile [llama.cpp](https://github.com/ggml-org/llama.cpp) from source, run a 4-bit quantized [Mixtral 8x7B](https://mistral.ai/news/mixtral-of-experts/) locally, and use `llama-bench` to prove the mixture-of-experts (MoE) trade-off with your own numbers: generation speed tracks the ~12.9B *active* parameters per token, while memory cost tracks all 46.7B. ## Prerequisites Verified against llama.cpp build **b10261** (August 4, 2026) and the current Hugging Face `hf` CLI. Commands are for macOS/Linux; Windows users should work inside WSL2. **Disk:**~40 GB free (both models).** Memory:**the Q4_K_M Mixtral weighs 26.49 GiB. You need one of: 32 GB+ system RAM (CPU-only), a GPU with 32 GB+ VRAM for full offload, or a 24 GB GPU plus ~16 GB RAM using the MoE CPU-offload flag covered in step 5. On Apple Silicon, 48 GB unified memory is comfortable; on 32/36 GB machines use the Q3_K_M file (21.0 GiB) instead.**Toolchain:**`git` , CMake ≥ 3.14, and a C++17 compiler (Xcode Command Line Tools on macOS,`build-essential` on Debian/Ubuntu).**NVIDIA only:**[CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit)12.x installed, with`nvcc` on your PATH. llama.cpp ships new tagged builds daily and flags do change — everything below is checked against b10261. ## 1. Build llama.cpp ``` git clone https://github.com/ggml-org/llama.cpp cd llama.cpp ``` On macOS, Metal GPU support is enabled by default, so a plain build is all you need: ``` cmake -B build cmake --build build --config Release -j 8 ``` On Linux with an NVIDIA GPU, turn on the CUDA backend: ``` cmake -B build -DGGML_CUDA=ON cmake --build build --config Release -j 8 ``` Binaries land in `build/bin/` . Confirm the two you'll use exist: ``` ls build/bin/llama-cli build/bin/llama-bench ``` ## 2. Download the models Install the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli) and pull single files — never clone a GGUF repo, it holds every quantization level: ``` curl -LsSf https://hf.co/cli/install.sh | bash hf download mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF \ Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf --local-dir models hf download Qwen/Qwen3-14B-GGUF \ Qwen3-14B-Q4_K_M.gguf --local-dir models ``` That's 28.4 GB for Mixtral and 9.0 GB for [Qwen3-14B](https://huggingface.co/Qwen/Qwen3-14B-GGUF), our dense baseline. Qwen3-14B is the fair fight: its 14.8B parameters sit right next to Mixtral's 12.9B active parameters, and both files use the same Q4_K_M quantization, so any throughput gap comes from architecture, not quant format. Skip the famous TheBloke Mixtral quants from December 2023 — they predate llama.cpp's 2024 switch to merged expert tensors and the GGUF chat-template metadata. The mradermacher conversion above uses the modern format. ## 3. Smoke-test Mixtral ``` ./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \ -c 8192 -ngl 99 ``` `-ngl 99` offloads every layer to the GPU (recent builds also accept `-ngl auto` ); `-c 8192` caps the context so the KV cache doesn't inflate memory use. Because the GGUF carries a chat template, `llama-cli` drops you straight into interactive conversation mode. Ask it something, confirm you get coherent output, then exit with `Ctrl+C` — it prints token-throughput timings on the way out. Watch the load logs: a line per layer ending in `offloaded 33/33 layers to GPU` (32 blocks plus the output layer) means the whole model is resident on the GPU. For a scripted one-shot run instead of chat, use `-no-cnv` with Mixtral's `[INST]` template: ``` ./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \ -c 4096 -ngl 99 -no-cnv -n 128 \ -p "[INST] Explain mixture-of-experts routing in two sentences. [/INST]" ``` ## 4. Benchmark MoE vs. dense `llama-bench` measures two phases: `pp` (prompt processing, compute-bound) and `tg` (token generation, memory-bandwidth-bound). Generation is where MoE shines, because each token only reads the router plus 2 of the 8 experts per layer. ``` ./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \ -m models/Qwen3-14B-Q4_K_M.gguf -p 512 -n 128 -ngl 99 ``` Each test runs 5 repetitions by default and reports mean ± stddev; add `-r 3` if you're impatient. ## 5. Fit it on a smaller GPU with --n-cpu-moe If Mixtral doesn't fit your VRAM, don't drop whole layers with a lower `-ngl` . Use `--n-cpu-moe N` , which keeps the MoE expert weights of the first N layers in system RAM while attention, norms, and the router stay on the GPU. Experts are ~96% of Mixtral's weights but only a quarter of them fire per token, so this costs far less speed than evicting entire layers: ``` ./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \ -c 8192 -ngl 99 --n-cpu-moe 12 ``` On a 24 GB card, start around `--n-cpu-moe 12` (Mixtral has 32 layers) and lower it until you hit an out-of-memory error, then back off. `llama-bench` takes the same flag as `-ncmoe` and accepts a comma-separated sweep, so you can find the knee of the curve in one command: ``` ./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \ -ngl 99 -ncmoe 12,16,24,32 -p 512 -n 128 ``` ## Verify it works Success looks like a Markdown table from step 4 in this shape (these reference numbers are from a 64 GB M3 Max — your absolute values will differ by hardware): ``` | model | size | params | backend | ngl | test | t/s | | -------------------- | --------: | ------: | ---------- | --: | ------ | -------------: | | llama 8x7B Q4_K_M | 26.49 GiB | 46.70 B | Metal,BLAS | 99 | pp 512 | 235.11 ± 1.82 | | llama 8x7B Q4_K_M | 26.49 GiB | 46.70 B | Metal,BLAS | 99 | tg 128 | 32.40 ± 0.21 | | qwen3 14B Q4_K_M | 8.38 GiB | 14.77 B | Metal,BLAS | 99 | pp 512 | 341.27 ± 2.05 | | qwen3 14B Q4_K_M | 8.38 GiB | 14.77 B | Metal,BLAS | 99 | tg 128 | 35.87 ± 0.33 | ``` The result that matters is the ratio, not the absolute numbers: Mixtral's `tg 128` should land within roughly 25% of the dense 14B's, despite carrying 3.2× the parameters. That's expert routing working — per token it reads about the same number of bytes as a 13B dense model. For contrast, a *dense* 46.7B model at Q4_K_M would stream all 26.49 GiB through memory for every token and generate roughly 3× slower. The MoE bill arrives in the `size` column instead: 26.49 GiB resident versus 8.38 GiB. ## Troubleshooting ** ggml_backend_cuda_buffer_type_alloc_buffer: allocating 26550.00 MiB on device 0: cudaMalloc failed: out of memory** — the full Q4_K_M doesn't fit your card. Add `--n-cpu-moe 12` (raise N until it loads) rather than cutting `-ngl` , or use the Q3_K_M file.** CMake Error at ggml/src/ggml-cuda/CMakeLists.txt (message): CUDA Toolkit not found** — CMake can't see `nvcc` . Install CUDA Toolkit 12.x, then reconfigure with the compiler pinned: `CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build -DGGML_CUDA=ON` .**Mac loads the model but generation crawls, with a log warning that allocated size exceeds recommendedMaxWorkingSetSize** — macOS caps GPU-wired memory at roughly 70% of unified RAM, so a 26.49 GiB model thrashes on 32/36 GB machines. Raise the cap (resets on reboot) with `sudo sysctl iogpu.wired_limit_mb=30720` , or switch to Q3_K_M.** hf: command not found** — the installer put the binary in `~/.local/bin` , which isn't on your PATH. Run `export PATH="$HOME/.local/bin:$PATH"` (add it to your shell rc), or `pip install -U huggingface_hub` to get the same CLI via pip.## Next steps Swap `llama-cli` for `llama-server` and you get an OpenAI-compatible API on `localhost:8080` with the same flags, including `--n-cpu-moe` . Run llama.cpp's `llama-perplexity` tool over both models to see the quality side of the trade — speed parity means little if the 47B MoE doesn't beat the dense 14B on your workload. Then point the same benchmark harness at newer MoE models like Qwen3-30B-A3B or OpenAI's gpt-oss-20b, where sparser routing (3B–4B active) makes the effect even more dramatic. When you outgrow static Q4_K_M files, look into imatrix-calibrated quants, which measurably cut quantization loss at the same size. ## Sources & further reading - [llama.cpp build documentation](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md)— github.com - [llama-bench README](https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md)— github.com - [llama-cli README](https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md)— github.com - [Mixtral-8x7B-Instruct-v0.1-GGUF model card](https://huggingface.co/mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF)— huggingface.co - [Qwen3-14B-GGUF model card](https://huggingface.co/Qwen/Qwen3-14B-GGUF)— huggingface.co - [Hugging Face CLI guide](https://huggingface.co/docs/huggingface_hub/en/guides/cli)— huggingface.co [Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. ## Discussion 0 No comments yet Be the first to weigh in.