Run Mixtral 8x7B Locally with llama.cpp and Benchmark MoE vs. Dense

Mistral AI's Mixtral 8x7B mixture-of-experts model can be run locally using llama.cpp build b10261, with generation speed tracking its ~12.9B active parameters while memory usage tracks all 46.7B parameters, according to a tutorial by Mariana Souza. The guide compiles llama.cpp from source, downloads quantized GGUF files (Mixtral Q4_K_M at 26.49 GiB and Qwen3-14B Q4_K_M at 9.0 GB as a dense baseline), and uses llama-bench to measure the trade-off. The tutorial requires ~40 GB disk space, 32 GB+ RAM or a 24 GB GPU with CPU offload, and verifies that the MoE architecture offers faster generation per active parameter but higher memory cost than dense models.

Run Mixtral 8x7B Locally with llama.cpp and Benchmark MoE vs. Dense Compile llama.cpp, run quantized Mixtral 8x7B, and measure the mixture-of-experts speed-versus-memory trade-off yourself. Mariana Souza https://sourcefeed.dev/u/mariana souza What you'll build / learn You'll compile llama.cpp https://github.com/ggml-org/llama.cpp from source, run a 4-bit quantized Mixtral 8x7B https://mistral.ai/news/mixtral-of-experts/ locally, and use llama-bench to prove the mixture-of-experts MoE trade-off with your own numbers: generation speed tracks the ~12.9B active parameters per token, while memory cost tracks all 46.7B. Prerequisites Verified against llama.cpp build b10261 August 4, 2026 and the current Hugging Face hf CLI. Commands are for macOS/Linux; Windows users should work inside WSL2. Disk: ~40 GB free both models . Memory: the Q4 K M Mixtral weighs 26.49 GiB. You need one of: 32 GB+ system RAM CPU-only , a GPU with 32 GB+ VRAM for full offload, or a 24 GB GPU plus ~16 GB RAM using the MoE CPU-offload flag covered in step 5. On Apple Silicon, 48 GB unified memory is comfortable; on 32/36 GB machines use the Q3 K M file 21.0 GiB instead. Toolchain: git , CMake ≥ 3.14, and a C++17 compiler Xcode Command Line Tools on macOS, build-essential on Debian/Ubuntu . NVIDIA only: CUDA Toolkit https://developer.nvidia.com/cuda-toolkit 12.x installed, with nvcc on your PATH. llama.cpp ships new tagged builds daily and flags do change — everything below is checked against b10261. 1. Build llama.cpp git clone https://github.com/ggml-org/llama.cpp cd llama.cpp On macOS, Metal GPU support is enabled by default, so a plain build is all you need: cmake -B build cmake --build build --config Release -j 8 On Linux with an NVIDIA GPU, turn on the CUDA backend: cmake -B build -DGGML CUDA=ON cmake --build build --config Release -j 8 Binaries land in build/bin/ . Confirm the two you'll use exist: ls build/bin/llama-cli build/bin/llama-bench 2. Download the models Install the Hugging Face CLI https://huggingface.co/docs/huggingface hub/en/guides/cli and pull single files — never clone a GGUF repo, it holds every quantization level: curl -LsSf https://hf.co/cli/install.sh | bash hf download mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF \ Mixtral-8x7B-Instruct-v0.1.Q4 K M.gguf --local-dir models hf download Qwen/Qwen3-14B-GGUF \ Qwen3-14B-Q4 K M.gguf --local-dir models That's 28.4 GB for Mixtral and 9.0 GB for Qwen3-14B https://huggingface.co/Qwen/Qwen3-14B-GGUF , our dense baseline. Qwen3-14B is the fair fight: its 14.8B parameters sit right next to Mixtral's 12.9B active parameters, and both files use the same Q4 K M quantization, so any throughput gap comes from architecture, not quant format. Skip the famous TheBloke Mixtral quants from December 2023 — they predate llama.cpp's 2024 switch to merged expert tensors and the GGUF chat-template metadata. The mradermacher conversion above uses the modern format. 3. Smoke-test Mixtral ./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4 K M.gguf \ -c 8192 -ngl 99 -ngl 99 offloads every layer to the GPU recent builds also accept -ngl auto ; -c 8192 caps the context so the KV cache doesn't inflate memory use. Because the GGUF carries a chat template, llama-cli drops you straight into interactive conversation mode. Ask it something, confirm you get coherent output, then exit with Ctrl+C — it prints token-throughput timings on the way out. Watch the load logs: a line per layer ending in offloaded 33/33 layers to GPU 32 blocks plus the output layer means the whole model is resident on the GPU. For a scripted one-shot run instead of chat, use -no-cnv with Mixtral's INST template: ./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4 K M.gguf \ -c 4096 -ngl 99 -no-cnv -n 128 \ -p " INST Explain mixture-of-experts routing in two sentences. /INST " 4. Benchmark MoE vs. dense llama-bench measures two phases: pp prompt processing, compute-bound and tg token generation, memory-bandwidth-bound . Generation is where MoE shines, because each token only reads the router plus 2 of the 8 experts per layer. ./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4 K M.gguf \ -m models/Qwen3-14B-Q4 K M.gguf -p 512 -n 128 -ngl 99 Each test runs 5 repetitions by default and reports mean ± stddev; add -r 3 if you're impatient. 5. Fit it on a smaller GPU with --n-cpu-moe If Mixtral doesn't fit your VRAM, don't drop whole layers with a lower -ngl . Use --n-cpu-moe N , which keeps the MoE expert weights of the first N layers in system RAM while attention, norms, and the router stay on the GPU. Experts are ~96% of Mixtral's weights but only a quarter of them fire per token, so this costs far less speed than evicting entire layers: ./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4 K M.gguf \ -c 8192 -ngl 99 --n-cpu-moe 12 On a 24 GB card, start around --n-cpu-moe 12 Mixtral has 32 layers and lower it until you hit an out-of-memory error, then back off. llama-bench takes the same flag as -ncmoe and accepts a comma-separated sweep, so you can find the knee of the curve in one command: ./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4 K M.gguf \ -ngl 99 -ncmoe 12,16,24,32 -p 512 -n 128 Verify it works Success looks like a Markdown table from step 4 in this shape these reference numbers are from a 64 GB M3 Max — your absolute values will differ by hardware : | model | size | params | backend | ngl | test | t/s | | -------------------- | --------: | ------: | ---------- | --: | ------ | -------------: | | llama 8x7B Q4 K M | 26.49 GiB | 46.70 B | Metal,BLAS | 99 | pp 512 | 235.11 ± 1.82 | | llama 8x7B Q4 K M | 26.49 GiB | 46.70 B | Metal,BLAS | 99 | tg 128 | 32.40 ± 0.21 | | qwen3 14B Q4 K M | 8.38 GiB | 14.77 B | Metal,BLAS | 99 | pp 512 | 341.27 ± 2.05 | | qwen3 14B Q4 K M | 8.38 GiB | 14.77 B | Metal,BLAS | 99 | tg 128 | 35.87 ± 0.33 | The result that matters is the ratio, not the absolute numbers: Mixtral's tg 128 should land within roughly 25% of the dense 14B's, despite carrying 3.2× the parameters. That's expert routing working — per token it reads about the same number of bytes as a 13B dense model. For contrast, a dense 46.7B model at Q4 K M would stream all 26.49 GiB through memory for every token and generate roughly 3× slower. The MoE bill arrives in the size column instead: 26.49 GiB resident versus 8.38 GiB. Troubleshooting ggml backend cuda buffer type alloc buffer: allocating 26550.00 MiB on device 0: cudaMalloc failed: out of memory — the full Q4 K M doesn't fit your card. Add --n-cpu-moe 12 raise N until it loads rather than cutting -ngl , or use the Q3 K M file. CMake Error at ggml/src/ggml-cuda/CMakeLists.txt message : CUDA Toolkit not found — CMake can't see nvcc . Install CUDA Toolkit 12.x, then reconfigure with the compiler pinned: CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build -DGGML CUDA=ON . Mac loads the model but generation crawls, with a log warning that allocated size exceeds recommendedMaxWorkingSetSize — macOS caps GPU-wired memory at roughly 70% of unified RAM, so a 26.49 GiB model thrashes on 32/36 GB machines. Raise the cap resets on reboot with sudo sysctl iogpu.wired limit mb=30720 , or switch to Q3 K M. hf: command not found — the installer put the binary in ~/.local/bin , which isn't on your PATH. Run export PATH="$HOME/.local/bin:$PATH" add it to your shell rc , or pip install -U huggingface hub to get the same CLI via pip. Next steps Swap llama-cli for llama-server and you get an OpenAI-compatible API on localhost:8080 with the same flags, including --n-cpu-moe . Run llama.cpp's llama-perplexity tool over both models to see the quality side of the trade — speed parity means little if the 47B MoE doesn't beat the dense 14B on your workload. Then point the same benchmark harness at newer MoE models like Qwen3-30B-A3B or OpenAI's gpt-oss-20b, where sparser routing 3B–4B active makes the effect even more dramatic. When you outgrow static Q4 K M files, look into imatrix-calibrated quants, which measurably cut quantization loss at the same size. Sources & further reading - llama.cpp build documentation https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md — github.com - llama-bench README https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md — github.com - llama-cli README https://github.com/ggml-org/llama.cpp/blob/master/tools/cli/README.md — github.com - Mixtral-8x7B-Instruct-v0.1-GGUF model card https://huggingface.co/mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF — huggingface.co - Qwen3-14B-GGUF model card https://huggingface.co/Qwen/Qwen3-14B-GGUF — huggingface.co - Hugging Face CLI guide https://huggingface.co/docs/huggingface hub/en/guides/cli — huggingface.co Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.