Compile llama.cpp, run quantized Mixtral 8x7B, and measure the mixture-of-experts speed-versus-memory trade-off yourself.
What you'll build / learn #
You'll compile llama.cpp from source, run a 4-bit quantized Mixtral 8x7B locally, and use llama-bench
to prove the mixture-of-experts (MoE) trade-off with your own numbers: generation speed tracks the ~12.9B active parameters per token, while memory cost tracks all 46.7B.
Prerequisites #
Verified against llama.cpp build b10261 (August 4, 2026) and the current Hugging Face hf
CLI. Commands are for macOS/Linux; Windows users should work inside WSL2.
Disk:~40 GB free (both models).** Memory:**the Q4_K_M Mixtral weighs 26.49 GiB. You need one of: 32 GB+ system RAM (CPU-only), a GPU with 32 GB+ VRAM for full offload, or a 24 GB GPU plus ~16 GB RAM using the MoE CPU-offload flag covered in step 5. On Apple Silicon, 48 GB unified memory is comfortable; on 32/36 GB machines use the Q3_K_M file (21.0 GiB) instead.Toolchain:git
, CMake ≥ 3.14, and a C++17 compiler (Xcode Command Line Tools on macOS,build-essential
on Debian/Ubuntu).NVIDIA only:CUDA Toolkit12.x installed, withnvcc
on your PATH.
llama.cpp ships new tagged builds daily and flags do change — everything below is checked against b10261.
1. Build llama.cpp #
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
On macOS, Metal GPU support is enabled by default, so a plain build is all you need:
cmake -B build
cmake --build build --config Release -j 8
On Linux with an NVIDIA GPU, turn on the CUDA backend:
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8
Binaries land in build/bin/
. Confirm the two you'll use exist:
ls build/bin/llama-cli build/bin/llama-bench
2. Download the models #
Install the Hugging Face CLI and pull single files — never clone a GGUF repo, it holds every quantization level:
curl -LsSf https://hf.co/cli/install.sh | bash
hf download mradermacher/Mixtral-8x7B-Instruct-v0.1-GGUF \
Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf --local-dir models
hf download Qwen/Qwen3-14B-GGUF \
Qwen3-14B-Q4_K_M.gguf --local-dir models
That's 28.4 GB for Mixtral and 9.0 GB for Qwen3-14B, our dense baseline. Qwen3-14B is the fair fight: its 14.8B parameters sit right next to Mixtral's 12.9B active parameters, and both files use the same Q4_K_M quantization, so any throughput gap comes from architecture, not quant format.
Skip the famous TheBloke Mixtral quants from December 2023 — they predate llama.cpp's 2024 switch to merged expert tensors and the GGUF chat-template metadata. The mradermacher conversion above uses the modern format.
3. Smoke-test Mixtral #
./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
-c 8192 -ngl 99
-ngl 99
offloads every layer to the GPU (recent builds also accept -ngl auto
); -c 8192
caps the context so the KV cache doesn't inflate memory use. Because the GGUF carries a chat template, llama-cli
drops you straight into interactive conversation mode. Ask it something, confirm you get coherent output, then exit with Ctrl+C
— it prints token-throughput timings on the way out. Watch the load logs: a line per layer ending in offloaded 33/33 layers to GPU
(32 blocks plus the output layer) means the whole model is resident on the GPU.
For a scripted one-shot run instead of chat, use -no-cnv
with Mixtral's [INST]
template:
./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
-c 4096 -ngl 99 -no-cnv -n 128 \
-p "[INST] Explain mixture-of-experts routing in two sentences. [/INST]"
4. Benchmark MoE vs. dense #
llama-bench
measures two phases: pp
(prompt processing, compute-bound) and tg
(token generation, memory-bandwidth-bound). Generation is where MoE shines, because each token only reads the router plus 2 of the 8 experts per layer.
./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
-m models/Qwen3-14B-Q4_K_M.gguf -p 512 -n 128 -ngl 99
Each test runs 5 repetitions by default and reports mean ± stddev; add -r 3
if you're impatient.
5. Fit it on a smaller GPU with --n-cpu-moe #
If Mixtral doesn't fit your VRAM, don't drop whole layers with a lower -ngl
. Use --n-cpu-moe N
, which keeps the MoE expert weights of the first N layers in system RAM while attention, norms, and the router stay on the GPU. Experts are ~96% of Mixtral's weights but only a quarter of them fire per token, so this costs far less speed than evicting entire layers:
./build/bin/llama-cli -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
-c 8192 -ngl 99 --n-cpu-moe 12
On a 24 GB card, start around --n-cpu-moe 12
(Mixtral has 32 layers) and lower it until you hit an out-of-memory error, then back off. llama-bench
takes the same flag as -ncmoe
and accepts a comma-separated sweep, so you can find the knee of the curve in one command:
./build/bin/llama-bench -m models/Mixtral-8x7B-Instruct-v0.1.Q4_K_M.gguf \
-ngl 99 -ncmoe 12,16,24,32 -p 512 -n 128
Verify it works #
Success looks like a Markdown table from step 4 in this shape (these reference numbers are from a 64 GB M3 Max — your absolute values will differ by hardware):
| model | size | params | backend | ngl | test | t/s |
| -------------------- | --------: | ------: | ---------- | --: | ------ | -------------: |
| llama 8x7B Q4_K_M | 26.49 GiB | 46.70 B | Metal,BLAS | 99 | pp 512 | 235.11 ± 1.82 |
| llama 8x7B Q4_K_M | 26.49 GiB | 46.70 B | Metal,BLAS | 99 | tg 128 | 32.40 ± 0.21 |
| qwen3 14B Q4_K_M | 8.38 GiB | 14.77 B | Metal,BLAS | 99 | pp 512 | 341.27 ± 2.05 |
| qwen3 14B Q4_K_M | 8.38 GiB | 14.77 B | Metal,BLAS | 99 | tg 128 | 35.87 ± 0.33 |
The result that matters is the ratio, not the absolute numbers: Mixtral's tg 128
should land within roughly 25% of the dense 14B's, despite carrying 3.2× the parameters. That's expert routing working — per token it reads about the same number of bytes as a 13B dense model. For contrast, a dense 46.7B model at Q4_K_M would stream all 26.49 GiB through memory for every token and generate roughly 3× slower. The MoE bill arrives in the size
column instead: 26.49 GiB resident versus 8.38 GiB.
Troubleshooting #
** ggml_backend_cuda_buffer_type_alloc_buffer: allocating 26550.00 MiB on device 0: cudaMalloc failed: out of memory** — the full Q4_K_M doesn't fit your card. Add
--n-cpu-moe 12
(raise N until it loads) rather than cutting -ngl
, or use the Q3_K_M file.** CMake Error at ggml/src/ggml-cuda/CMakeLists.txt (message): CUDA Toolkit not found** — CMake can't see
nvcc
. Install CUDA Toolkit 12.x, then reconfigure with the compiler pinned: CUDACXX=/usr/local/cuda/bin/nvcc cmake -B build -DGGML_CUDA=ON
.Mac loads the model but generation crawls, with a log warning that allocated size exceeds recommendedMaxWorkingSetSize — macOS caps GPU-wired memory at roughly 70% of unified RAM, so a 26.49 GiB model thrashes on 32/36 GB machines. Raise the cap (resets on reboot) with
sudo sysctl iogpu.wired_limit_mb=30720
, or switch to Q3_K_M.** hf: command not found** — the installer put the binary in
~/.local/bin
, which isn't on your PATH. Run export PATH="$HOME/.local/bin:$PATH"
(add it to your shell rc), or pip install -U huggingface_hub
to get the same CLI via pip.## Next steps
Swap llama-cli
for llama-server
and you get an OpenAI-compatible API on localhost:8080
with the same flags, including --n-cpu-moe
. Run llama.cpp's llama-perplexity
tool over both models to see the quality side of the trade — speed parity means little if the 47B MoE doesn't beat the dense 14B on your workload. Then point the same benchmark harness at newer MoE models like Qwen3-30B-A3B or OpenAI's gpt-oss-20b, where sparser routing (3B–4B active) makes the effect even more dramatic. When you outgrow static Q4_K_M files, look into imatrix-calibrated quants, which measurably cut quantization loss at the same size.
Sources & further reading #
llama.cpp build documentation— github.com - llama-bench README— github.com - llama-cli README— github.com - Mixtral-8x7B-Instruct-v0.1-GGUF model card— huggingface.co - Qwen3-14B-GGUF model card— huggingface.co - Hugging Face CLI guide— huggingface.co
Mariana Souza· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.