llama.cpp 0.7.0: MoE Cache Hits 2.2x — Server Port Has Changed

Llama.cpp 0.7.0, released October 15, adds a GPU expert cache that lifts MoE decode throughput up to 2.2x, changes the default llama-server port from 8080 to 9931, and introduces LiliCorr speculative decoding from NVIDIA's Nemotron research team. The new --moe-cache-mib flag keeps hot experts resident in GPU memory via an LRU cache, pushing Qwen3-30B-A3B decode from 30.8 to 67.8 tokens per second on an RTX 5090 and from 22.6 to 43.2 t/s on a 16 GB Tesla T4. The port change will silently break reverse-proxy and hardcoded-8080 deployments, while LiliCorr's NVIDIA paper reports 7–19% higher acceptance length at roughly 3% additional latency per block.

llama.cpp 0.7.0 landed on October 15 with three changes that matter for anyone running MoE models locally: a GPU expert cache that pushes decode throughput up to 2.2x, a new NVIDIA-derived speculative decoding technique, and a default server port change from 8080 to 9931 that will silently break your setup if you don’t act. The MoE cache alone makes this a mandatory upgrade for anyone running DeepSeek, Qwen3, or Meta Muse models that overflow their VRAM. The MoE Cache: What Changed and Why It’s Fast Mixture-of-Experts models route each token through only a small subset of their total parameters. That’s why a 30B-active-parameter model like Qwen3-30B-A3B can run at reasonable speed despite having 110B total weights — most of those weights aren’t touched per inference step. The problem arrives when the model exceeds your VRAM: llama.cpp offloads the unused experts to CPU RAM, and every token requires a fresh host-to-GPU transfer for each expert it needs. At scale, that back-and-forth kills throughput. The new --moe-cache-mib flag fixes this with an LRU cache. Hot experts — the ones your model hits repeatedly — stay resident in GPU memory. Only cache misses trigger uploads. For a 30B MoE model on an RTX 5090 paired with --cpu-moe , this pushed decode from 30.8 to 67.8 tokens per second — a 2.2x jump. On a free Tesla T4 16 GB , the same Qwen3-30B-A3B went from 22.6 to 43.2 t/s. Even an RTX 3060 running Gemma-4-26B-Q4 gained 29% with a 7,200 MiB cache. Allocate 8 GiB of GPU memory for the expert cache llama-server -m qwen3-30b-a3b-q4 k m.gguf --moe-cache-mib 8192 Set --moe-cache-mib as high as your available VRAM allows after the model’s non-expert layers are loaded. Multi-GPU setups are supported — the cache spans all devices. The community benchmarks from DEV.to https://dev.to/sampop3/llamacpps-moe-cache-nearly-doubled-30b-decode-speed-on-a-free-t4-15np show consistent gains across a range of hardware, including a near-doubling on a free Colab T4. Breaking: The Default Server Port Is Now 9931 If you run llama-server behind a reverse proxy or have any application hardcoded to port 8080, this upgrade will break your setup without warning. The new default port is 9931 — leetspeak for “GGML” https://github.com/ggml-org/llama.cpp/pull/26508 , which is as good a reason as any. Bare-metal deployments with nginx, Caddy, or Traefik proxying to localhost:8080 are directly affected. Docker users whose container-to-host port mapping was already explicit are not. Keep the old port behavior llama-server -m model.gguf --port 8080 Or update your reverse proxy upstream to port 9931. Either way, verify before you upgrade in any environment where downtime matters. LiliCorr Speculative Decoding llama.cpp 0.7.0 also adds LiliCorr, a speculative decoding improvement from NVIDIA’s Nemotron research team. The core problem with DFlash — llama.cpp’s diffusion-style drafter — is that it generates draft tokens independently per position. Tokens look plausible in isolation but often form incoherent sequences, dragging down acceptance rates. LiliCorr addresses this by running the entire candidate lattice through a lightweight two-layer Transformer in one forward pass. Self-attention spans all candidates across all positions, scoring joint coherence rather than per-position likelihood. The NVIDIA paper https://arxiv.org/abs/2608.20530 reports 7–19% higher acceptance length with only about 3% additional latency per block. In llama.cpp 0.7.0, it works best with GLM5-Next via the MTP speculative decoding path. Model Manager and New Models The web UI ships a long-overdue model manager that auto-discovers GGUF files in your configured directory, loads models on demand when first requested, and uses LRU eviction to keep up to four models resident simultaneously. Switching between models without restarting the server is now practical. New model support includes K2 Horizon dense and MoVA variants , Prism Bonsai 2 27B, EmbeddingGemma-2 text, vision, and audio , LiquidAI d1/d1-omni decision models, pplx-decider, MiniCPM-V 4.7, and Nemotron Super 3.5 VL. ggml is updated to v0.27.0. Upgrade Now If you’re running any MoE model that partially overflows your VRAM — and most popular open models in 2026 are MoE — the speedup is real and takes one flag to unlock. Check the full release notes https://github.com/ggml-org/llama.cpp/releases/tag/v0.7.0 for the complete changelog before upgrading in production. macOS via Homebrew brew upgrade llama.cpp From source with CUDA git pull cmake -B build -DGGML CUDA=on cmake --build build --config Release