{"slug": "qwen-3-6-27b-and-35b-mtp-vs-standard-on-16gb-gpu", "title": "Qwen 3.6 27B and 35B MTP vs Standard on 16GB GPU", "summary": "The article summarizes tests of Multi-Token Prediction (MTP) on Qwen 3.6 27B and 35B models using a 16GB RTX 4080 GPU. For the 27B model, MTP at a draft depth of 2 provided a 67% speed increase (75 t/s) but halved the usable context window to 40K, while a draft depth of 1 offered a balanced 39% speed gain (57 t/s) with a 70K context. The 35B MoE model showed speed improvements but was impractical on 16GB VRAM due to severely limited context windows, with even the lowest MTP setting yielding only 15K average context.", "body_md": "I tested Speculative decoding (Multi-Token Prediction, MTP) performance in Qwen 3.6 27B and 35B on an RTX 4080 with 16 GB VRAM.\nFor a broader view of token speeds and VRAM trade-offs across more models on the same hardware, see 16 GB VRAM LLM benchmarks with llama.cpp.\nMulti-Token Prediction is a form of speculative decoding built directly into certain model checkpoints. Instead of predicting one token per forward pass, the model carries extra \"MTP heads\" that propose several future tokens in a single step — then verifies them in parallel. If the guesses are accepted, the effective throughput rises without changing the output quality.\nThe Qwen 3.6 family ships both standard GGUF files and MTP-enabled variants. In llama.cpp, MTP is activated through:\n--spec-type draft-mtp --spec-draft-n-max 3\n--spec-draft-n-max\nis the key tuning knob. It sets how many speculative tokens the MTP head proposes at each step. Higher values give a potential speed boost but cost extra VRAM for the draft buffers — a real constraint on 16 GB cards.\nI tested how the two Qwen 3.6 models behave with MTP enabled versus standard decoding on a GPU with 16 GB VRAM (RTX 4080).\nTo fit model weights and KV cache into VRAM I used heavily quantised variants:\nQwen3.6-27B-UD-IQ3_XXS\nand Qwen3.6-27B-UD-IQ3_XXS-MTP\nQwen3.6-35B-A3B-UD-IQ3_S\nand Qwen3.6-35B-A3B-UD-IQ3_S-MTP\nTwo context budgets are tracked per run:\nA key reason for keeping the average context at a practical target is that Hermes Agent — which I use as the primary AI assistant connecting to llama.cpp on this machine — requires at least 64 K context by default and will reject models with a smaller window at startup. Models below that threshold cannot maintain enough working memory for multi-step tool-calling workflows. For llama.cpp this means passing --ctx-size 65536\nor larger. Any MTP configuration that compresses the average usable context significantly below 64 K is therefore unsuitable for daily Hermes workloads, which is why the Avg Ctx numbers in the tables below are the most decision-relevant ones.\nBoth KV cache quantisation levels were tested: q8 (higher quality, more VRAM) and q5 (lower VRAM, longer context). Be aware that moving from q8 to q5 KV cache can cause a noticeable quality drop — in my testing the degradation was significant enough to make q5 unsuitable for my workloads. The speed and context numbers for q5 are included for completeness, but you should test response quality on your own tasks before committing to it.\nWith q8 KV cache, MTP at --spec-draft-n-max 2\ndelivers ~67 % faster generation (75 vs 45 t/s) at the cost of halving the average context window from 80 K to 40 K. Prompt ingestion speed drops from 200 to ~150 t/s because MTP requires device-to-host transfers during the prefill phase.\nSwitching to q5 KV cache recovers meaningful context: --spec-draft-n-max 1\ngives 70 K average context at 57 t/s — a 39 % generation speedup over standard decoding while still keeping the context window at a useful size. At --spec-draft-n-max 3\nthe context drops to 60 K but generation reaches 67 t/s (+63 %).\nMTP is genuinely useful for the 27B dense model. The sweet spot on 16 GB VRAM is:\n--spec-draft-n-max 2\n— best raw speed (75 t/s), context down to 40–60 K--spec-draft-n-max 1\n— best speed-vs-context balance (57 t/s, 70 K avg context)The 35B model is a Mixture-of-Experts (MoE) architecture (35B-A3B\nmeans 35B total parameters, ~3B active per token). MoE models usually benefit more from MTP because the sparse routing keeps the MTP head computationally cheap relative to a full forward pass.\nThe MoE architecture delivers impressive raw generation speed with MTP (+27 % at max 1, +29 % at max 2 vs standard 146 t/s). But the practical problem is the average context. With q8 KV cache, even --spec-draft-n-max 1\nonly gives 15 K average context — barely enough for modest tasks. Higher draft depths have no viable average context at all on a 16 GB card.\nThis is the central VRAM cost question for MTP on consumer hardware: the extra draft buffers eat into the remaining VRAM budget directly, and the 35B-A3B model with q8 KV cache leaves very little headroom.\nq5 KV cache only marginally improves the average context story. --spec-draft-n-max 1\ngives 10 K average context at 151 t/s. Standard decoding at q5 gives 122 t/s with 120 K average context.\nOn a 16 GB GPU the 35B MoE model with MTP faces a hard wall: the average usable context collapses to 10–15 K tokens, making it impractical for real workloads. Standard decoding at 122–146 t/s with 80–120 K context is significantly more useful.\nIf you have 24 GB+ VRAM, the 35B + MTP combination becomes much more attractive — the context window issue disappears and you keep the speed benefit.\n--spec-draft-n-max\nValue\nThe question of how many speculative tokens to propose per step (--spec-draft-n-max\n) does not have a single right answer — it depends on both model architecture and available VRAM:\n--spec-draft-n-max 2\nwith q8 KV is the fastest, --spec-draft-n-max 1\nwith q5 KV is the most context-friendly.--spec-draft-n-max 1\nis the only option that keeps any usable context, and even then only marginally.3\n, 4\n) increase VRAM pressure without proportional speed gains — at max 4 you're spending roughly the same extra VRAM as max 2 but gen speed doesn't keep pace.Make sure you use an MTP-enabled GGUF (the filename contains MTP\n). If you are new to llama.cpp flags, the llama.cpp Quickstart with CLI and Server covers all the fundamentals. Then launch llama-server or llama-cli with:\nllama-server \\\n--model Qwen3.6-27B-UD-IQ3_XXS-MTP.gguf \\\n--ctx-size 40000 \\\n-ngl 99 --flash-attn on \\\n--cache-type-k q8_0 --cache-type-v q8_0 \\\n--spec-type draft-mtp \\\n--spec-draft-n-max 2\nFor q5 KV cache, replace q8_0\nwith q5_1\nor q5_0\nand adjust --ctx-size\nupward:\nllama-server \\\n--model Qwen3.6-27B-UD-IQ3_XXS-MTP.gguf \\\n--ctx-size 80000 \\\n-ngl 99 --flash-attn on \\\n--cache-type-k q5_1 --cache-type-v q5_1 \\\n--spec-type draft-mtp \\\n--spec-draft-n-max 1\nMTP is activated automatically once llama.cpp sees the MTP heads in the GGUF file and --spec-type draft-mtp\nis set.\nSo standard Qwen3.6-27B-UD-IQ3_XXS.gguf will not work in MTP mode, you will need Qwen3.6-27B-UD-IQ3_XXS-MTP.gguf.\nBut the Qwen3.6-27B-UD-IQ3_XXS-MTP.gguf can work in both Speculative decoding mode and autoregressive one.\nOn a 16 GB GPU (RTX 4080), with these quants, llama.cpp's MTP is a clear win for Qwen 3.6 27B and a net negative for Qwen 3.6 35B in practical use:\nQwen 3.6 27B (IQ3_XXS) — MTP is worthwhile:\n--spec-draft-n-max 2\nQwen 3.6 35B (IQ3_S) — MTP is not practical at 16 GB:\nOn paper, q5 KV cache is the obvious answer for maximising context window while keeping MTP speed gains — but in practice the quality drop moving from q8 to q5 can be significant. Test q5 on your own tasks before adopting it; for my workloads the degradation was unacceptable, and q8 with a tighter context budget remains the better trade-off.\nFor the wider picture of LLM serving options and infrastructure trade-offs, see the LLM Hosting in 2026 pillar and LLM Performance in 2026. If you are tuning Qwen 3.6 sampler settings alongside MTP, the Agentic LLM Inference Parameters Reference for Qwen 3.6 and Gemma 4 is a useful companion.", "url": "https://wpnews.pro/news/qwen-3-6-27b-and-35b-mtp-vs-standard-on-16gb-gpu", "canonical_source": "https://dev.to/rosgluk/qwen-36-27b-and-35b-mtp-vs-standard-on-16gb-gpu-42jd", "published_at": "2026-05-24 00:31:07+00:00", "updated_at": "2026-05-24 01:03:14.076891+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence", "open-source", "hardware"], "entities": ["Qwen", "RTX 4080", "llama.cpp", "Hermes Agent", "Qwen 3.6"], "alternates": {"html": "https://wpnews.pro/news/qwen-3-6-27b-and-35b-mtp-vs-standard-on-16gb-gpu", "markdown": "https://wpnews.pro/news/qwen-3-6-27b-and-35b-mtp-vs-standard-on-16gb-gpu.md", "text": "https://wpnews.pro/news/qwen-3-6-27b-and-35b-mtp-vs-standard-on-16gb-gpu.txt", "jsonld": "https://wpnews.pro/news/qwen-3-6-27b-and-35b-mtp-vs-standard-on-16gb-gpu.jsonld"}}