{"slug": "i-patched-llama-cpp-to-gain-20-prompt-processing-tps-help-me-make-a-pr", "title": "I patched llama.cpp to gain 20% prompt processing TPS. Help me make a PR", "summary": "A developer patched llama.cpp to improve prompt processing throughput by 20% when using Multi-Token Prediction (MTP) on GPU, recovering performance lost to MTP overhead. The proof-of-concept processes only the output row instead of the entire ubatch for the last layer MoE FFN, restoring prompt processing TPS to non-MTP levels while retaining most MTP benefits for token generation. The developer seeks collaborators to open a pull request, as the code is AI-generated and cannot be submitted directly due to llama.cpp's contribution policy.", "body_md": "I've been running Qwen3.6-35B-A3B locally on llama.cpp and noticed that prompt processing throughput gets too low with MTP. I got nerd-sniped.\n\nWhat started as curiosity turned into a two-week rabbit hole of experiments and ended with a PoC that fully recovers the MTP PP overhead on GPU, above any expectation I had.\n\nTL;DR: instead of processing the last layer MoE FFN for the entire ubatch tokens (usually 512-2048 tokens), this PoC processes only the output row (usually 1 token during prefill). The result is PP TPS is back to the same as with MTP disabled, keeping most of MTP's benefits to TG TPS, even with a slight drop in draft acceptance rate in one of the benchs.\n\nI'm not opening a PR to llama.cpp because this is AI-generated code, which goes against their contribution policy, which I support. If you know C++ and llama.cpp internals, I invite to work together with me to open a PR with a more mature implementation.\n\nComments URL: [https://news.ycombinator.com/item?id=48700782](https://news.ycombinator.com/item?id=48700782)\n\nPoints: 2\n\n# Comments: 1", "url": "https://wpnews.pro/news/i-patched-llama-cpp-to-gain-20-prompt-processing-tps-help-me-make-a-pr", "canonical_source": "https://news.ycombinator.com/item?id=48700782", "published_at": "2026-06-27 19:03:20+00:00", "updated_at": "2026-06-27 19:34:13.392788+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["llama.cpp", "Qwen3.6-35B-A3B", "GPU", "MoE"], "alternates": {"html": "https://wpnews.pro/news/i-patched-llama-cpp-to-gain-20-prompt-processing-tps-help-me-make-a-pr", "markdown": "https://wpnews.pro/news/i-patched-llama-cpp-to-gain-20-prompt-processing-tps-help-me-make-a-pr.md", "text": "https://wpnews.pro/news/i-patched-llama-cpp-to-gain-20-prompt-processing-tps-help-me-make-a-pr.txt", "jsonld": "https://wpnews.pro/news/i-patched-llama-cpp-to-gain-20-prompt-processing-tps-help-me-make-a-pr.jsonld"}}