A free 42x speedup for llama.cpp reveals the real 2026 AI cost lever

An open-source contributor's r/LocalLLaMA post, "42x Faster Prompt Lookup Drafting in llama.cpp," reports that llama.cpp's ngram-mod speculative-decoding path can draft repeated text up to 42 times faster on repetition-heavy workloads such as code edits and JSON output, with no second draft model and no extra VRAM. A widely shared HackMD benchmark of every speculative-decode mode llama.cpp ships, run on Qwen3.6-35B-A3B on an RTX 3090 with maintainers' recommended settings, found none of the modes beat the baseline on that setup, and a separate writeup by Assen Tarlov measured a nearly 1.5x speedup from a tuned Gemma 4 draft pairing. The gains are concentrated in tasks where the model repeats itself, while fresh prose generation has nothing to look up and therefore nothing to draft.

An open-source contributor found a way to make llama.cpp draft repeated text up to 42 times faster on certain workloads, no new hardware and no new model required. Inference speed, not bigger clusters, is where a lot of the real savings are hiding this year. The technique is called prompt lookup decoding, and it doesn't touch the model itself. It watches the conversation as it happens: it builds a lookup table of token sequences it has already seen in the prompt and the output. When the model is about to regenerate something it has already produced once, it drafts a whole chunk of tokens at once instead of grinding through them one at a time. Verify the draft in a single forward pass, and if it's right, you just got a dozen or more tokens for the price of one. A post titled "42x Faster Prompt Lookup Drafting in llama.cpp" went up on r/LocalLLaMA. It landed in front of the self-hosted crowd: the kind of people running quantized 30B and 70B models on a single RTX card instead of renting a GPU cluster. The llama.cpp docs describe ngram-mod as a lightweight speculative-decoding path that computes rolling n-gram hashes, stores the next token for each hash, and can draft variable-length runs with sample settings up to 64 tokens. That detail matters more than it sounds. Code edits, JSON output, repeated boilerplate, anything where the model is regenerating something close to what it already wrote, is exactly the situation where a lookup table of recent n-grams pays off hardest. You don't need a second draft model for this to work, which is the part that makes it interesting. Classic speculative decoding pairs a big model with a small, fast one that guesses ahead and lets the big model check its work. That's effective - Assen Tarlov's writeup on Gemma 4 speculative decoding found a nearly 1.5x speedup from a well-tuned draft pairing - but it means loading, running, and maintaining a second model. Prompt lookup decoding skips that step entirely. It's just a hash map from n-gram to next token, built and updated on the fly from context you already have loaded. No extra weights, no extra VRAM for a second model, no extra complexity in your serving stack. Here's the honest part. A 42x speedup on a narrow, repetition-heavy task is not the same as a 42x speedup on your average chatbot session, and anyone who runs the numbers themselves will find that out fast. A widely shared HackMD benchmark tested every speculative-decode mode llama.cpp ships, including ngram-mod, on Qwen3.6-35B-A3B running on an RTX 3090, using the maintainers' own recommended settings. The verdict: none of them beat the baseline on that setup. The gains prompt lookup decoding delivers are real, but they're concentrated almost entirely in tasks where the model repeats itself: code edits, structured JSON, diffs, refactors. Ask it to write a fresh paragraph of prose and there's nothing to look up, so there's nothing to draft. An Optane home server makes trillion parameter AI feel almost practical https://startupfortune.com/an-optane-home-server-makes-trillion-parameter-ai-feel-almost-practical/ A r/LocalLLaMA builder used secondhand Intel Optane Persistent Memory to run Kimi K2.5, a 1 trillion parameter model, at more than 4 tokens per second. The setup is not a cloud GPU replacement, but it shows how old enterprise memory could change the economics of local AI experimentation. - how to run trillion parameter models locally cheaply https://startupfortune.com/an-optane-home-server-makes-trillion-parameter-ai-feel-almost-practical/ - Intel Optane persistent memory for AI inference costs https://startupfortune.com/an-optane-home-server-makes-trillion-parameter-ai-feel-almost-practical/ That's not a knock on the technique. It's the actual shape of the 2026 inference economy. Frontier labs are still racing to build bigger clusters and buy more Blackwell chips, and that race gets most of the headlines. But for anyone actually running a product on top of an LLM, the cheaper lever has always been the decoding loop itself, not how many GPUs you can rent. That's true whether it's a self-hosted coding assistant or a startup burning through API credits: how many tokens you can generate per forward pass is what counts. A free, model-agnostic technique that cuts GPU time on exactly the workloads becoming the backbone of AI coding tools and agents, code generation and structured output, is a bigger deal for margins than another point of benchmark accuracy on a frontier model nobody self-hosts anyway. This is also why the open-source stack keeps closing the gap with the way expensive frontier APIs get used. Take an AI coding agent that spends most of its inference budget rewriting files it already has open: that's about as repetitive and structured as output gets. A technique like this can cut real dollars off a GPU bill without touching model quality at all. It costs nothing to try and nothing to adopt. You don't retrain anything, you don't fine-tune anything, you just flip on the lookup drafting flags in llama.cpp and measure your own workload. That's the actual lesson from a Reddit thread and a llama.cpp optimization most people will never read: the gains from here aren't going to come from waiting for the next trillion-parameter model. They're going to come from people who understand exactly how their decoding loop spends its time, and who go looking for the free 40x sitting in code nobody's optimized yet. Also read: MiniMax quietly ships a coding-only model https://startupfortune.com/minimax-quietly-ships-a-coding-only-model-as-chinas-ai-models-flood-the-market/ , as China's AI models flood the market • Why Developers Are Running AI Models on Mac Minis Instead of Nvidia GPUs https://startupfortune.com/why-developers-are-running-ai-models-on-mac-minis-instead-of-nvidia-gpus/ • Tesla Finally Puts the Semi Into Volume Production, Seven Years Late https://startupfortune.com/tesla-finally-puts-the-semi-into-volume-production-seven-years-late/ This article is posted in Entrepreneurship News https://startupfortune.com/category/entrepreneurship/ , check it out for more related stories. Join the discussion Open in the community → https://startupfortune.com/community/ Almost there. Sign in and your reply posts straight away. Local AI is becoming a founder infrastructure story https://startupfortune.com/local-ai-is-becoming-a-founder-infrastructure-story/ GGUF uploads on Hugging Face have reportedly nearly doubled in two months, showing rising interest in local AI inference. For startups, the trend points to lower API dependence, stronger privacy options, and a new layer of infrastructure decisions around small models and quantization. - local AI infrastructure for startup founders https://startupfortune.com/local-ai-is-becoming-a-founder-infrastructure-story/ - GGUF format adoption in startup development https://startupfortune.com/local-ai-is-becoming-a-founder-infrastructure-story/