{"slug": "llms-don-t-have-to-generate-one-token-at-a-time-how-medusa-and-multi-token-cheat", "title": "LLMs Don't Have to Generate One Token at a Time: How Medusa and Multi-Token Prediction Cheat Autoregression", "summary": "Shrijith Venkatramana, developer of LiveReview, explains how autoregressive LLM decoding can be accelerated using speculative decoding and Medusa-style multi-token prediction, which break the sequential token-by-token bottleneck by having a small draft model propose multiple tokens that a large model verifies in parallel.", "body_md": "*Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nA modern LLM can contain hundreds of billions of parameters, run on extremely expensive accelerators, and still spend most of its inference time doing something that looks embarrassingly sequential:\n\n``` php\ntoken 1 -> token 2 -> token 3 -> token 4 -> token 5 -> ...\n```\n\nThat is the awkward part of autoregressive generation.\n\nThe model may process a whole prompt in parallel during the initial prefill, but once generation starts, the next token depends on the previous token. So generating 100 tokens looks conceptually like running the model 100 times.\n\nAnd for many serving workloads, that is exactly where the money goes.\n\nA family of techniques tries to break this bottleneck by asking a deceptively simple question:\n\nWhat if the model could predict several future tokens at once, then verify them in parallel?\n\nThat idea leads to speculative decoding, Medusa-style multiple decoding heads, and the broader multi-token prediction approach used during training.\n\nThe interesting part is that these are not merely \"optimization tricks.\" They change the computational structure of decoding.\n\nThis article develops that idea from first principles and then gets into the engineering details.\n\nConsider ordinary autoregressive decoding.\n\nGiven a prompt:\n\n```\nThe capital of France is\n```\n\nthe model predicts:\n\n```\nParis\n```\n\nThen it feeds the new sequence back through the model:\n\n```\nThe capital of France is Paris\n```\n\nand predicts the next token.\n\nThen again:\n\n```\nThe capital of France is Paris .\n```\n\nand so on.\n\nFormally, the model factorizes the probability of a sequence as:\n\n```\nP(x1, x2, ..., xT)\n    = product over t of P(xt | x1, ..., x(t-1))\n```\n\nThat conditional dependence is what makes language modeling so useful.\n\nIt is also what makes decoding annoying.\n\nThis distinction matters enormously in production.\n\nSuppose a prompt has 2,000 tokens and we want 200 generated tokens.\n\nDuring prefill, the transformer can process many positions concurrently:\n\n```\nprompt tokens\n     |\n     v\nparallel transformer computation\n     |\n     v\nKV cache\n```\n\nDuring decoding:\n\n```\ntoken 2001\n    |\n    v\nrun model\n    |\n    v\ntoken 2002\n    |\n    v\nrun model\n    |\n    v\ntoken 2003\n    |\n    v\n...\n```\n\nThe transformer itself is highly parallelizable.\n\nThe *dependency graph* of generation is not.\n\nThis creates an unusual hardware situation. A giant model can spend much of its decoding time limited not by arithmetic throughput but by repeatedly moving a large set of model weights through memory.\n\nThis was one of the motivations behind the speculative decoding work by Yaniv Leviathan, Matan Kalman, and Yossi Matias, and later became a central observation in Medusa.\n\nThe core question is therefore not:\n\nHow do we make one forward pass cheaper?\n\nIt is:\n\nHow do we get more than one accepted token out of each expensive forward pass?\n\nThat is a much more interesting question.\n\nImagine that instead of asking the large model for one token, we had a tiny model that could cheaply guess several:\n\n```\nBig model:\n\"I need to generate the next token.\"\n\nSmall model:\n\"Here are my guesses:\n\n  the -> capital -> of -> France\n\"\n```\n\nThe big model can then evaluate those candidate tokens in parallel.\n\nSuppose the small model proposes:\n\n```\nthe capital of France\n```\n\nThe large model might agree with all four:\n\n```\nthe   ✓\ncapital ✓\nof    ✓\nFrance ✓\n```\n\nGreat. One expensive evaluation effectively produced four tokens.\n\nBut perhaps the draft is:\n\n```\nthe capital of Germany\n```\n\nand the large model says:\n\n```\nthe       ✓\ncapital   ✓\nof        ✓\nGermany   ✗\n```\n\nThen we keep the accepted prefix and let the large model continue from the rejected position.\n\nThis is the basic idea of **speculative decoding**.\n\nIt is beautifully simple:\n\n```\ncheap model\n    |\n    | proposes several tokens\n    v\ncandidate sequence\n    |\n    v\nlarge model verifies them in parallel\n    |\n    +------> accept several\n    |\n    +------> reject at first disagreement\n```\n\nLeviathan et al. showed that this could accelerate generation while preserving the output distribution exactly, rather than merely producing \"approximately similar\" text.\n\nTheir 2023 paper reported roughly 2x-3x acceleration on the models they evaluated.\n\nThe crucial insight was that autoregressive generation does not mean the expensive model must *discover* every token sequentially. It only means that the final accepted sequence has to respect the autoregressive distribution.\n\nSpeculation lets us take advantage of the fact that many consecutive tokens are easy to predict.\n\nFor example:\n\n```\nfor i in range(100):\n    generate(next_token)\n```\n\ncontains many stretches where the answer is nearly obvious:\n\n``` php\n\"San\" -> \"Francisco\"\n```\n\nor\n\n``` php\n\"def\" -> \" foo\" -> \"(\"\n```\n\nor\n\n``` php\n\"New\" -> \" York\"\n```\n\nThe large model is still being asked to perform the full computation, but we are trying to amortize that expensive computation across several tokens.\n\nThis idea naturally leads to the next question:\n\nWhy maintain a second model just to make guesses?\n\nThat is where Medusa becomes interesting.\n\nMedusa, introduced by Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason Lee, Deming Chen, and Tri Dao, takes a different route.\n\nInstead of:\n\n```\nlarge model + separate draft model\n```\n\nMedusa adds several lightweight decoding heads to the existing model.\n\nConceptually:\n\n``` php\n                    +--> head 1 --> token t+1\n                    |\ntransformer trunk --+--> head 2 --> token t+2\n                    |\n                    +--> head 3 --> token t+3\n                    |\n                    +--> head 4 --> token t+4\n```\n\nThe ordinary language-model head predicts:\n\n```\nP(x[t+1] | context)\n```\n\nA Medusa head can try to predict:\n\n```\nP(x[t+2] | context)\n```\n\nanother:\n\n```\nP(x[t+3] | context)\n```\n\nand another:\n\n```\nP(x[t+4] | context)\n```\n\nThe heads are cheap compared with running the entire transformer again.\n\nSo instead of doing this:\n\n``` php\nfull model\n    -> token t+1\n\nfull model\n    -> token t+2\n\nfull model\n    -> token t+3\n\nfull model\n    -> token t+4\n```\n\nwe try to do:\n\n``` php\none full model pass\n    |\n    +--> head 1 -> candidates for t+1\n    +--> head 2 -> candidates for t+2\n    +--> head 3 -> candidates for t+3\n    +--> head 4 -> candidates for t+4\n```\n\nThe important subtlety is that these predictions are *not independent in the final decoding procedure*. Medusa uses a tree of candidate continuations and then asks the full model to verify the candidates together.\n\nThis is why \"multi-token prediction\" can sound simpler than it actually is.\n\nThe prediction is parallel.\n\nThe **verification structure** is the clever part.\n\nSuppose Medusa predicts three future positions, and we keep the top two candidates at each position.\n\nNaively, we could have:\n\n```\nposition 1: A, B\nposition 2: C, D\nposition 3: E, F\n```\n\nBut not every combination is meaningful.\n\nThe structure is really:\n\n```\n             current context\n              /           \\\n             A             B\n           /   \\         /   \\\n          C     D       C'    D'\n         / \\   / \\     /  \\   / \\\n        ... ... ...    ... ... ...\n```\n\nThis is a candidate tree.\n\nWhy?\n\nBecause the second token depends on what happened at the first token.\n\nIf the first candidate is `A`\n\n, then predictions for later tokens are conditional on `A`\n\n.\n\nIf the first candidate is `B`\n\n, the continuation is different.\n\nA tree therefore lets the system represent multiple possible future sequences without running the full model separately for every path.\n\nThis is the key engineering trick.\n\nThe transformer can process the tree-shaped set of candidate continuations with a specially constructed attention pattern.\n\nConceptually:\n\n```\n                   context\n                      |\n               +------+------+\n               |             |\n               A             B\n             /   \\         /   \\\n            C     D       E     F\n```\n\nThe model verifies many of these positions in one batched computation.\n\nThis converts part of the problem from:\n\n```\nserial execution\n```\n\ninto:\n\n```\nparallel evaluation of possible futures\n```\n\nThat is exactly the kind of workload modern GPUs are good at.\n\nThink of ordinary decoding as exploring one path through a tree:\n\n``` php\nroot -> A -> C -> F -> J -> ...\n```\n\nAt every step you pay for another expensive model evaluation.\n\nMedusa says:\n\nSpend one expensive evaluation exploring a small local subtree, then keep the path that survives verification.\n\nThat is the entire game.\n\nThe cleanest way to reason about these methods is with one quantity:\n\n```\nL = average number of tokens accepted per expensive model evaluation\n```\n\nOrdinary decoding has approximately:\n\n```\nL = 1\n```\n\nbecause one model evaluation produces one token.\n\nSuppose a speculative or Medusa-style method gets:\n\n```\nL = 2.5\n```\n\naccepted tokens per verification step.\n\nThen generating 100 tokens requires roughly:\n\n```\n100 / 2.5 = 40\n```\n\nlarge-model evaluations instead of:\n\n```\n100 / 1 = 100\n```\n\nThat gives a first-order speedup of:\n\n```\n100 / 40 = 2.5x\n```\n\nBut this is only the idealized calculation.\n\nThere is extra work:\n\n```\nhead computation\ncandidate construction\ntree attention\nverification overhead\nsampling\nkernel launches\n```\n\nSo a more realistic model is:\n\n```\nbaseline time\n    ~= N * T_model\n\naccelerated time\n    ~= (N / L) * (T_model + T_overhead)\n```\n\nand therefore:\n\n```\nspeedup\n    ~= L * T_model / (T_model + T_overhead)\n```\n\nThis equation is worth remembering.\n\nIt explains why a method that achieves `L = 3`\n\ndoes not necessarily deliver a literal 3x wall-clock improvement.\n\nFor example, suppose:\n\n```\nT_model    = 20 ms\nT_overhead = 3 ms\nL          = 3\n```\n\nThen:\n\n```\nspeedup ~= 3 * 20 / 23\n        ~= 2.61x\n```\n\nThe accelerator is still doing extra work.\n\nIt is just doing substantially less *serial* expensive work.\n\nSuppose we try to predict four future tokens, and the probability each prediction is accepted is roughly:\n\n```\np = 0.8\n```\n\nA crude approximation for the probability of getting all four accepted is:\n\n```\n0.8^4 = 0.4096\n```\n\nSo only about 41% of branches would survive all four positions.\n\nBut we do not actually need all four to succeed.\n\nGetting:\n\n```\ntoken 1 ✓\ntoken 2 ✓\ntoken 3 ✓\ntoken 4 ✗\n```\n\nis still useful.\n\nThe expected number of consecutive accepted tokens is approximately related to:\n\n```\np + p^2 + p^3 + ... + p^K\n```\n\nfor a K-token proposal horizon.\n\nWith:\n\n```\np = 0.8\nK = 4\n```\n\nthat gives:\n\n```\n0.8 + 0.64 + 0.512 + 0.4096\n= 2.3616\n```\n\nSo even though all-four acceptance happens only about 41% of the time, we can still average roughly 2.36 accepted positions before considering the stop.\n\nThis is why improving head quality can be extremely valuable.\n\nA relatively small increase in acceptance probability compounds across the sequence.\n\nHere is where terminology gets confusing.\n\n\"Multi-token prediction\" can refer to an **inference architecture**, such as Medusa-style heads, but it can also mean a **training objective**.\n\nFabian Gloeckle and colleagues at Meta proposed training language models to predict multiple future tokens from the same shared representation.\n\nInstead of only optimizing:\n\n``` php\nh_t -> x_(t+1)\n```\n\nthe model can optimize several targets:\n\n``` php\nh_t -> x_(t+1)\nh_t -> x_(t+2)\nh_t -> x_(t+3)\nh_t -> x_(t+4)\n```\n\nwith separate output heads.\n\nA simplified loss looks like:\n\n```\nL = L_1 + L_2 + L_3 + ... + L_K\n```\n\nwhere:\n\n```\nL_1 = cross_entropy(head_1(h_t), x_(t+1))\nL_2 = cross_entropy(head_2(h_t), x_(t+2))\n...\n```\n\nThe trunk is shared.\n\nThat means one representation is being asked to encode information about several points in the future.\n\nThis turns out to have an interesting side effect: it can change what the network learns internally.\n\nGloeckle et al. reported improved downstream performance, especially on code-generation tasks. For their 13B models, the multi-token prediction setup improved results on HumanEval and MBPP relative to comparable next-token models, while also offering inference benefits.\n\nThat is an interesting departure from the usual story.\n\nUsually we think:\n\n```\nextra training objective\n        |\n        v\nsame model quality\n        |\n        v\nmaybe cheaper inference\n```\n\nBut multi-token prediction can potentially provide:\n\n``` php\nextra training signal\n        |\n        +--> better representations\n        |\n        +--> better downstream capability\n        |\n        +--> faster generation\n```\n\nSo the technique is not merely a serving hack.\n\nIt can change the model's learning problem.\n\nConsider code:\n\n```\nfor user in users:\n    print(user.name)\n```\n\nKnowing the immediate next token is useful.\n\nBut knowing what is likely to happen several tokens later gives the network a signal about longer local structure.\n\nPredicting:\n\n```\nfor\nuser\nin\nusers\n```\n\nfrom the same representation forces the model to preserve information about syntactic continuation.\n\nOne interpretation is that this encourages representations that contain a more explicit local plan.\n\nThat interpretation fits the experiments in the paper, where the authors found evidence connecting multi-token prediction with the development of induction-head-like behavior and algorithmic reasoning on small tasks.\n\nThe deeper lesson is that next-token prediction is not the only useful training signal available in an autoregressive model.\n\nThe really useful question is not:\n\n\"Is Medusa clever?\"\n\nIt is:\n\n\"When should I deploy something like this?\"\n\nThe answer depends on the workload.\n\nThis is the strongest case.\n\nThink:\n\n```\ninteractive coding assistant\nchat UI\nagent tool call\nvoice response\n```\n\nIf your user is waiting for tokens, reducing serial decode steps directly reduces time-to-completion.\n\nSuppose your baseline server produces:\n\n```\n50 tokens/sec\n```\n\nand the workload typically generates:\n\n```\n150 tokens\n```\n\nThat is approximately:\n\n```\n3 seconds\n```\n\nof decode time.\n\nIf a Medusa-style approach gets an effective 2x speedup, you're around:\n\n```\n1.5 seconds\n```\n\nbefore accounting for other system overhead.\n\nThat is a very meaningful product difference.\n\nNow the economics become less obvious.\n\nLarge batches already give the GPU more parallel work.\n\nYou may be trading:\n\n```\nfewer sequential iterations\n```\n\nagainst:\n\n```\nmore candidate computation\n```\n\nand more complicated scheduling.\n\nThis is one reason the exact operating point matters.\n\nA technique that is spectacular for:\n\n```\nbatch size = 1\n```\n\ncan have a much smaller payoff when the GPU is already saturated.\n\nInterestingly, the multi-token-prediction experiments reported by Gloeckle et al. also found inference benefits at larger batch sizes, so the idea is not inherently limited to interactive serving.\n\nThe correct engineering approach is empirical benchmarking, not assuming that a published speedup transfers directly to your serving stack.\n\nSuppose you are paying roughly:\n\n```\n$4 / GPU-hour\n```\n\nfor an accelerator.\n\nA workload that consumes:\n\n```\n1,000 GPU-hours/month\n```\n\ncosts roughly:\n\n```\n$4,000/month\n```\n\nIf an optimization genuinely cuts required GPU-hours by 40%, the theoretical savings are:\n\n```\n$1,600/month\n```\n\nper equivalent GPU-hour workload.\n\nAt hyperscale, that becomes enormous.\n\nBut the real economic variable is not \"speedup.\"\n\nIt is:\n\n```\ncost per generated token\n```\n\nYou should measure:\n\n```\nGPU cost / accepted output token\n```\n\nunder the actual workload distribution.\n\nAnd this is where acceptance rate, sequence length, batching, KV-cache behavior, kernel efficiency, and scheduling all matter.\n\nThe two approaches attack the same bottleneck from different directions.\n\n```\nlarge model\n    ^\n    |\nverify\n    ^\n    |\nsmall draft model\n    |\npropose many tokens\n```\n\nAdvantages:\n\nCosts:\n\n``` php\n                    +--> head 1\n                    |\nlarge model trunk --+--> head 2\n                    |\n                    +--> head 3\n                    |\n                    +--> head 4\n```\n\nAdvantages:\n\nCosts:\n\nThis is why Medusa introduced two variants.\n\n**Medusa-1** freezes the backbone and trains the new heads.\n\nThat is operationally attractive because the original model is essentially preserved.\n\n**Medusa-2** trains the backbone jointly with the new heads.\n\nThat can give better prediction quality and higher speedups, but the training procedure has to preserve the quality of the base model.\n\nThe published Medusa experiments reported more than 2.2x acceleration for Medusa-1 in their settings, while Medusa-2 reached roughly 2.3x-2.8x in the final ICML version.\n\nThose numbers are useful as evidence that the approach works.\n\nThey should not be interpreted as a universal multiplier for every model and serving stack.\n\nThere is a broader systems principle hiding underneath all of this.\n\nModern hardware is extremely good at:\n\n```\ndo many related things simultaneously\n```\n\nIt is much worse at:\n\n```\ndo one tiny thing\nwait\ndo another tiny thing\nwait\ndo another tiny thing\n```\n\nAutoregressive decoding creates exactly the second pattern.\n\nMedusa and speculative decoding change the shape of the workload.\n\nInstead of:\n\n``` php\n             ┌── expensive ──┐\ncontext ---> │ model          │ ---> token\n             └───────────────┘\n                    |\n                    v\n             ┌── expensive ──┐\ntoken ------>│ model          │ ---> token\n             └───────────────┘\n                    |\n                    v\n                  ...\n```\n\nwe try to construct:\n\n```\n                    context\n                       |\n              candidate generation\n                       |\n          +------------+------------+\n          |            |            |\n         A             B            C\n          |            |            |\n          +------------+------------+\n                       |\n                parallel verification\n                       |\n                 accepted prefix\n```\n\nThe model is no longer merely a function that maps:\n\n``` php\ncontext -> next token\n```\n\nfor execution purposes.\n\nWe are treating it more like a machine that can cheaply explore a small neighborhood of possible futures.\n\nThat perspective opens several avenues.\n\nYou can vary:\n\n```\nnumber of prediction heads\n```\n\nYou can vary:\n\n```\nnumber of candidate tokens per head\n```\n\nYou can vary:\n\n```\ntree shape\n```\n\nYou can dynamically allocate more candidates when the model is uncertain.\n\nYou can use different proposal models.\n\nYou can train models explicitly for longer-horizon prediction.\n\nAnd you can combine these approaches.\n\nThe common theme is always the same:\n\nSpend one expensive model evaluation to obtain more than one useful token.\n\nFor an engineer evaluating one of these systems, I would start with five measurements.\n\nMeasure:\n\n```\nmilliseconds / generated token\n```\n\nat the actual:\n\n```\nbatch size\nprompt length\nsequence length\nquantization\nGPU\n```\n\nyou care about.\n\nMeasure:\n\n```\naverage accepted tokens / verification step\n```\n\nrather than merely reporting the number of heads.\n\nFour heads with:\n\n```\nL = 1.2\n```\n\nmay be much worse than three heads with:\n\n```\nL = 2.4\n```\n\nMeasure:\n\n```\ncandidate generation\ntree construction\nattention\nsampling\nsynchronization\n```\n\nseparately.\n\nOtherwise it is easy to mistake a benchmark micro-optimization for an end-to-end improvement.\n\nMeasure:\n\n```\nGPU-seconds / accepted output token\n```\n\nrather than just:\n\n```\ntokens / second\n```\n\nA server that is 20% faster but requires 30% more GPU capacity may not actually be an improvement.\n\nThis is particularly important for Medusa-style adaptation.\n\nYou care about:\n\n```\nbase-model quality\n```\n\nand:\n\n```\naccelerated-model quality\n```\n\nseparately.\n\nA speedup obtained by quietly changing the generation distribution is a different trade-off from exact speculative decoding.\n\nThat distinction should be explicit in your architecture review.\n\nAt first glance, autoregressive generation appears fundamentally sequential:\n\n``` php\ntoken t\n  -> token t+1\n       -> token t+2\n            -> token t+3\n```\n\nBut that is only the structure of the *final dependency*.\n\nIt does not necessarily mean we have to perform one expensive full-model computation for every final token.\n\nSpeculative decoding exploits this with another model.\n\nMedusa exploits it with additional decoding heads and tree verification.\n\nMulti-token prediction attacks the problem earlier, at training time, by teaching the network to predict several future tokens from shared representations.\n\nThese techniques point toward a broader shift in how we think about LLM inference.\n\nThe interesting question is no longer simply:\n\n\"How fast can I run one forward pass?\"\n\nIt is:\n\n\"How much useful sequential progress can I extract from one forward pass?\"\n\nThat is a much richer optimization problem.\n\nAnd arguably, it is one of the more important ones in LLM systems engineering because the transformer has become so large that shaving milliseconds from a single operation is often less interesting than reducing how many times we need to perform the expensive operation in the first place.\n\nFor developers building inference infrastructure, agents, coding assistants, or high-volume generation systems, that distinction can translate directly into latency, GPU utilization, and dollars.\n\nThe next time you see an LLM generating 100 tokens one by one, it is worth asking:\n\n**What if those 100 tokens only required 40 expensive model evaluations?**\n\nThat is essentially the game Medusa is playing.\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n**Try LiveReview on your codebase:**", "url": "https://wpnews.pro/news/llms-don-t-have-to-generate-one-token-at-a-time-how-medusa-and-multi-token-cheat", "canonical_source": "https://dev.to/shrsv/llms-dont-have-to-generate-one-token-at-a-time-how-medusa-and-multi-token-prediction-cheat-8ej", "published_at": "2026-09-03 18:28:00+00:00", "updated_at": "2026-09-03 18:55:22.527928+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure", "developer-tools"], "entities": ["Shrijith Venkatramana", "LiveReview", "Medusa", "Yaniv Leviathan", "Matan Kalman", "Yossi Matias"], "alternates": {"html": "https://wpnews.pro/news/llms-don-t-have-to-generate-one-token-at-a-time-how-medusa-and-multi-token-cheat", "markdown": "https://wpnews.pro/news/llms-don-t-have-to-generate-one-token-at-a-time-how-medusa-and-multi-token-cheat.md", "text": "https://wpnews.pro/news/llms-don-t-have-to-generate-one-token-at-a-time-how-medusa-and-multi-token-cheat.txt", "jsonld": "https://wpnews.pro/news/llms-don-t-have-to-generate-one-token-at-a-time-how-medusa-and-multi-token-cheat.jsonld"}}