Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference

NVIDIA published the third post in its AI model co-design series, detailing how speculative decoding accelerates large language model inference by predicting multiple tokens per iteration with a small draft model verified in parallel by the target model. The post provides five guidelines for selecting draft length and draft mechanism across the Pareto frontier, emphasizing that optimal draft length depends on batch size and that speedup is maximized by balancing draft latency and acceptance length.

This post is the third in a series on AI model co-design. It explores how to accelerate LLM inference while maintaining accuracy using speculative decoding https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/ and offers five guidelines for selecting draft length and draft mechanism across the Pareto frontier. For a discussion of how model design choices impact both throughput and interactivity without sacrificing accuracy, see AI Model Co-Design: Hardware-Friendly LLM Design https://developer.nvidia.com/blog/ai-model-co-design-hardware-friendly-llm-design/ Part 1 . For an explanation of how group size query heads per KV head , head dimension, and sequence length shape the performance of dense attention, see Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference https://developer.nvidia.com/blog/co-designing-ai-model-attention-for-fast-interactive-long-context-inference/ Part 2 . What is speculative decoding? Speculative decoding is a technique for accelerating the autoregressive decoding phase of LLM inference by predicting multiple tokens per iteration. A small draft model first predicts several likely next tokens. These tokens are then verified in parallel with a single pass through the larger target model. This approach reduces the total number of decoding iterations while increasing the arithmetic intensity of the target model, without requiring higher concurrency. The target model accepts the proposed tokens in sequence until it encounters the first mismatch. The next prediction cycle then resumes from that position. Because only tokens accepted by the target model are retained, speculative decoding produces the same output sequence as standard decoding, unless the acceptance criteria are deliberately relaxed. Figure 2 illustrates this decoding flow. The draft length \ D\ is the number of proposed tokens per target iteration. The acceptance length \ AL\ is the number of tokens produced that is, accepted per target iteration. \ AL\ ranges from \ 1\ to \ 1 + D\ because the target can always produce one new ground-truth token in addition to the accepted draft tokens. The speedup from speculative decoding can be quantified as the ratio of time taken by the target model to sequentially generate \ AL\ tokens to the time taken to verify D tokens in parallel, while also accounting for the latency overhead of drafting \ D\ tokens: \ \mathrm{speedup} = \frac{T {\mathrm{verif}} B \times \mathrm{AL}}{T {\mathrm{verif}}\left B \times 1 + D \right + T {\mathrm{draft}} B, D }\ where \ B\ is the batch size, \ T {\mathrm{verif}} x \ is the target verification time for \ x\ tokens, and \ T {\mathrm{draft}} b, y \ is the time needed to generate a draft of length \ y\ for batch size \ b\ . It is evident that to maximize speedup, it’s necessary to find the optimal \ \left D, AL, T {\mathrm{draft}}\right \ tuple. Selecting the optimal draft length For simplicity, if we ignore draft model latency, speculation provides a speedup when: \ \frac{T {\mathrm{verif}}\left B \times 1 + D \right }{T {\mathrm{verif}} B } < AL\ During verification, compute scales with \ 1 + D \ , but memory access remains unchanged. Hence, the goal is to increase \ D\ until \ T {\mathrm{verif}}\ remains constant, typically up to the point where verification transitions from memory bound to compute bound. It is evident that this optimal value of \ D\ depends on \ B\ and is expected to vary across the Pareto frontier. Consider Figure 1 with this in mind. Draft length and performance of linear layers With speculation, the GEMM-\ M\ for each target linear layer GEMM grows from \ M\ to \ M \times 1 + D \ , where \ M\ is the GEMM-\ M\ with no speculation. Figure 3 shows how teraflops scale with batch for a representative expert GEMM size of 6144 x 6144 at various draft lengths. It is evident that higher draft lengths allow GEMMs to achieve peak performance at lower effective batch sizes. Notably, with \ D=7\ , one-eighth of the batch size is needed to become compute bound compared to \ D=0\ . As mixture-of-experts MoE https://www.nvidia.com/en-us/glossary/mixture-of-experts/ models become sparser and long-context workloads increase KV capacity pressure, the effective concurrency per expert decreases, making larger draft lengths attractive across the Pareto frontier. Guideline 1: Increase speculative decoding draft length to push GEMMs into the compute-bound region without increasing KV cache capacity pressure. Draft length and attention performance For reasoning and agentic workloads, attention tends to dominate the execution time in the throughput-oriented region. Decode attention has an arithmetic intensity of about \ 2 \times G\ , where \ G\ is the number of query heads that share one KV head, as explained in Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference https://developer.nvidia.com/blog/co-designing-ai-model-attention-for-fast-interactive-long-context-inference/ . Speculation increases this to \ 2 \times G \times 1 + D \ , because the speculated tokens reuse the same KV cache. The effective attention GEMM-\ M\ is \ G \times 1 + D \ . On current GPU devices, attention kernels achieve good hardware utilization at GEMM-\ M=128\ , thereby making \ D = \frac{128}{G} – 1\ the optimal draft length. Figure 4 shows normalized attention throughput as \ D\ increases for \ G = 8\ and \ G = 32\ across 32K and 128K KV sequence lengths. The \ G = 32\ variant reaches throughput saturation at a lower value of \ D\ . Beyond the throughput saturation point, attention is no longer DRAM bandwidth bound, and its runtime scales with \ D\ . Since \ AL\ grows sublinearly with \ D\ , increasing \ D\ beyond this point is likely to slow down an attention-heavy workload. Guideline 2: When attention dominates decode time, choose \ D = \frac{128}{G} – 1\ . Attention runtime also depends on tile size. Figure 5 shows that runtime increases in steps as \ G \times 1 + D \ crosses a multiple of 128, the software tile size for the benchmarked attention kernel. If \ G \times 1 + D \ falls between two tile boundaries, the last tile is only partially utilized but still costs about as much as a full tile. Guideline 3: If you choose \ D \frac{128}{G} – 1\ , prefer values where \ G \times 1 + D \ is a multiple of 128 to avoid tile underutilization. The relative weight of Guideline 1 compared to Guidelines 2 and 3 depends on the percentage of runtime spent in FFNs compared to attention at the preferred operating point. Communication also adds verification cost as D increases, although compute and communication overlap can mitigate the overhead. Draft length at the far right of the Pareto curve Moving towards the far right of the Pareto curve, \ B\ becomes very small. Here, fixed kernel setup and post-processing overheads dominate for both compute and communication kernels. These fixed costs don’t grow significantly with the number of verified tokens, keeping verification overhead largely invariant with the number of draft tokens. While MoEs do see more experts activated with increasing draft length, a combination of model sharding strategies and efficient kernels such as Grouped GEMMs can keep this overhead low. Larger draft lengths can therefore help in the low latency region, as long as acceptance remains high. At very low latency, the number of sequential kernel launches determines workload latency. Because kernel launches scale linearly with layer count, for an autoregressive draft model with a similar layer structure to the target, the speedup can be approximated as: \ \mathrm{speedup} = \frac{L {\mathrm{target}} \times \mathrm{AL}}{L {\mathrm{target}} + D \times L {\mathrm{draft}}}\ where \ L {\mathrm{target}}\ and \ L {\mathrm{draft}}\ are the layer counts in the target and draft models, respectively. Defining a constant draft depth ratio, \ \rho = \frac{L {\mathrm{draft}}}{L {\mathrm{target}}}\ : \ \mathrm{speedup} = \frac{\mathrm{AL}}{1 + \rho D}\ and draft overhead, \ O d = \rho D\ In other words, increasing \ D\ helps only while the gain in \ AL\ is large enough to offset the draft overhead. Guideline 4: At very low latency, increase \ D\ only while the gain in \ AL\ justifies the added draft cost. Selecting a draft mechanism Selecting \ D\ tells you how many tokens to speculate. Next, decide how to generate these tokens to maximize speedup. A multitude of techniques have been proposed over the years, with different tradeoffs in training, parameter, and runtime costs. External drafting suggests using a small standalone LLM, while MTP, EAGLE-3, DFlash, and DSpark use auxiliary layers combined with information from the target model to predict tokens. Suffix and n-gram methods are model-free and instead reuse patterns already seen in the token stream. Table 1 compares key drafting methods, including how they generate tokens, their serve-time memory, and their draft overhead. External draft model | EAGLE-3 | MTP | DFlash | DSpark | Suffix / n-gram | | Draft architecture | Small LLM | Decoder layer s + linear projection | Decoder layer s + linear projection | Decoder layer s + linear KV fusion | Decoder layer s + linear KV fusion + lightweight Markov head | String matching, no model | Inputs | Token IDs | Previous-token embedding + target hidden states | Target final hidden state | Fused target hidden states used as draft KV | Fused target hidden states used as draft KV | Token stream | Generation method | \ D\ autoregressive forward passes | Runs the module \ D\ times | Runs the module \ D\ times | Generates \ D\ tokens in one parallel step | Generates \ D\ tokens in one parallel step, followed by a lightweight sequential correction | One lookup | Release method | Separate model release | Separate post-training head | Part of the base model checkpoint | Separate post-training head | Separate post-training head | Independent of target | Training cost | From scratch: 1T-10T+ tokens Distilled: 100B-400B Adapted: 10M-1B | 1-10B additional tokens after target training | Usually trained with the target during pretraining | 1-10B additional tokens after target training | 1-10B additional tokens after target training | None | Activation memory cost | Draft weights + full KV cache | Weights + small KV cache | Weights + small KV cache | Weights + small KV cache built from target | Weights + small KV cache built from target | None | Speculation cost per \ D\ tokens | Full draft model forward | 1 layer, serial | 1 layer, serial | ~5 layers, once | ~5 layers once + lightweight Markov head, serial | O 1 lookup | Best for | LPU and LPU+GPU Not recommended for GPUs | Not recommended – lower \ AL\ than MTP | Best for larger models on GPUs | Smaller models, batch size 1 | Smaller models, batch size 1 | Workloads with high repetition | Table 1. Comparison of common draft mechanisms across inputs, generation method, training cost, serve-time memory, and draft cost To quantify the tradeoff between \ AL\ and \ O d\ , begin by observing how \ AL\ scales with \ D\ . Figure 6 shows how AL changes as D increases on SPEED-Bench https://huggingface.co/blog/nvidia/speed-bench , with Qwen 3.5 122B A10B as the target. SPEED-Bench is a speculative decoding benchmark developed by NVIDIA with the goal of representing realistic production workloads. It covers multiple task domains such as coding and summarization and has a variety of splits at different input sequence lengths. We recommend SPEED-Bench for \ AL\ comparisons. On the 32K split, Qwen 3.5 35B A3B reaches an \ AL\ of 6 at \ D = 9\ , while the 4B draft reaches an AL above 5. MTP and DFlash ALs level off as \ D\ grows. N-gram has lower acceptance on this workload and is better suited to workloads with repeated token patterns. Higher \ AL\ does not equal higher speedup. You also need to consider how much it costs to generate the draft. As shown in Figure 6, all external drafts reach higher \ AL\ than other draft methods at \ D 3\ . The Qwen 3.5 122B MTP is much smaller, with 2.5B total parameters and fewer than 150M active parameters. At \ D = 3\ , a larger external draft may not be worth the added cost. At \ D = 21\ , its higher AL may make that cost worthwhile, assuming an efficient serving strategy. Meanwhile, DFlash \ AL\ s level off relatively quickly. However, both DFlash and DSpark generate \ D\ draft tokens in parallel, lowering \ O d\ . For comparison, at minimum latency and with \ D=11\ , a one-layer MTP head takes 11 steps, leading to \ O d^{\mathrm{MTP}} = \frac{11}{L {\mathrm{target}}}\ , while a five-layer DFlash head generates the draft in one pass, giving \ O d^{\mathrm{DFlash}} = \frac{5}{L {\mathrm{target}}}\ . For a large target model with many layers, both overheads are negligible, but as \ L \mathrm{target}\ decreases, the draft overhead becomes significant. For small models at low latency, therefore, DFlash or DSpark can be the optimal choice even with lower \ AL\ . To navigate the tradeoff between \ AL\ and draft latency, it is crucial to benchmark both accurately. In the context of speculative decoding, this means benchmarking with realistic prompts and covering a wide set of task domains. We recommend SPEED-Bench https://huggingface.co/blog/nvidia/speed-bench for \ AL\ measurements and high performance inference frameworks such as NVIDIA TensorRT LLM https://github.com/NVIDIA/TensorRT-LLM to quantify draft overhead. Beyond inference performance, draft training scope and costs deserve consideration. While MTP must be co-trained with the target model, EAGLE, DFlash, and DSpark can be added to the final model checkpoint. Similarly, a continuum of options exists for external draft model creation. Training from scratch and distillation from the target yield the highest ALs, while adapting an existing model to the desired target through cross-model adaptation techniques can significantly reduce training costs at the expense of \ AL\ . Fine-tuning the target model can change its output distribution and hidden representations. Learned drafters trained for a particular target checkpoint can therefore lose acceptance even when the target model improves. After changing the target, remeasure \ AL\ on representative workloads. The required adaptation depends on the draft mechanism. MTP is part of the target model and should continue to be trained during fine-tuning or be realigned in a dedicated follow-up stage. Target-attached drafters such as EAGLE-3, DFlash, and DSpark use the target’s hidden states and will need to be adapted to the updated checkpoint with additional training. External draft models do not consume target hidden states, but they still approximate the target’s output distribution and may require fine-tuning or distillation. Suffix and n-gram methods have no learned drafter and therefore require no retraining, although their effectiveness still depends on how much repetition appears in the deployed workload. Guideline 5: Choose the draft mechanism that gives the best decode speedup for your workload and hardware, considering \ AL\ , draft latency, and training and deployment cost. Get started co-designing speculative decoding Use the five guidelines summarized below as a checklist for selecting \ D\ and draft mechanism across the Pareto frontier. - Increase speculative decoding draft length to push GEMMs into a compute-bound region without increasing KV cache capacity pressure. - When attention dominates decode time, choose \ D = \frac{128}{G} – 1\ as a starting point. - For larger draft lengths, prefer values where \ G \times 1 + D \ is a multiple of 128 to align with the attention kernel tile size. - At very low latency, use a fast draft mechanism and increase \ D\ only while the gain in \ AL\ justifies the added draft cost. - Choose the draft mechanism by balancing \ AL\ and draft overhead for your workload and hardware. Benchmark both under realistic serving conditions and consider training and deployment cost. For post-trained drafters, ready-to-run training examples are available in NVIDIA/Model-Optimizer https://github.com/NVIDIA/Model-Optimizer for EAGLE-3, DFlash, and DSpark. See how we did this for NVIDIA Nemotron 3.5 Lightning: fine-tune DSpark https://github.com/NVIDIA/Model-Optimizer/blob/main/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hf streaming dspark warmstart.yaml , then quantize it to FP8 https://github.com/NVIDIA/Model-Optimizer/blob/main/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hf dspark ptq fp8.yaml or NVFP4 https://github.com/NVIDIA/Model-Optimizer/blob/main/tools/launcher/examples/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/hf dspark ptq nvfp4.yaml . Use the examples as starting points, then validate AL and end-to-end speed on your own model, workload, and hardware. Acknowledgments This post is a cross-team NVIDIA effort. We are grateful to Bhargava Gopireddy, Ritika Borkar, Dor Tsur, Andrii Skliar, Benjamin Chislett, Yaniv Galron, Talor Abramovich, Yoav Miron, Rabeeh Karimi Mahabadi, Roger Waleffe, Udi Karpas, Ran Zilberstein, Brian Pharris, and Eduardo Alvarez for their help.