Speculative Decoding in vLLM on AMD GPUs AMD's experiments with speculative decoding in vLLM on AMD Instinct MI300X and MI355X GPUs using the ROCm platform show that output-token throughput gains vary by drafting method, proposal length, model family, and workload. The post reviews five drafting approaches—native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark—and provides practical tuning guidance for the technique, which lets a target model verify multiple drafted tokens in a single pass. Exploring Speculative Decoding in vLLM on AMD GPUs TL;DR: Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting methods and proposal lengths, and also depended on the model family, draft checkpoint, workload, and acceptance behavior. Introduction Large language models support a wide range of applications, but serving them at scale requires careful optimization. Standard autoregressive decoding is the baseline used by most LLM serving systems: the model generates one token, appends it to the sequence, and then uses the updated sequence to generate the next token. This process is simple and reliable, but the serving loop still advances one committed token at a time because output tokens must be produced in strict left-to-right order. Speculative decoding \ 1\ ref-1 builds on this baseline through a draft-and-verify mechanism. A lightweight draft component proposes candidate future tokens, and the target model verifies those candidates before they are committed. When several draft tokens are accepted, the system can commit multiple output tokens from a single target-model verification step while preserving the target model's output behavior. This post explores how speculative decoding works in vLLM and shares measurements from our test environment. We first review the autoregressive decoding baseline and the draft-and-verify process. We then examine five speculative-drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. These methods differ in how the draft component receives information from the target model and whether candidate tokens are generated sequentially, autoregressively, in parallel, or through a hybrid approach. Finally, we show how to enable the methods tested in our environment, report measurements from our experiments on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ open software platform, and discuss practical tuning and observability considerations. The autoregressive decoding baseline In standard autoregressive decoding, each decode step produces and commits one new token. For example, generating four output tokens requires four sequential decode steps: After each step, the generated token is appended to the sequence and becomes part of the input for the next step. This makes the decoding loop straightforward, but it also requires one model decode step for every output token. During long generations, this token-by-token loop can dominate latency and limit serving throughput. The key question behind speculative decoding is therefore: Can we preserve the output behavior of the original model while reducing how often generation advances by only one token at a time? Speculative decoding addresses this by separating proposal from verification. A draft component first proposes several candidate future tokens. The original model, acting as the target model, then verifies those candidates before they are committed. Core idea of speculative decoding Speculative decoding does not replace the original model. Instead, it keeps the original model as the target model, which remains responsible for the final output, and adds a faster proposal stage in front of it. The process has two parts: - Draft: propose several candidate future tokens. - Verify: use the target model to check those candidates. During each speculative decoding round, as illustrated in Figure 1, a lightweight draft component proposes one or more future tokens. These tokens are only candidates and are not committed immediately. The target model then evaluates the candidate token sequence in one verification pass. Verification proceeds from left to right. Each draft token is checked using the target model's result at the corresponding position. Accepted tokens are committed to the output sequence. When a draft token is rejected, later candidates from the same proposal are no longer accepted. If a draft token is rejected, the target model provides the next token. The remaining draft tokens are discarded, and generation continues from the updated sequence. Conceptually, standard autoregressive decoding advances like this: Speculative decoding instead allows several candidate positions to be evaluated together: This can reduce the number of target-model decoding rounds when multiple candidates are accepted. When the draft component produces tokens that the target model accepts, several output tokens can be committed from one target-model verification step. When a proposal is rejected, the target-side result determines how generation continues. A simple accept/reject example Figure 2 gives an example of one speculative decoding round. Green boxes are draft tokens that survive verification, the red box marks the first rejected draft token, and the gray box is a later draft token that is discarded. The blue token in the output comes from the target model, not from the draft proposal. Suppose the current prompt is: The draft component proposes several future tokens: The target model verifies the draft tokens from left to right: The first two draft tokens, sunny and and, are accepted. At the third position, the draft proposes warm, but the target model selects clear. The remaining candidate, outside, is discarded because it follows the first rejected position. The next decoding round therefore continues from: How the drafting methods work Although all speculative decoding methods follow the same overall draft-and-verify process, they differ in how the draft component is designed and how it works with the target model. The main differences are: - The type of information received from the target model. - How this information is incorporated into the drafting process. - Whether candidate tokens are generated sequentially or in parallel. Based on these differences, the drafting methods discussed in this post can be grouped into three broad categories: native MTP modules, separate MTP drafters, and dedicated target-conditioned draft networks. - Native MTP modules: built directly into the target-model architecture; use a model-native auxiliary prediction path; generate candidate tokens sequentially. - Separate MTP drafters: use a separate checkpoint paired with a specific target model; use target-model activations and shared KV-cache information during inference; generate candidate tokens sequentially. - Dedicated target-conditioned draft networks: use separate speculator models trained for a specific target model, including EAGLE-3, DFlash, and DSpark. EAGLE-3 drafts autoregressively from target-model hidden states, DFlash drafts parallel blocks from target-model hidden states, and DSpark adds lightweight causal correction and confidence-based prefix selection. These categories describe the draft component architecture, not the target-model family. A target model may support native MTP while also having separately trained EAGLE-3, DFlash, or DSpark draft models. The draft component does not operate entirely on its own. Depending on the method, the draft component may receive: - A hidden representation from the target model. - Hidden states from several selected target layers. - The target model's KV cache. - Features produced by combining multiple target-model representations. The following sections explain how each method uses this information and how it generates candidate tokens. Native MTP Multi-Token Prediction, or MTP, refers to a family of model-native mechanisms for predicting tokens beyond the immediate next token. In vLLM, native MTP is available when the target model includes a compatible auxiliary prediction component \ 2\ ref-2 . The exact MTP architecture varies across model families, but each implementation provides an auxiliary path for proposing future tokens. At the first speculative step, the MTP component combines a hidden representation from the target model with information from the current token to predict the first draft token. At subsequent steps, the newly drafted token and the hidden state produced by the previous MTP step are used to predict the next candidate. After the configured number of candidates has been proposed, the target model evaluates them together in one verification pass. First draft token Subsequent draft tokens Many native MTP implementations follow a similar pattern. A hidden representation from the target model or from the previous MTP prediction is combined with the embedding of a shifted input token or the latest drafted token: The two inputs serve different purposes: 1 the hidden representation carries information about the preceding sequence; and 2 the token embedding identifies the latest token from which drafting continues. In common implementations, they are combined along the hidden dimension and transformed before entering the auxiliary prediction layer. The number of physical MTP layers and the configured speculative length are separate concepts. When num speculative tokens exceeds the prediction depth directly provided by the checkpoint, vLLM can reuse the MTP path through additional forward passes. A larger value therefore proposes more candidates before verification, but also introduces more sequential drafting work. Native MTP is closely tied to the target-model architecture. In many implementations, parts of the MTP path share components with the target model, which can keep the additional memory overhead relatively modest. However, generating multiple speculative tokens still requires sequential drafting before verification. Gemma 4 MTP Gemma 4 uses a separately packaged MTP draft component paired with a specific target model \ 3\ ref-3 . Although the draft component has its own checkpoint, it remains closely connected to the target model during inference. The draft component uses activations produced by the target model and shares the target model's KV cache. This allows it to reuse contextual information that the target has already computed instead of processing the accepted prefix independently. As with native MTP, the number of layers in the draft component is separate from the configured speculative length. When several candidate tokens are requested, the draft component generates them sequentially: EAGLE-3 EAGLE-3 uses a dedicated draft network trained for a specific target model. The draft component has its own execution path, but it remains closely conditioned on information produced by the target model \ 4\ ref-4 . During the target-model forward pass, EAGLE-3 records hidden states from three stages of the target Transformer: near the beginning, around the middle, and near the end. These are contextual representations of the same accepted sequence at different stages of target-model processing. The three hidden states are concatenated and projected into a single fused target feature. This fused representation is then combined with the embedding of the sampled token before entering the EAGLE-3 draft decoder. The two inputs serve different purposes: - The fused target feature summarizes the accepted sequence using information from several stages of the target-model forward pass. - The sampled-token embedding identifies the token from which drafting continues. EAGLE-3 generates draft tokens autoregressively. For the first draft token, it uses the fused target feature computed from the accepted sequence together with the sampled-token embedding. After a draft token is produced, its embedding is fed into the next drafting stage. Because the target model has not yet processed the later speculative positions, target-model hidden states for those positions are not available. EAGLE-3 therefore uses the previous draft-component output when continuing the draft sequence. First draft token Subsequent draft tokens This sequential feedback gives later draft tokens direct dependence on earlier drafted tokens along the proposed sequence. However, generating more speculative tokens also requires more sequential drafting work before verification. DFlash DFlash uses a dedicated draft network trained for a specific target model. Unlike MTP and EAGLE-3, which generate candidate tokens sequentially, DFlash predicts a whole block of future positions in parallel \ 5\ ref-5 . DFlash begins each draft block with an anchor token. The anchor is a known token produced or confirmed by the target model, so DFlash does not need to predict it. Instead, it provides a known starting point for the masked positions that follow. In later decoding rounds, this is typically the additional target token returned by the previous verification pass. The anchor occupies the first position of the block, while the remaining positions are masked and predicted in parallel: A draft block starts with a confirmed anchor token, followed by masked positions: Here, anchor is the known target-model token, while the masked positions are predicted by DFlash. A single DFlash forward pass predicts all masked positions together: Like EAGLE-3, DFlash first combines hidden states from several target-model layers into a fused representation. The main difference is how this fused representation is used. EAGLE-3 combines it with the sampled-token embedding at the input of its autoregressive draft network. DFlash instead converts the fused target context into additional Key and Value representations that are available in every layer of the draft network. Queries from the masked draft positions can therefore attend to both: - Key and Value representations derived from the target model. - Key and Value representations produced from the draft block itself. Available in every draft layer The target-model context therefore remains available throughout the draft network, rather than being supplied only once at its input. After the draft block has been generated, the target model evaluates all proposed tokens in one verification pass. The acceptance decision is then applied from left to right: accepted tokens are committed until the first rejection, and the remaining candidates are discarded. Here, the target-model token replaces the first rejected draft token, while the remaining draft tokens are discarded. A defining characteristic of DFlash is that all masked positions are predicted together in one draft-network forward pass. This differs from sequential drafting: Because all masked positions are predicted together, a later position is not conditioned on the sampled output of an earlier position during the same pass. This removes the token-by-token feedback used by autoregressive drafting. The effectiveness of later positions therefore depends on the trained checkpoint and workload, particularly when longer draft blocks are used. DSpark DSpark extends parallel drafting with two additional mechanisms: - A lightweight sequential head that introduces dependence between tokens within the draft block. - Confidence-based selection of the prefix submitted for target-model verification. DSpark uses a modified DFlash model as its parallel backbone \ 6\ ref-6 . The backbone performs the main draft computation for all positions in one forward pass, producing a hidden state and a set of base logits for each draft position. It therefore inherits the target-context conditioning described in the DFlash section. A fully parallel draft component predicts every position without first seeing the tokens selected at earlier positions in the same block. When several continuations are plausible, this can produce inconsistent combinations. For example, both "of course" and "no problem" may be reasonable continuations, but independent position-wise predictions could produce "of problem." DSpark addresses this behavior by applying a lightweight sequential head after the parallel backbone. The backbone still computes the base logits for every position together. The sequential head then selects tokens from left to right, adjusting each position using information from the previously selected draft tokens. DSpark applies a lightweight Markov head that introduces dependence between the selected draft tokens. For each position, the Markov head uses the immediately preceding selected token to produce a small bias. This bias adjusts the base logits produced by the parallel backbone: The main draft network processes all candidate positions together in one forward pass. After that, only the lightweight Markov head runs from left to right to adjust each position using the previously selected draft token. This allows later draft tokens to depend on tokens already selected within the same block without running the full draft network again for every position. The DSpark design also includes a confidence head that can select a shorter draft prefix for target-model verification. This feature was not active in the vLLM path used for our experiments, so the benchmark results reflect only the parallel draft network and lightweight Markov correction. The target model evaluates the proposed sequence in one verification pass, and draft tokens are committed from left to right until the first rejection. Summary of the drafting methods Figure 3 gives a visual side-by-side view of the five drafting methods: what the draft component looks like, which target-model information it uses, and whether candidate tokens are generated sequentially or in parallel. The table below the figure restates the same comparison in a compact form. In all five methods, the target model still evaluates the proposed sequence in one verification pass, and the acceptance decision is applied from left to right until the first rejected draft token. | Method | Draft component | Target-model information used | How draft tokens are generated | |---|---|---|---| | Native MTP | Model-native auxiliary MTP path | A target-model or previous MTP hidden representation combined with current draft-token information | Sequentially through repeated use of the MTP path | | Gemma 4 MTP | Separate MTP draft component paired with the target model | Target-model activations and the shared target KV cache | Sequentially through the paired MTP component | | EAGLE-3 | Dedicated autoregressive draft network | Hidden states captured near the beginning, around the middle, and near the end of the target-model forward pass, fused into one representation | Sequentially, with each drafted token influencing the next | | DFlash | Dedicated parallel draft network | Fused target-model hidden states provided as additional Key and Value information in every draft layer | All candidate positions are predicted together in one parallel forward pass | | DSpark | DFlash-style parallel draft network with a lightweight Markov head | The same target-conditioned information used by the parallel draft network | One parallel forward pass followed by lightweight sequential adjustment of token selection | How to enable speculative decoding in vLLM In vLLM, speculative decoding is configured through --speculative-config . The main differences are the method name, whether a separate draft checkpoint is required, and the number of candidate tokens requested. Current vLLM supports mtp, eagle3, dflash, and dspark as method values. | Method | Separate draft checkpoint | Typical configuration | |---|---|---| | Native MTP | No | "method": "mtp" "num speculative tokens":