# Doubling LLM token rates in ComfyUI > Source: > Published: 2026-11-11 16:56:59+00:00 ComfyUI runs a variety of large language models (LLMs) locally for enhancing prompts, writing stories, describing images, and even basic chatting. Recently, we rewrote the hot path of that text generation with a specific focus on the Qwen3.x family of LLMs. The result is that **the same checkpoints on the same GPU can now generate tokens up to twice as fast, with no change to the model or the quality of what it writes.** This also speeds up many of the prompt enhancers included with the recent state-of-the-art image generation models. *On a RTX 5090, Qwen3.5 9B went from 106 to 219 tokens per second and on a RTX 5080 it went from 91 to 129 tokens per second. Qwen3.8 27B describing a 1536×1536 image went from 55 to 122 on a RTX 5090.* *Qwen3.8 27B W4A8 on the RTX 5090, describing a 1500×1500 image and writing a story: 137 tokens per second with* `mtp` *on* `auto`*.* This blog discusses some architectural and implementation details of LLM inference and the recent speedups. Everything covered here is available in ComfyUI v0.45.0. Just load a Qwen3.x checkpoint, leave the Text Generate node’s `mtp` option on `auto`, and go. ## What is the bottleneck? Generating text is sequential: each new token depends on the one before it and requires a full read of the model weights per token. Large scale LLM inference can usually economize this weight read cost by batching together multiple inferences, such as multiple completely independent chats from multiple users. However, a single ComfyUI workflow usually just generates a single sequence, so every step is a batch of one. For a 27B parameter model quantised to four bits, like. Qwen3.8 W4A8, one step reads roughly 14 GB of weights to produce one token, and can do as little as one mathematical operation with each weight. The bottleneck of token generation is driven by how fast your GPU can read the model from VRAM. In contrast, diffusion models used for image or video generation in ComfyUI process an entire respective latent output on every step, with each weight applying to the full sequence length. Unlike LLMs, diffusion models are bottlenecked by compute power, defined by how fast your GPU can calculate. This creates a very different optimization problem to the image and video generation world. Most of the weights in LLM models are linear matrix multiplications. These are processed by linear kernels, known as either GEMM/GEMV kernels, which are optimized GPU kernels (programs) for matrix multiplication. In our LLM case, a dense, often square matrix is multiplied by either a vector `M=1` for a single token’s state, or a very thin matrix like `M=3` for a small-token-count Multi-Token Prediction (MTP) verification state. This is the “skinny” linear problem. In these recent changes significant optimizations for VRAM bus utilization and efficiency were added to re-optimize purely for VRAM read speed and VRAM memory bus uptime. ## What is Multi-token Prediction ? Qwen3.x checkpoints ship with a multi-token-prediction (MTP) head: a smaller extra layer compared to the the main model that utilizes the model’s hidden state to guess the next token too. ComfyUI uses this MTP feature for speculative decoding. The MTP head drafts a few tokens, then on the next token forward pass, all the candidate draft tokens are speculatively verified together. This process costs about the same as one token, because the pass is still bottlenecked by VRAM speed. Every MTP draft that matches what the main model would have chosen is accepted by the model, saving a full forward pass, and in turn saving a full model read for each correctly guessed MTP token. This is how MTP can more than double the output token rate for significantly increased performance. In these recent changes, we both optimize the compute kernel for MTP while also enhancing the MTP algorithm itself with the new adaptive-tree MTP algorithm. ## Optimizing the Linear Kernel Qwen3.x checkpoints in ComfyUI are typically W4A8 (4-bit weights, 8-bit activations) or W6A8 (6-bit weights, 8-bit activation) . In ComfyUI v0.40.0, these were handled by a general-purpose INT8 path: quantise the activation, run a matrix-vector linear kernel that is written to only match the W4A8 VRAM layout stored in safetensors files. We replaced this with an optimized matrix-vector (GEMV) kernel where the weights and their per-group scales are interleaved in memory in the order the kernel consumes them. This way, each thread block streams a contiguous run instead of having to hop between a weight tensor and a scale tensor, reducing VRAM seeking and improving efficiency of bulk VRAM read pipelines. This kernel specific VRAM re-layout is done in-place in VRAM and managed by the dynamic VRAM weight load engine, only needing to be done once per model load. This process requires no extra RAM or VRAM. This re-layout is only needed for our skinny linears, as the fatter diffusion-model linears bottleneck on compute and hide the latency of the above VRAM seeking. Most modern GPUs feature tensor cores, which compute a tile of matrix multiplication operations with a single GPU instruction. NVIDIA GPUs starting from the 30 series optimize for a minimum tile dimension of `8`, with the other rectangular dimensions able to be larger. The new kernel uses this matrix-multiply-accumulate (MMA) instruction with the weights and inputs arranged such that the skinny dimension `M,` shown in the example image above, maps onto the minimum hardware supported tile dimension of `8`. This allows for use of the single instruction while minimizing wasted un-used compute rows (when `M < 8`). For a pure single token generation case where `M=1`, this is roughly the same performance as using non-tensor (CUDA) core instructions (dp4a). However, the tiled approach instantly becomes faster when considering MTP where `M > 1`. Ultimately, up to eight rows of MTP guesses becomes almost free, as our tiler can compute up to `M=8` at the same speed as `M=1`. New prologue fusions were added to the linear kernel: the activation quantization now absorbs the RMSNorm or SwiGLU that precedes it, so those intermediate tensors are never written to memory in between, which minimizes cache memory traffic and kernel launch overhead. This in turn minimizes the small pure compute gaps between VRAM bus activity giving a small extra speedup. ## Paying attention to the attention! Qwen3.5 and 3.8 are hybrid-attention models: a type of model that uses two different attention schemes applying to separate transformer blocks. Three of every four layers are DeltaNet linear attention with a small recurrent state, and the fourth is ordinary grouped-query attention with a KV cache. For the attention layers, we added a decode kernel that handles the short query for either a single token, or the handful of draft tokens from MTP and its causal verification rows directly, without materializing an attention mask. It splits the cached prefix across many thread blocks so only a tiny query fills the GPU, and the last block to finish combines the partial results inside the same launch. For the DeltaNet layers, the expensive part was speculation. Verifying a draft of several tokens needs the recurrent state after each of them, and if a draft is rejected the state has to be restored. Previously, that meant snapshotting the full state at 64 KiB per head for every layer after every speculative token. Now the kernel records the small per-token intermediates and defers the state update so that the next call only replays the accepted tokens and writes the committed state once. The kernel in turn trades a little extra arithmetic for a lot of additional memory traffic, which fits into our core bottleneck problem of being VRAM limited while having compute spare. Previously, the KV-cache autoregressive attention kernel was distinct from the newly added optimized Comfy-Kitchen attention scheme. Comfy-Kitchen attention, along with KV Cache support, has been added in ComfyUI v0.45 enabling INT8 attention for the LLM specific attention scheme covered earlier in the blog. This reduces the VRAM footprint of the KV cache while also significantly increasing the pre-fill speed for long prompts, system prompts or pre-existing contexts. You can launch ComfyUI with `--use-ck-attention` to enable Comfy-Kitchen attention. ## Adaptive tree MTP In ComfyUI v0.40.0, MTP drafting was initially limited to a single chain of token guesses corresponding to a single guess of what the next 2 to 5 tokens would be. The chain would draft one token, feed it back to the model, then draft the next token up to a set depth value for MTP. By default, setting MTP to `auto` gave it 2 to 5 tokens adapting based on sampling mode or an interim success measurement.The weakness of this chain approach is that it bets everything on this single guess, and every guess further down the chain can be lost if incorrect. If the MTP head’s top choice at depth 1 is wrong, the whole step yields one token, even if the second best guess at depth 1 was right. *Chain MTP drafts only one path. Tree MTP keeps the runner-up tokens at each level as siblings, verifies the whole tree in one pass, and re-plans the tree’s shape from what has been hitting.* An alternative approach to avoid this potential kink in the chain, is using an adaptive tree algorithm. At each level, the MTP head’s top choice continues the main chain as before with its best guess per draft, but a small number of runner-up guesses are kept as sibling candidates for the following MTP verification. This forms a small draft tree structure, with these extra candidate leaves that get verified but are not further drafted from. Theoretically, you could draft any number of guesses with diminishing returns for each additional guess in any depth tree structure, however our linear kernel computes up to `M=8` for “free”. This sets the limit of the guesses to seven (`M=8-1`) to match our compute kernel and hardware. The verify pass carries up to eight rows now; the committed token plus seven draft tokens, while having to treat each one as several alternative branching possible futures. The type of text for both the prompt and output determines what shape the tree should take. Predictable output such as code, lists, or a description that repeats the prompt, rewards depth meaning long chains get accepted as a whole. Creative writing rewards breadth: the top choice is often wrong but the right token is usually in the top three. Adaptive MTP continually measures So MTP when set to auto mode continues to measure and re-plans as it goes. It keeps an exponentially decayed count of which rank the accepted token had, whether it was the first choice, second, third, … or none, per draft level, and it times the draft and verify phases with CUDA events. For every 32 steps, it scores all 64 ways of laying out seven draft rows as a chain of widths, estimating expected accepted tokens per step from the conditional hit probabilities and dividing by the measured cost, `verify_ms + depth × draft_ms`. It switches only if the winner is at least 3% better, to ensure the shape does not thrash, and because every shape uses the same eight rows the CUDA graphs captured for the step keep working across the switch. Adaptive tree MTP can be used when the MTP value is set to `auto` in the **Generate Text** node. ## To summarize Between ComfyUI v0.40.0 and v0.45.0, Qwen3.x text generation in ComfyUI got significantly faster on the same hardware. Longer contexts that used to run out of memory now fit cleanly, and the speculative decoder now shapes itself to the text it is writing. These improvements run on NVIDIA and AMD, Windows and Linux, and do not require additional steps from you except an up-to-date ComfyUI.