Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM AMD detailed MonoKernels written in FlyDSL, a domain-specific language hosted in the ROCm GitHub repository, which fuse multiple dependent operator stages into a single GPU kernel launch to cut decode latency for interactive LLM serving in vLLM on AMD Instinct GPUs. The approach uses a persistent launch, inspired by TileRT, in which a fixed grid of thread blocks reuses blocks across stages, prefetches later weights such as output-projection and expert up/gate weights during earlier computation, and hands off intermediate results through tagged mailboxes at the granularity the consuming stage needs. AMD frames the design as targeting low-concurrency, per-user responsiveness rather than aggregate throughput, and notes that kernel boundaries are model-specific rather than a universal one-layer, one-kernel template. Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM Introduction Interactive LLM applications depend on low decode latency. A coding agent must finish generating its current action before it can call the next tool, so decode sits on the task’s critical path. Serving these workloads at low concurrency prioritizes per-user responsiveness over aggregate throughput. A decode step executes a chain of GPU kernels for operations such as projections, attention, and expert computation. At small batch sizes, many of these kernels do only a little work, yet each still incurs GPU-side dispatch, startup, and completion costs. A dependent kernel cannot proceed until its inputs are ready, so whole-kernel boundaries also limit overlap. Repeated across a long chain, these costs can account for a substantial part of decode latency, while the small amount of parallel work leaves GPU capacity unused. HIP graphs reduce CPU launch overhead, but do not eliminate these GPU-side costs. Figure 1 illustrates the repeated stages; widths and gaps are schematic, not measured durations or an overhead breakdown. This post explores how MonoKernels written in FlyDSL https://github.com/ROCm/FlyDSL reduce decode latency in vLLM for interactive LLM serving on AMD Instinct GPUs. We explain the approach and its integration with vLLM, then examine performance and accuracy. How MonoKernels Reduce Decode Latency A MonoKernel combines multiple dependent operator stages inside a single GPU kernel launch, using an explicitly designed schedule rather than a separate launch at every operator boundary. Here, mono means a single kernel execution boundary, not an entire model in one kernel. A model’s mono-decode path can contain several MonoKernels, and a decoder layer does not have to map to exactly one launch. The goal is not just to make individual operators faster, but to shorten the waits between them. Inspired by TileRT https://github.com/tile-ai/TileRT , the implementations discussed here use a persistent launch : a fixed grid of GPU thread blocks works through multiple stages. A thread block is a group of GPU threads executing together; it can continue with subsequent work once the required dependencies are satisfied. Both paths in Figure 2 can be replayed under HIP graphs. The difference is between multiple independent kernels and an author-designed persistent schedule within a launch. Some whole-kernel boundaries become finer-grained handoffs between portions of the computation. - Reuse thread blocks across stages. After completing its current work, a block can take on ready work from the next stage without a separate kernel launch. This reduces repeated GPU kernel startup and drain. - Read later weights while earlier work computes. For example, output-projection or expert up/gate weights can be prefetched during earlier computation. Where the schedule and resources allow it, that overlaps part of the HBM traffic with useful work rather than exposing all of it on the critical path. It does not eliminate weight reads or guarantee that all memory latency is hidden. - Hand off data at the granularity the consumer needs. A tile is a portion of a tensor. A tagged mailbox holds an intermediate result and a tag indicating readiness. A downstream stage, or a reduction on a peer GPU, can proceed once the required data is ready rather than waiting for unrelated work in the producing stage. Consumers still wait for all inputs required by their own computation. Choosing Model-Specific Kernel Boundaries A key design challenge is deciding where a MonoKernel begins and ends : which operations share one launch, and which remain in separate kernels. Fusing more stages is not automatically better. The boundary must fit the model’s data dependencies, communication pattern, and GPU resource limits. MonoKernel boundaries are therefore model-specific, rather than a universal one-layer, one-kernel template. For example, the initial DeepSeek-V4.1-Flash implementation 60397 https://github.com/vllm-project/vllm/pull/60397 divides the standard decoder-layer path into K1 and K2. Figure 3 shows the operations within each boundary, not their execution timing. K1 prepares attention inputs and updates the KV cache. K2 completes attention and continues through the feed-forward work, with both TP all-reduces using symmetric peer memory inside the kernel. These reductions sum in fp32 in rank order, then convert to bf16, and overlap with surrounding work. Their placement inside K2 does not mean that communication or cross-rank synchronization disappears. Operator-level mapping for this example | Launch | Stages | |---|---| | K1 | mHC hyper-connection mixing before attention → attention front: wqkv , q/kv norms, KV-cache insert, wq b | | K2 | attention back split attention, combine, wo a , wo b → TP all-reduce → mHC mixing before the FFN → MoE router, top-6, shared expert, MXFP4 experts → TP all-reduce | A narrower boundary: MiniMax-M3. The initial implementation 62712 https://github.com/vllm-project/vllm/pull/62712 fuses the FFN portion, including its surrounding communication, while attention stays on the stock path. Compared with the DeepSeek example, it chooses only part of the layer for mono execution. These examples illustrate different boundary choices, not a fixed recipe that transfers unchanged to every model. Why FlyDSL Fits This Execution Model The boundary and schedule are designed by the kernel author, not automatically created by choosing a language. FlyDSL https://github.com/ROCm/FlyDSL , AMD’s Python DSL and MLIR compiler for high-performance ROCm kernels, provides the controls needed to express and compile that design. The author designs, FlyDSL compiles, and vLLM selects and integrates the execution path. Figure 4 shows the hand-off from Figure 2 between two thread blocks, with the FlyDSL primitive behind each step. Block i stores its result, waits until the stores have landed s waitcnt and a barrier , then raises one flag per token with a mailbox write. Block j issued its stage-B weight loads earlier, so the weights arrive while it polls the flags it needs; it starts stage B as soon as they are up, with no kernel boundary in between. FlyDSL compiles this order as written; it does not fuse or reorder stages on its own. The DeepSeek-V4.1-Flash attention front https://github.com/vllm-project/vllm/blob/7691ff6e5b70930b80d09ec14fed84d1a2ea612f/vllm/models/deepseek v41/amd/mono/attention/front.py in vLLM follows this pattern. The model, GPU architecture, and supported step widths still determine which variants can run. The following section explains how vLLM integrates them without replacing the rest of the serving engine. Implementation details: layouts, compilation and supported configurations Control data movement and synchronization. To overlap stages, the implementation must decide how work is divided across blocks, waves, and threads, when loads are issued, and when results become available. FlyDSL lets the author express these choices down to specific buffer and matrix instructions where needed. The benefit comes from the resulting schedule, not from Python syntax itself. Match layouts and low-precision computation to each stage. Projections, attention, and experts do not all use the same data layout. FlyDSL lets the author choose how each stage partitions and accesses its tensors, and map low-precision computation to the GPU's matrix instructions. On MI355X, the mono implementation can read vLLM's loaded weights in place, including AITER's shuffled MXFP4 expert layout, rather than keeping a second copy. These controls also support porting, but do not make it automatic: CDNA3 needs tile sizes suited to its smaller shared memory LDS and different matrix-instruction paths from CDNA4. Specialize and compile for the supported configuration. Kernels written with @flyc.kernel and @flyc.jit are compiled on first use and cached. The implementation is specialized for its model, TP size, and supported step widths—the number of token rows processed in a forward step, including speculative positions and graph padding. vLLM builds the required width variants during warm-up and replays them under HIP graphs. This is targeted specialization, not a guarantee that one schedule serves arbitrary models or batch sizes. The kernel source remains in the model's vLLM directory, with the flydsl wheel as its new runtime dependency. Deterministic compilation also supports ISA and resource-usage comparisons during review. Specialized Decode in vLLM To vLLM, a MonoKernel path is a specialized decode : a model- and hardware-specific decode forward pass behind vLLM’s normal interfaces. The scheduler, KV-cache manager, sampling, speculative decoding, and HIP graphs stay unchanged; only the decoder layers run differently, and only on steps the kernels can serve. Every merged mono PR follows the rules agreed in RFC 60904 https://github.com/vllm-project/vllm/issues/60904 : - Opt-in, refused when unsupported. The path is off by default; --monokernel turns it on equivalent to --additional-config '{"decode backend": "latency specialized"}' , which Kimi-K3 reads . A deployment the kernels can’t serve fails at startup with every reason listed. - The model owns the code. Kernels, runtime, and tests live under vllm/models/