Kimi K3 on vLLM: Up to 370 Tokens/sec

VLLM announces efficient day-0 support for Moonshot AI's Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model, achieving up to 370 tokens per second with speculative decoding on 16 NVIDIA GB300 NVL72 GPUs. The open-source serving engine supports Kimi K3's hybrid recurrent and full-attention architecture, including Kimi Delta Attention and MXFP4 weights, with production features like prefill/decode disaggregation and agentic KV caching.

Kimi K3 Is Here: Efficient Day-0 Support on vLLM We're thrilled to announce efficient day-0 vLLM support for Kimi K3, one of the most powerful open-weight models ever released. Last week, we previewed https://vllm.ai/blog/2026-07-22-kimi-k3-preview the production-scale integration work for Kimi K3; today, Moonshot AI's weights are public and the support is live. Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model 16 of 896 experts active per token built on Kimi Delta Attention KDA and Attention Residuals AttnRes , with a 1M-token context window and native vision. For us, the most exciting challenge Kimi K3 brings is making KDA, MXFP4 MoE, KV cache management, prefill/decode disaggregation, speculative decoding, and long-context deployment recipes work together in a runnable serving engine. The preview post https://vllm.ai/blog/2026-07-22-kimi-k3-preview explained the kernel and cache architecture, in particular the challenge of prefix caching that works on recurrent state. This release post is the practical guide: how vLLM adapts to Kimi K3's architecture, the kernel work behind the numbers, and what is ready on day 0. Quick start The easiest way to run Kimi K3 is to use 8 NVIDIA B300 GPUs or 8 AMD MI355X GPUs with the above command. Inferact has also trained and open-sourced a DSpark speculator https://huggingface.co/Inferact/Kimi-K3-DSpark for Kimi K3. Enable it by adding the following option to the serve command: For more details, including Docker images for various platforms and deployment strategies, refer to the detailed recipes https://recipes.vllm.ai/moonshotai/Kimi-K3 . Because of complicated dependencies, only Docker images are usable now. The Docker images depend on several pre-release dependencies, including FlashInfer https://github.com/flashinfer-ai/flashinfer . TL;DR A 2.8-trillion-parameter multimodal MoE: Kimi K3 activates 16 of 896 experts per token, supports a context window of up to 1M tokens, and is built on Kimi Delta Attention, Attention Residuals, LatentMoE, and native MXFP4 4-bit weights. Up to 370 tok/s per user: vLLM serves Kimi K3 at 118 tok/s without speculative decoding and 370 tok/s a 3.14× improvement with DSpark on 16 NVIDIA GB300 NVL72 GPUs, powered by extensive optimizations for Kimi K3's architecture. Broad production feature support: vLLM supports speculative decoding, prefill/decode disaggregation, agentic KV caching with Mooncake, tool calling, reasoning output, and structured output, with NVIDIA Hopper and Blackwell and AMD MI355X support at launch. Open-source DSpark support: vLLM supports the state-of-the-art block-diffusion speculative decoding algorithm for Kimi K3, trained with vLLM and TorchSpec and open-sourced by Inferact. Hybrid prefix caching: serving Kimi K3's recurrent and full-attention design required a redesign of hybrid prefix caching over recurrent KDA state. This change now benefits every hybrid linear model. Kimi K3's architecture, and how vLLM serves it Kimi K3 architecture innovations, from the original release blog post. Kimi K3's architecture departs from a standard Transformer in a few ways, and each one changes what a serving engine has to do. The preview post https://vllm.ai/blog/2026-07-22-kimi-k3-preview covers the internals in depth; here we recap what's new and focus on how vLLM adapts to serve it. Kimi Delta Attention: a hybrid recurrent + full-attention stack What's new: Most of Kimi K3's layers are KDA, a linear-attention mechanism that keeps a fixed-size recurrent state instead of a growing KV cache, interleaved with periodic full-attention layers that preserve exact global recall. That is what makes a 1M-token context affordable. How vLLM serves it: A single hybrid KV-cache manager holds two kinds of memory side by side under one scheduler: paged KV blocks for the full-attention layers, and compact recurrent-state blocks for the KDA layers. A dedicated KDA attention backend runs FlashKDA for prefill and a fused CUDA kernel or the Flash-Linear-Attention/Triton path when running speculative decoding for decode. The hardest part is prefix caching across Kimi K3's hybrid cache: full-attention layers store per-token KV, while KDA layers update recurrent and convolution state at every token but cannot afford to retain a snapshot at every possible prefix boundary. vLLM decouples the large physical KDA state blocks from fine-grained prefix matching, registering state snapshots within those blocks and copying them before extension so long shared prompts can reuse both KDA state and paged KV. This hybrid-cache machinery is new in vLLM core https://vllm.ai/blog/2026-07-22-kimi-k3-preview and now benefits every hybrid model similar to Kimi K3. Kimi K3 interleaves Kimi Delta Attention layers with periodic full-attention layers; vLLM's hybrid cache manages recurrent state and paged KV together. Attention Residuals: learned mixing of residual contributions across depth What's new: For each token, Block AttnRes replaces ordinary residual accumulation with depth-wise attention: every Transformer sublayer uses a learned pseudo-query to weight RMS-normalized residual states from preceding layer blocks, then receives the corresponding weighted combination as its input. How vLLM serves it: vLLM uses optimized Triton and CUDA kernels to compute the depth-wise attention logits, softmax, and hidden-state aggregation in a single fused operation. Residual updates and output RMSNorm are folded into the same kernel where supported, reducing intermediate memory traffic and kernel-launch overhead in both prefill and decode. Stable LatentMoE: quantile-balanced latent-space experts at 16-of-896 sparsity What's new. LatentMoE https://research.nvidia.com/labs/nemotron/LatentMoE/ , introduced by NVIDIA, projects dispatched token activations into a narrower latent dimension for routed-expert computation, then projects the combined expert output back to the model width—reducing expert-weight bandwidth and all-to-all traffic so more experts can be used at similar inference cost. Kimi K3's Stable LatentMoE https://www.kimi.com/blog/kimi-k3 scales this design to 896 experts with 16 active per token and uses Quantile Balancing https://kexue.fm/archives/11619 to derive expert allocation from router-score quantiles instead of heuristic balancing updates. How vLLM serves it: Experts are sharded with expert parallelism. vLLM offers two MoE backends tuned for different topologies: TRT-LLM-Gen for tensor-parallel TP 1 and MegaMoE for disaggregated/expert-parallel DEP . It also supports optional Expert-Parallel Load Balancing EPLB to ensure each rank has a similar amount of compute. The weights execute natively in MXFP4 on the MoE path. Chat template: a render program, not a Jinja template What's new: Kimi K3's chat template must encode system, user, and assistant messages, multimodal content, tool definitions, and tool results using exact control tokens. Instead of the common approach of a Jinja chat template https://huggingface.co/moonshotai/Kimi-K2.7-Code/blob/main/chat template.jinja that renders the request as text before tokenization, Kimi K3 uses a Python program to build the prompt token sequence directly. Its output likewise contains distinct regions for reasoning, answer text, and tool calls that must be parsed into an API response. How vLLM serves it: vLLM implements both the input renderer and streaming output parser in its Python and Rust frontends, preserving control-token boundaries while treating user-supplied and tool-supplied text as ordinary content. For tool calls and structured outputs, vLLM integrates Kimi K3's format with XGrammar https://xgrammar.mlc.ai/ so structured regions are constrained during decoding and returned as separate reasoning, content, and tool-call fields. Built for production Serving a 2.8T hybrid MoE well means being fast for each user, efficient for many concurrent sessions, and scalable for agents. vLLM ensures Kimi K3 is ready on all three. Ultra-low latency: speculative decoding with DSpark To reach ultra-low latency on a 2.8T-parameter model like Kimi K3 without accuracy loss, speculative decoding is the natural choice. That is why vLLM supports DSpark, a state-of-the-art speculative decoding algorithm, from day 0—and why we trained and released a DSpark speculator https://huggingface.co/Inferact/Kimi-K3-DSpark for Kimi K3. The draft model is trained with vLLM using TorchSpec https://github.com/lightseekorg/TorchSpec to achieve full numerical parity between speculator inference and training. DSpark uses a block-diffusion backbone to generate multiple speculative tokens in one parallel pass based on Kimi K3's rich intermediate states, so drafting cost stays flat as the block deepens. A low-rank Markov head supplies the intra-block dependency, and a confidence head predicts the likelihood for each draft to be accepted. We made the draft MLA-native, mirroring Kimi K3's own attention, so draft and target share a similar KV layout to be maximally compatible with advanced KV management and P/D-disaggregated setups. Kimi K3 DSpark positional acceptance rates across various datasets. With DSpark, we achieve a 3.14× speedup on a single-user request, from 118 tok/s to 370 tok/s, measured using SPEED Bench. We also benchmarked the acceptance rate and speedups on different tasks, with the results shown above. For coding and other low-entropy tasks, we achieve around 4.73 accepted tokens per step. For high-entropy tasks such as creative writing, we achieve around 2.61 accepted tokens per step. Confidence-based scheduling with DSpark is an ongoing effort in vLLM. Once enabled, it uses the confidence head included in the DSpark model to predict how likely each drafted token is to be accepted, prioritizing strong proposals and pruning weak ones so verification is not spent on tokens that will not survive. Both the draft model https://huggingface.co/Inferact/Kimi-K3-DSpark and the inference support are open source as of this release. See the deployment guide below to enable it. A lightweight DSpark draft proposes candidate tokens that Kimi K3 verifies in a single parallel pass, accelerating single-stream decode. Sequence parallelism for TEP prefill Sequence parallelism shards token ownership across ranks; the attention residual is applied per shard, and one all-gather rebuilds the full batch before the next layer's QKV projection. For the prefill phase, we combine attention tensor parallelism with MoE expert parallelism TEP . Compared with pure TP, TEP reduces communication overhead and keeps whole experts on each rank, yielding more efficient expert GEMM shapes. However, the naive TEP implementation performs two all-reduce operations per layer—one after the attention output projection and one after the MoE—so every rank materializes the full batch and redundantly applies the attention residual to all of it. To address this, we implement sequence parallelism https://arxiv.org/abs/2205.05198 : the all-reduce after o proj is replaced with a reduce-scatter so that each rank owns a shard of the tokens, the attention residual is applied per shard, the MoE's all-to-all performs dispatch and combine, and a single all-gather restores the full batch before the next layer's QKV projection. This design provides two key advantages: Reduced communication overhead: Reduce-scatter + all-to-all dispatch + all-to-all combine + all-gather are theoretically cheaper than two all-reduces. In practice, however, NCCL's reduce-scatter and all-gather are not optimized for prefill's message sizes and yield no speedup. We therefore implement custom reduce-scatter and all-gather kernels that are 1.7×–4.5× faster than NCCL, especially at small-to-medium message sizes. Sharded attention residual: The attention residual stays sharded across ranks throughout the layer, so each rank computes and maintains only its shard of the tokens rather than the entire batch. This matters especially for Kimi K3, where AttnRes turns the residual stream into persistent cross-layer state with its own compute and memory footprint. Sequence parallelism is enabled by default when appropriate: when using TP with the MegaMoE kernel, or when combining TP + DP + EP. No extra flags are needed. Large-scale serving: prefill/decode disaggregation For high-throughput settings, vLLM serves Kimi K3 with expert and data parallelism across nodes and with prefill/decode PD disaggregation, which runs prefill-heavy and decode-heavy work on separate replicas so each is sized for its own bottleneck. One of our validated topologies routes TEP8 prefill to DEP16 decode, with NIXL as the KV transfer engine. PD disaggregation is unforgiving for a hybrid model: the recurrent KDA state, the full-attention paged KV, and the block tables all have to arrive correctly. The NIXL connector treats the shared KV-cache page as two logical views: token-level MLA cache and request-level KDA state, including convolution and recurrent state. During the handshake, it exchanges the MLA/KDA metadata, then builds separate transfer descriptors for each transfer. Under heterogeneous TP, vLLM's hybrid allocator uses different block sizes for prefill and decode. To support that case, vLLM's NIXL connector tracks the logical-to-physical block mapping and zeroes any untransferred tail regions, preventing stale data from previous requests from leaking through padding or layout gaps. Prefill TEP8, 8 GPUs, ≈350B params/GPU hands the KV, KDA state, and block tables to a separate decode tier DEP16, 16 GPUs, ≈175B params/GPU over NIXL. Reconciling partial block cache hits and KV cache offloading As described in the Kimi K3 preview https://vllm.ai/blog/2026-07-22-kimi-k3-preview , vLLM supports fine-grained prefix hits that may end inside a physical cache block. This introduces a subtle challenge for KV offloading: vLLM may first find a local GPU hit with a partial tail, then discover a longer prefix in an external store such as Mooncake. With full-block hits, remote reuse extends cleanly beyond the local prefix. A partial tail, however, can overlap with the remote result. The vLLM scheduler therefore compares the exact reusable token lengths from both tiers and selects the longer prefix. If the remote hit wins, it releases the block reserved for the shorter local tail and reconciles all cache groups to the new prefix length. Importantly, we built this mechanism entirely through the existing KV Connector APIs, which already provide all the required semantics. This allows MooncakeStoreConnector , SimpleCPUOffloadConnector , and other connectors to support multi-tier partial-prefix reuse without model-specific integration paths. The design is tracked in the RFC https://github.com/vllm-project/vllm/issues/45702 and implemented across PR 45939 https://github.com/vllm-project/vllm/pull/45939 , PR 46384 https://github.com/vllm-project/vllm/pull/46384 , and PR 49502 https://github.com/vllm-project/vllm/pull/49502 . Agentic serving: smarter cache retention policies Kimi K3's linear-attention layers require only a constant-size KDA state, making them memory-efficient at long context lengths. A single layer's KDA state is roughly equivalent to the MLA cache for a few thousand tokens. Although large, this state does not grow with sequence length, unlike a conventional KV cache. That distinction becomes significant for agentic workloads spanning hundreds of thousands to one million tokens. The same design complicates prefix caching. KDA state is updated in place during decoding, so vLLM must copy the state at a selected prefix boundary before the next forward pass overwrites it. Caching at every token position would be prohibitively expensive: each KDA checkpoint is much larger than one token's MLA cache and would quickly exhaust even a distributed cache pool. To improve cache-space efficiency while preserving useful prefixes, vLLM supports two complementary retention policies. Interval-based retention Caching every KDA state is wasteful, but caching too sparsely forces the next request to recompute a large suffix. Interval-based retention balances these costs by treating selected positions as checkpoints—for example, one every 32K tokens. Prompt boundaries are even better checkpoints. In agentic workloads, the next turn usually begins by replaying the previous turn's prompt, so the state at the end of that prompt is especially likely to be reused. vLLM detects and retains these boundaries automatically. Users can control periodic checkpointing with VLLM PREFIX CACHE RETENTION INTERVAL . Setting it to 0 disables periodic checkpoints and retains only prompt-end states, which is a good fit for workloads dominated by multi-turn conversations. Larger intervals trade some recomputation for lower cache usage. Interval-based retention was introduced for DeepSeek V4 and hybrid sliding-window-attention models in PR 43447 https://github.com/vllm-project/vllm/pull/43447 , with day-0 support for Kimi K3 and hybrid linear-attention models added in PR 45845 https://github.com/vllm-project/vllm/pull/45845 . Interval-based cache retention. MLA caches KV for every block, while a KDA state is kept only at checkpoints: prompt ends green are always retained, and fixed-interval checkpoints orange are configurable. Marconi-style selective retention Prompt-end retention works well for conversational state, but valuable shared prefixes can appear elsewhere. A system prompt, repository snapshot, or tool specification may be reused across many requests without aligning with a prompt boundary. Marconi-style retention MLSys '26 https://mlsys.org/virtual/2025/poster/3260 handles these cases with a simple rule: cache on the second hit. The first observation provides evidence that the prefix exists; the second shows that it is actually shared. Only then does vLLM spend cache capacity on its KDA state. This turns retention into a demand-driven decision. One-off prefixes do not crowd the cache, while recurring prefixes are promoted automatically—without requiring users to predict which parts of their workload will become hot. Selective retention was introduced in PR 37898 https://github.com/vllm-project/vllm/pull/37898 , with day-0 Kimi K3 support added in PR 47782 https://github.com/vllm-project/vllm/pull/47782 . Selective cache retention. Request 1 keeps a KDA state only at its own prompt end, past the shared prefix, so request 2 gets a KV hit but a KDA miss. That second sighting is evidence the prefix is shared, so a state is cached at the prefix boundary, and request 3 reuses it. Together, the policies cover both predictable and emergent reuse: interval retention checkpoints structurally important boundaries, while Marconi-style retention learns which other prefixes are worth keeping. Performance optimizations Serving large models like Kimi K3 brings its own challenges because of its size. The entire model can barely fit in a single NVIDIA DGX B300 and requires a minimum of 16 NVIDIA B200/GB200 GPUs to serve on that hardware generation. Serving must trade off interactivity against total system throughput: tensor parallelism is good for interactivity but offers low overall throughput because effective KV cache size is limited, while large-scale expert parallelism can limit per-user output-token speed because of network-bandwidth bottlenecks. Here we highlight optimizations that improve performance in both cases so users can choose the recipe that suits their workload. Many of these optimizations are already covered in our preview blog https://vllm.ai/blog/2026-07-22-kimi-k3-preview . Attention Residuals Kimi K3 uses Block AttnRes, attending over up to eight cached block representations plus the current within-block residual. For each token, vLLM computes logits from RMS-normalized sources, applies softmax across these depth-wise candidates, and aggregates their representations. Its implementation resembles FlashAttention's online-softmax strategy but operates across model depth rather than sequence positions and has at most nine sources. vLLM performs this mixing in a single fused kernel, incorporating the residual update at the input and optionally applying RMSNorm to the output. A portable Triton implementation covers the general path, while a specialized CUDA kernel accelerates supported Blackwell configurations. KDA decode The fused KDA decode kernel folds the causal convolution, recurrent update, and RMSNorm into a single launch instead of a chain of separate kernels. A KDA layer involves many operations: input projections, causal 1D convolutions, QK norm, gate computation, KDA recurrent update, and output gated RMSNorm. On supported configurations, vLLM fuses the post-projection decode path—from the causal convolutions through gated RMSNorm—into a single specialized CUDA kernel. The kernel updates the convolution and recurrent states in place and writes the normalized output directly, avoiding intermediate tensors, repeated state traffic, and per-operation launch overhead across Kimi K3's many KDA layers. Portable Triton fallback paths cover unsupported configurations. KDA prefill KDA prefill became one of our favorite examples of open-source development in practice. Moonshot AI first released FlashKDA https://github.com/MoonshotAI/FlashKDA , a high-performance CUTLASS implementation of KDA. We quickly integrated it into vLLM and worked through less glamorous production details: broader GPU coverage, metadata dtypes, tensor layouts, and reliable vendoring. Shikhar Mishra https://github.com/Itssshikhar then optimized the kernels for H100 and published Flash-Flash-KDA https://github.com/Itssshikhar/Flash-Flash-KDA , improving data movement while preserving numerical correctness. Within a day, we validated the improvements on GB300 NVL72, refined the recurrence pipeline and synchronization, and folded them into our FlashKDA integration. The result was not a one-way handoff, but a continuous loop in which an open kernel was extended by the serving community, improved by an independent contributor, and quickly carried into production. KDA metadata builder During Kimi K3 DSpark bring-up, KDA metadata preparation emerged as a significant source of overhead. Kimi K3 initially reused the generic GDN metadata builder, which prepared FLA metadata that K3 does not consume and used sequences of small eager PyTorch operations to assemble and stage GPU metadata. We introduced a dedicated Kimi K3 KDA metadata builder that prunes the unused paths and replaces those operation sequences with fused Triton kernels, reducing each sequence to a single launch. At batch size 1, this reduced metadata-preparation latency by 96%, from 870 μs to 34 μs, and improved end-to-end DSpark latency by 6%. Low-latency BF16 GEMM In low-batch-size, latency-sensitive settings, we replace generic BF16 GEMM—used in several linear projection layers—with our own skinnyGEMM implementation. Generic cuBLAS kernels do not achieve the best performance here because they are optimized for more general shapes. In the kernel, we bypass shared-memory data staging, load activations and weights directly into registers, and use CUDA Core FMA instructions to perform the math. This avoids the heavy TMA and Tensor Core setup phase used to achieve maximum throughput. Our microbenchmarks show kernel-level speedups ranging from 8% to 100% and an end-to-end latency reduction of about 10% in small-batch settings. Low-latency MoE tail fusion The LatentMoE tail optimization replaces two all-reduces, RMSNorm, latent up-projection, and an elementwise add with three kernels to reduce compute and better overlap communication and computation. vLLM uses a novel strategy to reduce latent-MoE tail latency in ultra-low-latency serving. At the end of LatentMoE, the reduced activation from routed experts must be normalized with RMSNorm and up-projected before it is added to the shared-expert output. In the normal TP case, this requires two all-reduces on the routed and shared experts—or one all-reduce with concatenation—and replicates the up-projection. To avoid redundant compute in the replicated linear projection, vLLM instead performs reduce-scatter on the shared experts and keeps all-reduce on the routed experts because their activations need to be normalized. The replicated routed-expert activation then performs matrix multiplication with the up-projection in a column-parallel fashion and is added elementwise to the already-sharded shared-expert output. Finally, the results are all-gathered onto each rank using broadcast. We observe about a 20% latency reduction in this step and about a 7%–8% end-to-end speedup. Quality and Performance Benchmarks Accuracy and correctness evaluation vLLM takes accuracy as seriously as speed. We validated Kimi K3 end to end through a served OpenAI-compatible endpoint, with exact configurations in the recipes, and it passes the accuracy evaluations cleanly. At the maximum reasoning-effort level, Kimi K3 on vLLM scores 0.976 on GSM8K, 0.939 on GPQA-Diamond, 0.889 on OCRBench, and 0.818 on MMMU Pro Vision. One caveat worth knowing for evaluation: Kimi K3 thinks a lot before it answers. A low score is more often a truncated answer than a wrong one, so increase the reasoning effort, set max tokens generously, and check for cut-off generations before debugging anything else. Serving performance Kimi K3 decode throughput at batch size 1, measured on GB300 NVL72 GPUs in TP8 and TP16 configurations. At launch, vLLM achieves 111 tok/s per user on TP8 and 118 tok/s per user on TP16 at batch size 1. DSpark speculative decoding boosts interactivity by roughly 3×, reaching 331 tok/s per user on TP8 and 370 tok/s per user on TP16. We also present initial Pareto-frontier performance results for serving Kimi K3 on GB300 NVL72 across a range of scenarios, from high-throughput serving at 2K+ TPGS to low-latency serving at 100+ TPS/user. Reproduce our benchmark Here are the full recipes to reproduce the decode throughput numbers above for TP8 with DSpark: Full recipes, including multi-node, expert-parallel, and vision configurations, are in the Kimi K3 recipes https://recipes.vllm.ai/moonshotai/Kimi-K3 . Important Deployment Tips Prefix caching: --enable-prefix-caching turns prefix caching on. Prefix caching is typically enabled by default in vLLM, but it is currently disabled by default for Kimi K3 while the hybrid-cache design continues to evolve. Pass the flag explicitly. Tool calling: Validate on your own traffic before depending on it. We've occasionally seen K3 emit a tool-call format its own parser does not expect, yielding an empty tool calls result, while clean probes on the same setup parse perfectly. It is prompt- and run-dependent, not a blanket failure, but production agents should validate against your schema, retry or fall back when tool calls comes back empty, and consider strict or structured tool calling, which constrains the output grammar during generation. All-to-all backend: --all2all-backend determines how the MoE backend communicates during expert parallelism. Use flashinfer nvlink one sided for NVIDIA NVLink and deepep v2 for RDMA. MoE backend: vLLM has several MoE backends for different scenarios. We recommend deep gemm mega moe for any DEP environment and flashinfer trtllm for TP 1. Rust frontend: Set VLLM USE RUST FRONTEND=1 to enable the Rust frontend, which fully supports this model. ViT parallelism: --mm-encoder-tp-mode=data is enabled by default. K3's vision encoder has head size=12 , which cannot be sharded evenly under TP=8. Because K3's vision encoder has fewer than 1B parameters while the backbone has about 2T, we enable ViT DP by default to avoid all-reduce overhead from the encoder. Kimi K3 vLLM FAQ How many GPUs do I need to serve Kimi K3? At least one 8× B300 or GB300 NVL72 node is required; 16× B200 is also supported. Most production deployments run multi-node with expert and data parallelism, connected over RDMA or NVLink. How do I enable DSpark speculative decoding? Add: It roughly triples single-stream decode on reasoning and coding workloads. Which MoE and all-to-all backend should I use? Use deep gemm mega moe for disaggregated or expert-parallel DEP deployments and flashinfer trtllm for TP 1. Choose the all-to-all backend to match your interconnect: flashinfer nvlink one sided for NVLink and deepep v2 for RDMA. Does Kimi K3 support prefix caching, and is it on by default? It supports prefix caching over both full-attention KV and recurrent KDA state, but it is not enabled by default, so pass --enable-prefix-caching . Does vLLM support Kimi K3 on AMD GPUs? Yes. ROCm support ships at launch, with broader tuning on the roadmap. How is this different from the Kimi K3 preview post? The preview https://vllm.ai/blog/2026-07-22-kimi-k3-preview is the architecture and kernel deep dive, including how KDA prefix caching and the kernels are built. This post is the practical launch guide and includes the artifacts: how vLLM adapts to Kimi K3, recipes, flags, performance, and what Kimi K3 is ready for in production. Roadmap and Future Work RL support for Kimi K3: vLLM rollout support has already been added. We will work closely with RL ecosystem projects to support end-to-end RL training for Kimi K3. Continuous performance improvement: continue improving performance after day 0. Decode Context Parallelism DCP : our prototype shows good speedup, and we will soon upstream the support. Early experiments show 40% higher throughput than TP8 under selected workloads. Expert-Parallel Load Balancing EPLB : improve EPLB performance. Confidence-based scheduling: use the confidence head in DSpark to prune the number of draft tokens to verify. Broader AMD ROCm tuning. Quick links Model: moonshotai/Kimi-K3 https://huggingface.co/moonshotai/Kimi-K3 DSpark draft: Inferact/Kimi-K3-DSpark https://huggingface.co/Inferact/Kimi-K3-DSpark Recipes and Docker images: recipes.vllm.ai/moonshotai/Kimi-K3 https://recipes.vllm.ai/moonshotai/Kimi-K3 Kimi K3 technical blog: kimi.com/blog/kimi-k3 https://www.kimi.com/blog/kimi-k3 vLLM design for K3: the preview post https://vllm.ai/blog/2026-07-22-kimi-k3-preview Acknowledgements Thank you to Moonshot AI for creating K3, sharing the architecture ahead of release, and co-designing the KDA-aware caching; to the Inferact team for the end-to-end vLLM integration and deployment validation; to NVIDIA for the fused KDA decode, KDA prefill, and Attention Residual kernels and the MXFP4 MoE collaboration; to AMD for ROCm bring-up; to our inference partners, including Alibaba Cloud, Baseten, DigitalOcean, and Modal; to Shikhar for Flash-Flash-KDA; and to the vLLM community. The cache infrastructure built for Kimi K3 now belongs to every hybrid model with a similar architecture. We can't wait to see what you serve.