{"slug": "inkling-2-small-on-vllm-day-0-support", "title": "Inkling 2 Small on vLLM: Day-0 Support", "summary": "VLLM added day-0 support for Thinking Machines' Inkling 2 Small, a multimodal text-and-vision MoE model with 594B total and 17B active parameters and up to 1M tokens of context. vLLM reports serving performance of up to 630 tokens per second per user on GB200 at tensor parallelism 4 with an acceptance length of 3.5, and also supports LoRA and reinforcement learning via SkyRL. The 48-layer model uses 384 experts with top-8 routing, mixed-precision experts (NVFP4 in layers 0–39, MXFP8 in layers 40–47) so it can be served on a single node, and a native DSpark speculator that drafts 7 tokens per step.", "body_md": "# Inkling 2 Small on vLLM: Day-0 Support\n\nvLLM supports Thinking Machines’ [Inkling 2 Small](https://thinkingmachines.ai/news/inkling-2-small/) on day 0. It is a multimodal (text and vision) MoE model with 594B total and 17B active parameters and up to 1M tokens of context. The model also comes with a native DSpark speculator. vLLM supports it with optimized performance (up to 630 TPS/user at TP4 and acceptance length 3.5) and also supports LoRA.\n\n## TL;DR\n\n- **594B total, 17B active, 1M context.** A 48-layer MoE with 384 experts (top-8) that takes images and text and outputs text.\n- **Architecture upgrade from Inkling.** Block attention residuals, attention sink, gated attention, and ViT image encoder.\n- **Mixed-precision MoE.** The MoE uses NVFP4 (layers 0–39) and MXFP8 (layers 40–47), so the model can be served with a single node.\n- **Native DSpark speculator.** DSpark with an RNN head drafts 7 tokens per step.\n- **Extensive optimizations.** Kernel fusion, PDL, highly tuned Gluon attention kernel, and sequence parallelism, achieving up to 630 TPS/user on GB200 at TP4 and acceptance length 3.5.\n- **RL and LoRA support.** vLLM supports RL (via SkyRL) and LoRA for the model.\n\nYou can serve the model with:\n\n```\nvllm serve thinkingmachines/Inkling-2-Small \\\n  --tensor-parallel-size 8 \\\n  --reasoning-parser inkling --tool-call-parser inkling --enable-auto-tool-choice \\\n  --speculative-config '{\"method\":\"dspark\",\"num_speculative_tokens\":7,\"draft_sample_method\":\"probabilistic\"}'\n```\n\n## Model Architecture\n\n*Figure 1. Inkling 2 Small architecture. Gold marks what is new or changed since Inkling.*\n\n|  | Inkling | Inkling 2 Small | \n|---|---|---|\n| Layers | 66 (55 sliding-window + 11 full) | 48 (40 sliding-window + 8 full) | \n| Residual path | residual stream | block attention residuals (8 blocks of 6 layers) | \n| Attention | GQA with relative bias, no RoPE | same | \n| Heads (query / KV) | 64 / 8 full, 64 / 16 sliding | 48 / 8 in all layers | \n| Sliding window | 512 | 1,024 | \n| Attention output | plain | sigmoid gate and per-token sink | \n| Short convolutions | 4 per layer (K, V, attention out, MLP out), kernel size 4 | same | \n| MoE | 256 routed (top-6) + 2 shared expert sinks, first 2 layers dense | 384 routed (top-8), no shared experts, all layers MoE | \n| Expert precision | NVFP4 | NVFP4 (layers 0–39), MXFP8 (layers 40–47) | \n| Modality | text, image, audio in; text out | text, image in; text out | \n| Image encoder | 4-layer hMLP on 40×40 patches | 23-layer ViT (1,024 wide) on 20×20 patches | \n| Speculative decoding | 8 chained MTP heads | DSpark draft, 7 tokens per step | \n\n**Inkling backbone + recent advances.** Inkling 2 Small keeps Inkling’s backbone: the 5:1 mix of sliding-window and full attention, GQA with a learned relative bias instead of RoPE, the four short convolutions per layer and a sigmoid-routed MoE. On top of it, it adds recent advances. [Block attention residuals](https://arxiv.org/abs/2603.15031) replace the residual stream, so before each sublayer a learned query mixes the outputs of earlier 6-layer blocks. [Gated attention](https://arxiv.org/abs/2505.06708) puts a sigmoid gate on each head’s output, together with a per-token [attention sink](https://arxiv.org/abs/2309.17453). It replaces the hMLP image encoder with a 23-layer ViT.\n\n**Mixed-precision MoE with Four-Over-Six NVFP4.** Out of Inkling 2 Small’s 48 layers, the experts of the first 40 use NVFP4 and those of the last 8 use MXFP8. The NVFP4 activations use a per-token global scale plus dual-candidate block scales ([Four Over Six](https://arxiv.org/abs/2512.02010)): for each 16-element block, the quantizer tries scaling the block’s maximum to 6 and to 4, and keeps whichever gives the lower reconstruction error (MAE).\n\n**DSpark speculator with an RNN head.** The checkpoint also ships a DSpark draft in place of Inkling’s 8 chained MTP heads. The speculator has 5 layers and drafts 7 tokens per step. The head is the RNN variant that [DeepSeek’s DSpark paper](https://arxiv.org/abs/2607.05147) described.\n\n## vLLM Integration & Optimization\n\n*Figure 2. Kernels in one decoder layer, for decode and large prefills.*\n\n**Kernel fusion and PDL.** vLLM aggressively fuses the collective communication and memory-bound ops between sublayers into a single kernel. With attention residuals, each boundary needs an all-reduce, a short convolution, a residual update, the attention-residual mix and an RMSNorm: five memory-bound steps on the same hidden states. The fused kernel, `allreduce_conv_mix` (gold in Figure 2), runs the all-reduce over NVLink multicast and keeps everything after it in registers. The GEMM, attention, routing and activation kernels launch with programmatic dependent launch (PDL), so each one’s prologue overlaps the tail of the previous kernel, which matters at small batch sizes, where a decode step is a long chain of short kernels.\n\n**MoE with fused activation quantization.** vLLM quantizes the expert activations (NVFP4 or MXFP8, matching each layer) inside the kernels that produce them, so no separate quantization pass touches the hidden states: the routing kernel quantizes the tokens before the gate/up projection, and the softcapped SwiGLU quantizes its output before the down projection. The expert GEMMs run on TensorRT-LLM’s batched-GEMM kernels, shipped with FlashInfer.\n\n**Sequence parallelism for prefill.** Under plain tensor parallelism, every GPU runs the short convolution, attention-residual mix and RMSNorm on all tokens after each all-reduce: the same work, repeated on every rank. That is cheap in decode but costly in prefill. vLLM removes the redundancy with sequence parallelism (purple and gold in Figure 2). Each GPU reduce-scatters, runs the short convolution, residual update, attention-residual mix and RMSNorm in one fused `conv_mix` kernel on only its slice of tokens, and all-gathers the result for the next GEMM. This cuts an 8K-token prefill step by 22% at TP4 and 26% at TP8, and frees 1.3–1.9 GiB per GPU for KV cache.\n\n**Optimized Gluon relative attention kernel for decode.** Inkling’s FlashAttention-4 decode path is slow for Inkling 2 Small: the 6:1 GQA ratio silently disables GQA packing. To fix this, we wrote a warp-specialized Gluon kernel for decode and verify that packs the 6 heads into one 64-row tile and reads KV once: a TMA warp streams KV, a tcgen05 warp runs the MMAs and four softmax warps apply the relative bias. A batch-1 decode step at 100K context drops from 15.1 ms to 4.8 ms. Prefill keeps Inkling’s sheared-bias FlashAttention-4 path.\n\n### Performance\n\nWe benchmarked Inkling 2 Small on an agentic workload at TP8 (two 4-GPU nodes) with DSpark (probabilistic drafting) and prefix caching. Each session keeps a long context and, on every turn, appends the previous answer and a ~2K-token tool output, so only about 3K tokens per turn are new; each turn then generates 1,024 tokens with reasoning on at temperature 1.0. The table gives output throughput per GPU at the concurrency where each user gets 100 tokens/s.\n\n| Session context | Output tok/s per GPU at 100 tok/s/user | Concurrent sessions | Prefix-cache hit rate | Acceptance length | \n|---|---|---|---|---|\n| 32K | ~1,070 | ~84 | 91% | 3.3 | \n| 64K | ~915 | ~72 | 95% | 3.4 | \n| 128K | ~700 | ~55 | 98% | 3.5 | \n| 256K | ~450 | ~37 | 99% | 3.5 | \n\nFor a single user at TP4, the model decodes at about 630 tok/s with an acceptance length of 3.5.\n\n## Accuracy Evals\n\nvLLM keeps the reference implementation’s numerics: the same data types at every step, FP32 attention output and router, and the same NVFP4/MXFP8 activation quantization, with only the reduction order and the all-reduce’s BF16 partials differing. End to end, we verified accuracy on NIAH up to 1M tokens, MMMU-Pro and HLE with [inkling-verify](https://github.com/thinking-machines-lab/inkling-verify), with DSpark enabled; the scores match our runs without speculative decoding.\n\n## Acknowledgements\n\nWe thank the Thinking Machines team for their close collaboration in integrating and optimizing Inkling 2 Small in vLLM. The work was led by Aurick Qiao (Thinking Machines) and Woosuk Kwon (Inferact).", "url": "https://wpnews.pro/news/inkling-2-small-on-vllm-day-0-support", "canonical_source": "https://vllm-project.github.io/2026/11/12/inkling2-small.html", "published_at": "2026-11-12 09:00:00+00:00", "updated_at": "2026-11-12 21:22:40.398749+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "machine-learning", "ai-tools"], "entities": ["vLLM", "Thinking Machines", "Inkling 2 Small", "DSpark", "SkyRL", "GB200"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/inkling-2-small-on-vllm-day-0-support", "markdown": "https://wpnews.pro/news/inkling-2-small-on-vllm-day-0-support.md", "text": "https://wpnews.pro/news/inkling-2-small-on-vllm-day-0-support.txt", "jsonld": "https://wpnews.pro/news/inkling-2-small-on-vllm-day-0-support.jsonld"}}