Day-0 Support for Flint on vLLM

vLLM supports Recursive’s Flint from day 0. Both its NVFP4 and BF16 checkpoints run on a single GPU, and a single request decodes at about 500 tokens/s on one GB200, even with 100K tokens of context.

Flint is a 40B-parameter MoE that activates only about 2.5B parameters per token, with sliding-window attention in most layers and a 262K-token context. It ships as two separate checkpoints: NVFP4 (quantization-aware trained) and BF16.

Quick start #

vllm serve RecursiveSI/flint-nvfp4-gguf \
  --hf-config-path RecursiveSI/flint --tokenizer RecursiveSI/flint \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --reasoning-parser glm47 --exclude-tools-when-tool-choice-none

vllm serve RecursiveSI/flint \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --reasoning-parser glm47 --exclude-tools-when-tool-choice-none

Model architecture #

Flint is a 49-layer pre-norm MoE with hidden size 2048. What sets it apart is how little of it looks at the whole context.

Figure 1: Flint’s layer stack and its two sublayers. Gold marks what differs from a standard pre-norm MoE. Open it full-screen.

• Mostly local attention. 35 layers attend to a 513-token sliding window with RoPE; the other 14 attend to the full context with no positional encoding. Only those 14 keep KV for the whole sequence, so the cache costs 28 KiB per token.
• Learned attention sinks , GQA with 32 query and 4 KV heads, and per-head Q/K RMSNorm.
• 256 experts, sigmoid top-6, plus a shared expert. Layers 0 and 1 use a dense MLP instead.

The NVFP4 checkpoint is a GGUF file whose blocks are NVIDIA’s NVFP4 format in a different byte order. vLLM repacks it losslessly at load time and runs it weight-only (W4A16), with BF16 activations and KV cache, as Recursive’s reference implementation does. Every Flint kernel follows the reference’s rounding points, and accuracy matches it on GSM8K, GPQA-Diamond, long-context MRCR and a tool-calling suite.

Optimizations #

Our first implementation, built from vLLM’s stock layers, was slow at small batch sizes: a decode step ran 1,096 kernels, many of them tuned for much larger batches. We replaced them with Flint-specific kernels one change at a time, checking each against the reference.

Figure 2: Model-forward latency on one GB200 after each change, for batch-1 decode at 100K context and an 8,192-token prefill. Toggle the checkpoint, or show the table for every shape. Open it full-screen.

• Fused Q/K norm, RoPE and KV-cache write (−8%): one kernel instead of three.
• Small-batch MoE kernels (−27%): router and expert GEMVs that read only the selected experts’ weights. Also the largest prefill gain.
• Small-batch dense GEMV (−25%) on the same packed NVFP4 weights as the large-batch kernel.
• Norm and SiLU kernels (−10%) that follow the reference’s rounding.
• Decode attention (−5%) with sinks and sliding windows, in CuTe DSL.
• Low-latency decode path (−14%), shown below.

For up to 8 requests, the low-latency path runs each MoE layer as seven kernels: Q/K norm, RoPE, routing and the expert combine are fused into the neighboring GEMV kernels. Each kernel is launched with programmatic dependent launch, so it loads its weights while the previous one is still running.

Figure 3: The seven kernels of one MoE layer on the low-latency decode path.

In total, NVFP4 decode went from 4.81 to 1.78 ms (2.7×), BF16 decode from 3.53 to 2.34 ms (1.5×), and an 8K-token prefill from 207 to 145 ms (1.4×). A forward now runs 346 kernels instead of 1,096. Served end to end, batch-1 decode at 100K context takes 2.06 ms per token (about 485 tokens/s), which is the forward plus the LM head and sampling.

Usage notes #

• Flint emits GLM-4.7-style tool calls and thinking blocks, hence the glm47 parsers.--exclude-tools-when-tool-choice-none drops tool definitions from the prompt whentool_choice is"none" , matching the reference.
• The default sampling settings are temperature 1.0, top-p 0.95 and top-k 20. Reasoning counts toward max_tokens , so allow at least 32K for hard problems.
• The Flint kernels target a single Blackwell GPU (TP=1, BF16 KV cache); other configurations fall back to vLLM’s standard layers.