NVIDIA Flare: a Rust server for Spark-class local inference

NVIDIA open-sourced Flare, an Apache-2.0 local inference server for SM120/SM121 GPUs including the DGX Spark's GB10, built as two binaries: a Python-side Ignition that compiles and captures CUDA graphs into a runtime package, and a Rust-side Helios that serves it without Python installed. Flare ships six launch recipes, including Nemotron 3.5 Bolt 74B-A9B 24GB on one GPU, Qwen3.8 27B in NVFP4, Kimi K3 in MXFP4-CSF on sixteen GPUs, and MiMo V2.6 Pro MOPD and Step-5 Preview at eight GPUs, with KV storage defaulting to FP8 E4M3. NVIDIA positions Flare as a specialized option alongside vLLM, and on a DGX Spark the package must be prepared on the SM121 machine itself, with a documented `[memory] reserve` of four GiB so the desktop does not starve the model's KV cache budget.

NVIDIA just open sourced Apache-2.0, hours old today Flare : a local inference server built specifically for SM120/SM121 GPUs - the RTX workstation class and, most importantly, DGX Spark’s GB10 . This is a first-party engine aimed squarely at the “one Spark on the desk” local-first market the site covers, and the architecture is the interesting part: two binaries instead of one. Two binaries: Ignition builds, Helios serves The design principle NVIDIA states plainly: use the standard PyTorch ecosystem at build time; serve from Rust. Ignition Python side: PyTorch, Transformers, FlashInfer/b12x loads your chosen checkpoint, tunes kernels, compiles the forward graph, and captures CUDA graphs into a runtime package . Helios the Rust runtime serves that package with no Python installed. All compilation and graph capture happen before serving - what serves is fixed, pre-planned memory and execution. That is a different contract from vLLM which NVIDIA explicitly names, praises for breadth, and politely positions around: “think of Flare as a specialized option when its hardware and model coverage fit your machine” - the vLLM CUDA-graph link sits right in the README . Flare’s model list is deliberately short and prepared: recipes are curated, capacities are planned, and what you can run is bounded by what has been compiled. The launch recipes Six recipes ship at announcement, and each maps to a real local-market tier the site covers: - Nemotron 3.5 Bolt 74B-A9B 24GB on one SM120/SM121 GPU the 74B/9B-active agent model NVIDIA sized for a single card, plus a LiLiCorr variant checkpoint - Qwen3.8 27B nvidia/Qwen3.8-27B-NVFP4 , text generation on one GPU - Qwen3.8 Flash and DeepSeek V4.1 Flash the alias selects deepseek-ai/DeepSeek-V4.1-Flash-1017 , the same checkpoint we track - Kimi K3 in MXFP4-CSF - the 93-layer KDA/latent-attention decoder, prepared on sixteen GPUs with DSpark speculators seven draft tokens per round - MiMo V2.6 Pro MOPD and Step-5 Preview at eight GPUs with DFlash/MTP draft paths KV storage defaults to FP8 E4M3 with per-model exceptions stated precisely Qwen Flash keeps BF16 selector keys; GLM and DeepSeek keep their packed latent formats , which is the kind of memory-planning detail that makes the “explicit state” design real. On a Spark specifically If you are doing this on a DGX Spark: prepare the package on the Spark itself. Spark’s GB10 is SM121, an SM120 package fails the architecture check, and the two are not interchangeable. The reason for the fuss is one shared pool of memory: GPU weights, KV cache, host buffers, resident embedding tables, and your own OS all draw on the same physical DRAM. Before serving, set memory reserve four GiB is the documented example so the desktop part of the machine does not starve the model’s cache budget, and let the loader account for resident embedding scales before it sizes KV. What you get for that setup discipline: a package that fits the box you actually own, with KV sized to what is really left over. For the catalog The engines catalog gains a row today: Flare NVIDIA, Apache-2.0, SM120/SM121 Linux, not OpenAI-compatible-verified in the docs we read, GUI no, repo hours old at 133 stars and growing . The honest one-word classification: Spark-specialized . It does not replace llama.cpp or MLX for desktops; it competes for the class of machine the site’s Spark cluster coverage belongs to - the exact box sitting in Niklas Lenz’s rig and the Spark build pages. Sources: NVIDIA’s flare-local-inference announcement repo Apache-2.0 + its model-preparation docs fetched live Oct 21 UTC ; model pages on HuggingFace per recipe. Related on this site: Twenty-one ways to run a model locally: the engine catalog https://tokenstead.ai/guides/local-inference-engines - Hermes Agent ships on NVIDIA’s RTX Spark launch https://tokenstead.ai/guides/hermes-rtx-spark - Niklas Lenz’s 4xGB10 Spark rack rig https://tokenstead.ai/rigs/niklas-lenz-spark-rack