NVIDIA just open sourced (Apache-2.0, hours old today) Flare: a local inference server built specifically for SM120/SM121 GPUs - the RTX workstation class and, most importantly, DGX Spark’s GB10. This is a first-party engine aimed squarely at the “one Spark on the desk” local-first market the site covers, and the architecture is the interesting part: two binaries instead of one.
Two binaries: Ignition builds, Helios serves #
The design principle NVIDIA states plainly: use the standard PyTorch ecosystem at build time; serve from Rust. Ignition (Python side: PyTorch, Transformers, FlashInfer/b12x) loads your chosen checkpoint, tunes kernels, compiles the forward graph, and captures CUDA graphs into a runtime package. Helios (the Rust runtime) serves that package with no Python installed. All compilation and graph capture happen before serving - what serves is fixed, pre-planned memory and execution.
That is a different contract from vLLM (which NVIDIA explicitly names, praises for breadth, and politely positions around: “think of Flare as a specialized option when its hardware and model coverage fit your machine” - the vLLM CUDA-graph link sits right in the README). Flare’s model list is deliberately short and prepared: recipes are curated, capacities are planned, and what you can run is bounded by what has been compiled.
The launch recipes #
Six recipes ship at announcement, and each maps to a real local-market tier the site covers:
Nemotron 3.5 Bolt 74B-A9B 24GB onone SM120/SM121 GPU (the 74B/9B-active agent model NVIDIA sized for a single card, plus a LiLiCorr variant checkpoint) #
Qwen3.8 27B (nvidia/Qwen3.8-27B-NVFP4 ), text generation on one GPU #
Qwen3.8 Flash andDeepSeek V4.1 Flash (the alias selectsdeepseek-ai/DeepSeek-V4.1-Flash-1017 , the same checkpoint we track) #
Kimi K3 in MXFP4-CSF - the 93-layer KDA/latent-attention decoder, prepared onsixteen GPUs with DSpark speculators (seven draft tokens per round) #
MiMo V2.6 Pro MOPD andStep-5 Preview at eight GPUs with DFlash/MTP draft paths
KV storage defaults to FP8 E4M3 with per-model exceptions stated precisely (Qwen Flash keeps BF16 selector keys; GLM and DeepSeek keep their packed latent formats), which is the kind of memory-planning detail that makes the “explicit state” design real.
On a Spark specifically #
If you are doing this on a DGX Spark: prepare the package on the Spark itself. Spark’s GB10 is SM121, an SM120 package fails the architecture check, and the two are not interchangeable. The reason for the fuss is one shared pool of memory: GPU weights, KV cache, host buffers, resident embedding tables, and your own OS all draw on the same physical DRAM. Before serving, set [memory] reserve (four GiB is the documented example) so the desktop part of the machine does not starve the model’s cache budget, and let the account for resident embedding scales before it sizes KV. What you get for that setup discipline: a package that fits the box you actually own, with KV sized to what is really left over.
For the catalog #
The engines catalog gains a row today: Flare (NVIDIA, Apache-2.0, SM120/SM121 Linux, not OpenAI-compatible-verified in the docs we read, GUI no, repo hours old at 133 stars and growing). The honest one-word classification: Spark-specialized. It does not replace llama.cpp or MLX for desktops; it competes for the class of machine the site’s Spark cluster coverage belongs to - the exact box sitting in Niklas Lenz’s rig and the Spark build pages.
Sources: NVIDIA’s flare-local-inference announcement repo (Apache-2.0) + its model-preparation docs (fetched live Oct 21 UTC); model pages on HuggingFace per recipe. Related on this site: Twenty-one ways to run a model locally: the engine catalog - Hermes Agent ships on NVIDIA’s RTX Spark launch - Niklas Lenz’s 4xGB10 Spark rack (rig)