Nvidia Flare Local Inference

Nvidia has published Flare, a local inference server for its SM120/SM121 GPUs aimed at long conversations, coding agents and repeated tool use on workstations and compact systems such as DGX Spark. Flare splits the stack in two: Ignition, a build-time pipeline using PyTorch, Transformers and FlashInfer/b12x to load models, tune kernels, compile forwards and capture CUDA graphs, and Helios, a Rust runtime that serves the resulting package without Python. Nvidia positions Flare as a narrower alternative to vLLM, trading model, hardware and deployment breadth for a focused serving contract in which supported recipes and prepared capacities determine what can run.

Coding agents: skip this README and read AGENTS.md https://github.com/NVIDIA/flare-local-inference/blob/main/AGENTS.md directly. Flare is a local inference server for NVIDIA SM120/SM121 GPUs, built for long conversations, coding agents, and repeated tool use. Its goal is to make the memory, bandwidth, and compute in workstations and compact systems such as DGX Spark go further, while keeping generation responsive. The design rests on a few principles: - Embrace the standard PyTorch ecosystem at build time; serve from Rust. Ignition uses PyTorch, Transformers, and FlashInfer/b12x to load models, tune kernels, compile forwards, and capture CUDA graphs into a runtime package. Helios, the Rust runtime, serves that package without Python. Compilation and graph capture finish before serving. - Build on the model ecosystem. Reuse canonical model implementations wherever their dataflow fits, and specialize the parts that need different layouts or execution. Model development keeps the tools and libraries that make the Python ecosystem productive. - Make memory and state explicit. Plan buffers and execution capacities ahead of time. Treat conversation state—including recurrent state alongside KV caches—as a whole, so reuse, branching, and resuming preserve it correctly. - Keep the runtime small and shared. Model recipes own model computation; the runtime owns scheduling, cache storage, and request handling. A focused hardware and kernel stack keeps those boundaries manageable. vLLM https://docs.vllm.ai/en/latest/ covers a much broader range of models, hardware, and deployment patterns. Flare trades that breadth for a focused serving contract around selected models and local workloads. That focus lets model preparation, kernels, memory management, and scheduling be designed together. It also means supported recipes and prepared capacities determine what you can run. Think of Flare as a specialized option when its hardware and model coverage fit your machine. Rust gives the serving runtime explicit resource ownership; performance still depends on the GPU work, memory traffic, and scheduling. vLLM also uses CUDA graphs https://docs.vllm.ai/en/latest/design/cuda graphs/ . Compare the engines on your actual model and workload. Start with model preparation and serving https://github.com/NVIDIA/flare-local-inference/blob/main/docs/model-preparation.md . See the CLI guide https://github.com/NVIDIA/flare-local-inference/blob/main/docs/cli-design.md for configuration and the core design https://github.com/NVIDIA/flare-local-inference/blob/main/docs/core-design.md for the architectural reasoning.