NVIDIA's Switchyard Matters More Than Its New Model

NVIDIA released Nemotron 3.5 Lightning, a 30B-parameter mixture-of-experts model with 3B active parameters, and NeMo Switchyard, an open-source routing library, on August 11. The router, which dynamically selects models per request or step, is the more significant release as it enables cost-efficient heterogeneous inference for agent workloads. NVIDIA claims up to 4x faster output and 30% faster agentic task completion, but these figures are based on its own testing.

AI https://sourcefeed.dev/c/ai Article NVIDIA's Switchyard Matters More Than Its New Model Nemotron 3.5 Lightning is a solid agent workhorse, but the open-source router is what changes agent economics. Priya Nair https://sourcefeed.dev/u/priya nair NVIDIA shipped two things on August 11: Nemotron 3.5 Lightning https://blogs.nvidia.com/blog/nemotron-lightning-switchyard-rtx-dgx/ , a 30B-parameter mixture-of-experts model with just 3B active parameters, and NeMo Switchyard, an open-source library that routes agent traffic across models mid-task. The model will get the headlines. The router is the part that changes how you'll build agents. Another small MoE — the shape is right, the numbers are NVIDIA's Lightning's design is exactly what the last year of agent deployments has been asking for: a sparse MoE small enough to run on a single GPU NVIDIA lists GeForce RTX 5090, DGX Spark, and Jetson , with only a tenth of its weights active per token. That's the same bet OpenAI made with gpt-oss-20b and Alibaba with its small Qwen MoEs — activate little, decode fast, and treat the model as a high-volume worker rather than a generalist. NVIDIA trained it with multi-token prediction and what it calls harness-optimized training for agent frameworks, ships NVFP4 and BF16 checkpoints, and released it under the permissive OpenMDW-1.1 license on Hugging Face https://huggingface.co/nvidia , ModelScope, OpenRouter, and build.nvidia.com. It runs in vLLM, SGLang, TensorRT-LLM, and the llama.cpp/Ollama/LM Studio local stack from day one — a broader launch matrix than most open releases manage. The performance claims deserve the standard vendor discount. "Up to 4x faster output" and "30% faster agentic task completion" are measured against unnamed "models in its class," and the headline benchmark — 86% on PinchBench while completing 10,000 tasks 30% faster than Qwen3.6 35B — is NVIDIA's own testing. SiliconANGLE noted the same thing: every number in this launch traces back to NVIDIA or its partners. None of that makes the claims false, and the 3B-active architecture makes the throughput story mechanically plausible. But wait for third-party numbers before you rip anything out of production. There's also a quieter release worth flagging: Nemotron-RL-Agentic-Terminal-Pivot, an open RL dataset for post-training coding agents. NVIDIA keeps publishing the training data alongside the weights, which is more than most "open" labs do and matters if you actually plan to fine-tune. Switchyard: routing goes from research idea to infrastructure Model routing isn't new. LMSYS published RouteLLM in 2024, Martian and Not Diamond built startups on it, and OpenRouter has offered auto-routing for years. What's been missing is a neutral, open, production-grade implementation that slots into existing stacks — and that's precisely what Switchyard https://github.com/NVIDIA-NeMo/Switchyard is. It's a Rust proxy and SDK that accepts OpenAI, Anthropic, and Responses API requests, translates between formats, and routes each call — per request or per step within a multi-turn task — to whichever model a configurable policy selects. Your agent code doesn't change; the gateway decides. The routing algorithms are the interesting engineering. Three work without tuning: an LLM-classifier router that judges each request and keeps session affinity, a stage router that inspects recent tool activity to guess how much capability the next step needs, and an escalation router that starts every conversation on the cheap model and upgrades when it detects difficulty. A fourth, the prefill router, reads the model's residual stream to estimate query complexity — a genuinely novel trick to see in shipping infrastructure rather than a paper. Why is NVIDIA, of all vendors, giving this away? Because routing is the argument for heterogeneous inference. If the pattern of "small local model 90% of the time, frontier API when it matters" becomes standard, most tokens get generated on hardware you own — RTX workstations, DGX boxes, Jetson devices. NVIDIA is commoditizing the model layer to sell the layer underneath it. That's a familiar strategy, and here it happens to align with what developers actually want. The trade-off nobody should skip past NVIDIA's headline claim is frontier-level accuracy at roughly one-third the task cost of running Claude Opus 4.8 alone. The partner data is more instructive because it includes the losses. LangChain https://www.langchain.com/ ran 145 multi-turn Deep Agents tasks routing between Lightning and Opus 4.8 with the escalation router: 74% cost reduction, only 7% of calls hitting the frontier model — and a roughly 6-point accuracy drop. Cognition routed between Opus 5 and Kimi K2.7 on FrontierCode Main and landed within 2.8 points of Opus 5 at about 28% lower mean cost. Six points is not noise. For an internal support agent, it's a great trade. For a coding agent whose output merges to main, it can be the difference between a PR you review and a PR you revert. Routing also adds real operational cost: your evals now need to record which model answered each step, failures become harder to reproduce because the model mix varies run to run, and a mid-task model swap throws away KV cache and any prompt tuning you did for one model's quirks. The router is a new component that can itself be wrong. How to actually try this The low-risk path: keep your current stack, put Switchyard behind your gateway — LiteLLM is adding it as a proxy plug-in, so if you're already on LiteLLM the integration is a config change — and start with the escalation router, which is the easiest policy to reason about and audit. Pull Lightning onto a 5090 or DGX Spark as the cheap arm; the NVFP4 checkpoint exists specifically for that. Then run your existing eval suite twice, routed and unrouted, and look at the accuracy delta per task type before you look at the cost savings. CodeRabbit reportedly trained a custom router for $85 in about two hours using NVIDIA's auto recipe, which suggests tuning the policy to your own traffic is cheap enough to be worth doing early rather than eventually. My read: Lightning is a competent entry in an increasingly crowded small-MoE field, and its benchmarks need independent confirmation. Switchyard is the durable contribution. The economics of agents are shifting from "which model do we standardize on" to "what's our mix, and who decides per step" — and NVIDIA just shipped the first credible open answer to the second question. Sources & further reading - NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI https://blogs.nvidia.com/blog/nemotron-lightning-switchyard-rtx-dgx/ — blogs.nvidia.com - NVIDIA Nemotron 3.5 Lightning Delivers Fast, Accurate Specialized Task Execution for Long-Running Agents https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/ — developer.nvidia.com - Route AI Agents Across Models with NVIDIA NeMo Switchyard https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/ — developer.nvidia.com - Nvidia releases Nemotron 3.5 Lightning and NeMo Switchyard to give enterprise AI capability options https://siliconangle.com/2026/08/11/nvidia-releases-nemotron-3-5-lightning-nemo-switchyard-give-enterprise-ai-capability-options/ — siliconangle.com - NVIDIA-NeMo/Switchyard https://github.com/NVIDIA-NeMo/Switchyard — github.com Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.