NVIDIA NeMo Switchyard Cuts Agent Costs 74% With Open Model Routing

NVIDIA released Nemotron 3.5 Lightning, a 30B mixture-of-experts model, and NeMo Switchyard, an open-source routing library, on August 11 to cut AI agent costs by routing tasks to cheaper models. LangChain's benchmark on 145 production-style tasks showed a 74% cost reduction with only 7% of calls reaching a frontier model, while Ramp reported a 58% cost cut and Boomi routed 59% of traffic to a faster fine-tuned model.

Most AI agent bills are bloated with frontier-model tokens doing work a small model handles fine. Tool calls, retries, output validation — none of that needs GPT-5-class reasoning, yet teams route everything to the top of the stack out of habit. NVIDIA shipped two open tools on August 11 to stop that: Nemotron 3.5 Lightning , a 30B mixture-of-experts model purpose-built for agent execution tasks, and NeMo Switchyard , an open-source routing library that decides per-step which model gets the call. LangChain tested it on 145 production-style tasks https://www.langchain.com/blog/switchyard-agent-routing-benchmark and cut costs 74% — with only 7% of calls ever reaching a frontier model. The Router Is the Real Story Nemotron 3.5 Lightning is a solid model. But the more interesting release is Switchyard. It sits between your agent and the model layer and routes each workflow step independently — weighing task complexity, latency budget, cost, and live workflow state — then dispatches to whichever model clears the bar most cheaply. Four routing strategies ship out of the box: LLM Classifier: A judge LLM categorizes each task’s difficulty and picks a candidate model. It maintains session affinity, so follow-on turns that haven’t materially changed don’t get reclassified from scratch. Stage Router: Reads signals the agent already produces — error loops, tool call depth, exploration patterns — to infer which phase of the workflow it’s in. No extra model call per decision. A coding agent deep in implementation gets a cheaper model; one thrashing through errors escalates automatically. Escalation Router: Start cheap by default, escalate when evidence warrants. Runs the efficient model unconditionally, then uses a judge to evaluate the response. If it’s not good enough, re-routes to the frontier model and returns that output instead. Prefill Router: Learns from your production residual streams to predict which model will succeed before committing the call. Needs historical data to train, but delivers the tightest routing decisions for teams with enough volume. The Stage and Escalation routers are the practical defaults: they work without labeled training data, reading signals your agent already generates. Switchyard accepts OpenAI, Anthropic, and Responses API formats, and it’s already wired into LangChain, LiteLLM, Kong AI Gateway, and Hermes. License is Apache-2.0. What the Benchmarks Show NVIDIA ran its own numbers — $72 vs $180 per workload when routing instead of sending everything to Claude Opus 4.8 — but the more credible figures come from partners who tested on their own production workloads: LangChain: 74% cost cut across 145 multi-turn agentic tasks. Only 7% of calls hit the frontier model. Accuracy dropped ~6 points — a tradeoff most production teams will take. Ramp: 58% cost reduction on its internal SWE-Bench variant, 33% faster runtime, and frontier-matched accuracy. No accuracy tradeoff at all. Boomi: 59% of traffic routed to a 5x faster fine-tuned model, 100% domain-routing accuracy, and 21% lower latency on later turns. The consistent finding: the vast majority of agent steps — in some cases 93% of them — don’t need a frontier model. Teams just haven’t had an automated way to enforce that until now. Nemotron 3.5 Lightning: The Execution-Layer Model The model Switchyard routes to for high-volume, low-complexity steps is Nemotron 3.5 Lightning https://developer.nvidia.com/blog/nvidia-nemotron-3-5-lightning-delivers-fast-accurate-specialized-task-execution-for-long-running-agents/ . Its architecture is a hybrid Mamba-2 and mixture-of-experts stack — 30B total parameters, 3B active per token — combined with speculative decoding NVIDIA’s D-Flash and D-Spark methods, derived from DeepSeek’s approach . That combination delivers up to 4x output speed over dense models of similar size: 115.75 tok/s on a single stream, 421.85 tok/s across 8 concurrent streams. Both BF16 and NVFP4 checkpoints are available. The BF16 version is the reference weight for fine-tuning and distillation; NVFP4 is the inference deployment target. On SWE-bench Verified, the quantized NVFP4 version scores slightly higher 52.80 vs 51.56 than BF16 — so you’re not giving anything up by deploying the faster, smaller checkpoint. License is OpenMDW-1.1: open weights, training data, training recipes, and commercial use permitted. How to Start Using It Several on-ramps exist, including a free tier: Free via OpenRouter: Model ID nvidia/nemotron-3.5-lightning:free — no query cost NVIDIA NIM API OpenAI-compatible : endpoint at https://integrate.api.nvidia.com/v1 Local deploy: Docker NIM container from NGC, or run via vLLM, SGLang, or TensorRT-LLM Hugging Face: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 For NeMo Switchyard https://github.com/NVIDIA-NeMo/Switchyard : pip install nemo-switchyard , then point it at your existing agent. Because it accepts OpenAI and Anthropic API formats natively, adding a routing layer shouldn’t require rewriting your agent code. The Actual Takeaway The standard approach — call the best available model for every agent step — is not a performance decision, it’s a defaults decision. Nobody optimized it; it’s just what happens when you wire the first API key that works. Switchyard’s benchmark data makes the cost of that laziness concrete: 60–74% overspend on tasks that a 3B-active-parameter model handles correctly 93% of the time. The Escalation Router is the right default for most teams: start cheap, escalate when the agent is clearly struggling, never pay frontier rates for formatting a tool call output. That’s not a capability tradeoff — it’s basic engineering. NVIDIA just open-sourced the infrastructure to enforce it. Read NVIDIA’s Switchyard technical blog https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/ for the full routing algorithm breakdown.