AIArticle Nemotron 3.5 Lightning is a capable workhorse, but NeMo Switchyard targets the escalation logic every agent team keeps hand-rolling.
Mariana Souza Nvidia shipped two things today, and the one getting less attention is the one that matters more. Nemotron 3.5 Lightning is a 30B-parameter open mixture-of-experts model with 3B active parameters, a 1M-token context window, and same-day availability on Amazon SageMaker JumpStart. It's a good model. But it shipped alongside NeMo Switchyard, an Apache-2.0 model router, and the pairing tells you exactly what Nvidia is doing: turning the messiest, most hand-rolled layer of every agent stack into open infrastructure — and making sure a cheap open model running on your own GPU is the default path through it.
A model built to be the cheap branch #
The spec sheet reads like it was designed backwards from a routing decision. Lightning is a hybrid of Mamba-2, attention, and MoE layers — the architecture line Nvidia has been developing since Nemotron-H — distilled from Nemotron 3 Ultra and quantized to NVFP4 so it fits on a single H100 or a DGX Spark. Nvidia claims roughly 410 tokens/second and about 4x the throughput of comparable models; take vendor throughput numbers with the usual salt, but 3B active parameters decoding with speculative decoding is going to be fast on almost any serving stack. Day-one support covers vLLM, TensorRT-LLM, and SGLang, with toggleable reasoning and tool calling built into the chat template.
The 30B-A3B shape isn't novel — Qwen3-30B-A3B staked out that exact configuration in 2025, and Lightning even uses the Qwen3 Coder parser for tool calls — but it's become the sweet spot for a reason. It's the largest model you can run comfortably on one card while keeping per-token cost close to a 3B dense model. What's changed is how much capability now fits in that envelope: Nvidia's model card reports 52.8% on SWE-bench Verified and 75.6% on GPQA Diamond. Those were frontier-class numbers in late 2024, from something you can now run on hardware that sits under a desk. Artificial Analysis scores it 24 on their Intelligence Index, up nine points from Nemotron 3 Nano.
The SageMaker JumpStart listing matters for a duller reason: procurement. A one-click deploy inside an existing AWS account skips the security review that self-hosting weights from Hugging Face often triggers at larger shops. For teams already running agents against Bedrock or per-token APIs, this is the lowest-friction way to get an owned, fixed-cost inference tier into production.
The part everyone keeps rebuilding #
Here's the layer Switchyard is aimed at. Every production agent system eventually grows the same organ: some pile of heuristics deciding which model handles which step. Send everything to a frontier model and your unit economics die; send everything to a small model and quality dies. So teams write escalation logic — try cheap, detect failure, retry expensive — and bolt on fallback chains in LiteLLM configs or hand-rolled if/else. There's real prior art in making this a product (LMSYS's RouteLLM, Martian, Not Diamond, OpenRouter's auto mode), but none of it became standard, partly because routing quality is workload-specific and partly because nobody trusted a startup's black box with the decision that determines both their costs and their accuracy.
Switchyard's design acknowledges that history. It's a provider-agnostic SDK, not a hosted service: you define a pool of models under semantic names, and the router picks per request. The tuning-free algorithms map directly onto what teams already build by hand — an LLM-classifier router, a stage router that watches recent tool errors and escalates when things go sideways, and an escalation router that starts cheap and upgrades on sustained difficulty. Kong, LiteLLM, and LangChain integrations exist at launch, and the reference server speaks OpenAI and Anthropic API formats, so it can sit in front of an existing agent without a rewrite.
The tunable option is the interesting one. The prefill router reads the model's residual stream during prefill and maps it, via a small trained MLP, to predicted accuracy for each model in the pool — then blends that with cost and latency constraints. Notice what that requires: white-box access to activations. You can't do that through a frontier API. The most capable routing mode only fully works with open weights on hardware you control, which is not an accident. Nvidia's incentive here is transparent and, for once, aligned with developers': every workload that routes 90% of its steps to a self-hosted open model is a workload buying GPUs instead of tokens.
The numbers are real, and vendor-run #
The launch benchmarks are genuinely striking. On LangChain's deep-agents evaluation — 145 multi-turn tasks routing between Lightning and Claude Opus 4.8 — the escalation router cut costs 74% versus frontier-only, with just 7% of calls reaching the big model, at a cost of roughly six accuracy points. Cognition ran Devin against FrontierCode with an Opus 5 + Kimi K2.7 pool and landed within 2.8 points of Opus-only accuracy at about 28% lower cost.
Both results come from Nvidia and its launch partners, so treat 74% as the ceiling, not your planning number. And that six-point tradeoff deserves more respect than the press materials give it. In a single-shot task, six points is a dial you turn. In a 40-step agent chain, small per-step quality losses compound, and a router that quietly degrades is worse than one that visibly fails. Adopting Switchyard means adopting the evaluation burden that comes with it: per-step traces, a regression suite that reruns when you touch the model pool, and an escalation-rate metric watched like an SLO — a 7% rate drifting to 20% is your early warning that either the workload shifted or the cheap model is out of its depth.
Where this leaves you #
If you're running agents today, the concrete play: deploy the NVFP4 weights on a single H100 (or via JumpStart), put Switchyard's escalation router in front with your current frontier model as the fallback, and shadow-route for a week before flipping traffic. What it replaces is code you probably resent maintaining anyway. The larger shift is worth naming. Routing is following the same path serving did — from every team's bespoke internal mess to shared open infrastructure, the way vLLM absorbed a hundred custom inference stacks. Nvidia giving the router away makes sense because it inverts the default: frontier models stop being where requests go and become the 7% exception you escalate to. That's good news for anyone paying per token, and quietly uncomfortable news for anyone whose business is selling them.
Sources & further reading #
[NVIDIA Nemotron-3.5-Lightning-30B-A3B-NVFP4 model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4)— huggingface.co -
[Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/)— developer.nvidia.com -
NVIDIA Nemotron 3.5 Lightning model is now available on Amazon SageMaker JumpStart— aws.amazon.com -
[NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)— github.com -
[NVIDIA launches Nemotron 3.5 Lightning](https://artificialanalysis.ai/articles/nemotron-3-5-lightning-launch)— artificialanalysis.ai -
[Nvidia's Router Is the Part of Agents Everyone Keeps Rebuilding](https://dev.to/reidmarlow/nvidias-router-is-the-part-of-agents-everyone-keeps-rebuilding-20e1)— dev.to -
[Nvidia launches a smaller, faster Nemotron model and a router to put it to work](https://thenewstack.io/nvidia-nemotron-lightning-switchyard/)— thenewstack.io
[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.