Soaring AI infrastructure costs and model pricing, combined with uncertain returns on investment, threaten to stall enterprise adoption.
To make enterprise AI spend a bit more manageable, Nvidia this week unveiled a new software platform that blurs the line between expensive proprietary models and open weights alternatives.
Announced alongside Nemotron 3.5-30B-A3B-Lightning, Nvidia’s latest open weights model, NeMo Switchyard is the GPU giant’s latest overture to enterprise. So what exactly is it? Well, it’s a router.
The idea is simple. Switchyard essentially functions as a proxy that sits between the inference server’s API endpoint and the models. But rather than sending every request to the same model, Switchyard can be configured to route prompts to different models in order to optimize for cost, latency, or output quality.
By routing some requests to smaller, cheaper, and potentially locally hosted AI models, Nvidia claims Switchyard can cut job completion costs by 74 percent relative to using Claude Opus 4.8 alone, albeit with an approximately six-point accuracy tradeoff.
The right tool for the job
The key metric in all of this is completion cost rather than price per token. A model might cost one-tenth as much as OpenAI’s or Anthropic’s top model, but if it requires 10x the tokens to complete the request, it isn't actually cheaper.
Certain elements of an AI workload may benefit from a larger, smarter model, but not all do.
For example, it’d be overkill to ask Claude Opus to generate a title card or summarize a website. It’ll certainly work, but it’ll also cost a fortune compared to Haiku or a locally hosted model that’s been fine tuned just for that purpose. The fewer tokens you burn on the big smart model, the less expensive your API bill is going to be. Nvidia software teams have spent the last several years developing models for this reason. The Lightning model announced this week is only its latest. The 30 billion-parameter MoE model is positioned as a low-latency, general purpose model that can either be used on its own or in conjunction with a larger, smarter model via a router like Switchyard.
The company has also developed several application-specific models. Nemotron Parse is one such example. “It’s a small model, one billion parameters, and it’s really good at one task, which is taking a PDF in and then explaining the context inside that PDF whether it’s charts or graphs or tables,” Joey Conway, senior director of AI software and models at Nvidia, explained in a recent interview with The Reg.
Many frontier models struggle with this task because PDFs are designed by humans for humans, so by off that work to task-specific models, enterprises can not only improve the accuracy of their AI apps, but also reduce costs in the process.
This all might sound familiar: It's not the first time we’ve seen model routers employed as a cost-saving measure.
Back when OpenAI launched GPT-5, ChatGPT would dynamically route prompts to different versions of the model based on their complexity.
As we wrote at the time, OpenAI’s router was likely implemented to reduce the number of compute cycles spent on mundane tasks like rewording emails to sound more professional ("not only … but also").
OpenAI wasn't alone in using routers to reduce model costs. The Wall Street Journal recently reported that AT&T has implemented a “smart router” of its own to automatically select which model to use. Switching from proprietary to open-weight models has reportedly saved the telecommunications giant between 80 and 90 percent in certain applications. Today about 25 percent of the company’s AI workloads are powered by open models. The company’s leadership expects that over the next few years that’ll climb to 70-80 percent.
The implementation challenge
While the idea of off simpler requests to smaller, cheaper-running models sounds intuitive, it’s easier said than done. Title cards and web summaries are relatively straightforward to implement. Open source chatbots like Open WebUI have supported this kind of functionality for more than a year now because it just makes sense. However, sometimes it’s not obvious when and where these task models should be used. Switchyard is Nvidia’s latest attempt to simplify this by automatically routing requests to the right model for the job. However, it’s not the only approach Nvidia is exploring.
AI agents and code assistants have the ability to work through problems and then generate skills — essentially standard operating procedures — documenting the process for future reference.
Through this iterative process, Conway suggests, agents could essentially teach themselves when and where they can get away with using a smaller, cheaper task model, and where a larger frontier model may be required.
“We’re starting to see signs of this sort of agent and subagent type workflow,” Conway said, describing how a frontier model might function as an orchestrator that farms out work to smaller models that are faster and more specialized.
It reflects the way companies are structured, he said. “We have people who are specialists and then we have people who help orchestrate that and understand the complexity of the problem.”
As an added step, it’s possible for the agents to generate training data on the fly, which could then be used to fine-tune the models to operate more efficiently.
Regardless of which approach ultimately wins out, anything that promotes enterprise AI adoption is a win for Nvidia. ®