“Don’t take a Ferrari to a grocery store” is the new AI mantra. It means that you shouldn’t use the best frontier AI model for routine tasks, but delegate them to something simpler and cheaper, which will save you money. There is a more formal industry term for this — “model routing”.
Like many AI engineers these days, I have access to several models. Both Anthropic and OpenAI release not just a single model, but a whole family of them, ranging in price and capabilities. In my case, I also have a local AI set up. This is how my half manual, half hard-coded routing looked like:
Usually, during the design phase I would ask Astra to identify which tasks can be handed off to Sol or Qwen, and I admit that some manual copy-pasting of prompts still happened.
So, I wasn’t necessarily taking a Ferrari to a grocery store. But I wanted to understand whether I could optimize this setup and save more tokens. For example, defer some tasks to Terra or Luna instead of using Sol. To be fair, there might be other reasons for model routing — privacy, quality, speed, to name a few — but my main goal was efficiency and cost saving, and this is what this article is focused on.
Instead of writing this logic on my own, I decided to try a third-party model-routing tool — NeMo Switchyard from NVIDIA. In the article, you’ll find the details of my setup and the experiments I ran. But before we dive into that, I’d like to give my main conclusions upfront:
Model routing as it is today is still an advanced area. While Switchyard is a pretty decent tool, the complexity of the issue it’s solving can lead to the opposite result — spending more tokens and dollars instead of saving them.
Getting back to the Ferrari metaphor — no, you don’t need it to get to a grocery store. But choosing between a Ferrari, a bicycle, a truck, and a Toyota Corolla for each trip might not be super trivial in today’s AI world. You might save yourself a lot of trouble by simply sticking to Toyota every time.
The tool I chose to experiment with was NeMo Switchyard from NVIDIA because it satisfied my personal requirements:
Here is a quick overview of how it works. It has two main methods of classifying the task and choosing the model to route it to:
There are some combinations of these two options. For example, the LLM classifier can use a confidence threshold or a customized prompt.
The general setup with the LLM classifier looks pretty straightforward, but Switchyard has an additional built-in feature. While usually you select a model per thread or task, here the model can be changed either for each LLM request or per “user turn”.
A “user turn” doesn’t necessarily mean a new message from a human: coding agents can generate user-role messages themselves, for example when starting subagent tasks or resuming after context compaction. Interestingly enough, “per LLM call” is the default setting, while “per user turn” is the recommended one. I followed the recommended option and chose the “user turn” trigger for my experiments.
To route requests and allow users to rely on cloud subscriptions from OpenAI and Anthropic, Switchyard installs a small lightweight server that acts as a proxy, but if you use metered API calls, you can integrate with its library instead.
Two things worth noting here. First, model routing can be as complex as you wish it to be. For example, you can use your own classifier or tune NVIDIA’s Prefill Router. I didn’t venture that far into the weeds — my goal was to see if there is a quick and easy way to save tokens and money and make my overall setup simpler, not more complex.
Second, most of the Switchyard components as of November 3, 2026, when I am writing this article, are still marked as “Alpha”, “Beta”, or even “Demo”, so expect things to keep changing.
Of course, my installation consisted of a simple prompt to my coding agent asking to install and configure Switchyard on my machine. And I have to say, this part went pretty well. I use OpenCode Desktop as my main IDE and after a few minutes, I had a picker with available models in the UI, a running Switchyard server, and everything just worked.
However, the initial simplicity was misleading. My coding agent had installed everything, but figuring out whether it was making good routing decisions was a different and more difficult problem.
For example, you can see in the diagram that I set the escalation model as Sol Medium, not Astra High. Astra High is in the mix of the models to choose from for the classifier, but if a smaller model fails, you don’t want to immediately escalate to Astra — this will lead to an increase in token usage on average. I will talk more about possible issues in the “Learnings” section, but for now let’s focus on the happy path. Ideally, the setup I wanted looks like the diagram below.
However, the fact that I would love to have this setup doesn’t mean that it would lead to my desired outcome — saving me tokens and possibly dollars. So I needed to test it and analyze the results.
To test the model routing a bit more formally, I decided to run a few different configurations with two coding tasks from SWE-bench. I like these test suites since they have real repositories, real bugs, and also standard verifiers that check if the model succeeded with the task.
I didn’t want to run the whole benchmark test suite because it is too expensive, so I chose a couple of tests to focus on:
Next, I defined five different configurations, tailored to my own setup:
To summarize, I wanted to see whether I was getting any real savings with the model routing and whether delegating to Qwen helps or simply means redoing the work. And I settled on two different tests and five different configurations (10 tests total) to get the answers.
Overall, I got a total of 10 variations (2 tasks with 5 configurations each). I ran each variation three times to account for the variability of LLMs’ token usage.
In the table, you can see typical test run results for one selected attempt for each configuration and task. PASS means local patch validation, not an official SWE-bench result or a success rate. The costs are estimated because I am using a subscription, not the actual metered APIs. Also, the costs don’t include calls to the classifier model itself or the electricity and hardware costs for running my local Qwen, but the estimates do account for input/cached/output token pricing.
Overall, this is not a perfect benchmark test, but a good comparative illustration.
The main findings from the test can be summarized as follows:
So, the answer is that I can get some savings if I am very careful with when and how I use routing, and simply relying on a generic LLM to make the decisions for me won’t necessarily bring the best results.
However, my biggest learnings and “aha” moments didn’t come from running the tests. They came from trying to configure the routing, doing my normal day-to-day coding work and fixing the routing issues. This is what the next section is about.
As soon as I had Switchyard working in my IDE, I configured the agents to always use it. Both OpenCode and Switchyard collect pretty good telemetry on token usage, so I could later analyze if I am really saving tokens simply by looking back at my historical data.
Analysis of this data and all the misconfigurations I ran into is where my major understanding of this area came from. So let’s look at them one by one.
My initial setup for Switchyard was to choose Astra if the task was unclear or if a weaker model couldn’t handle the task. I noticed almost immediately that my subscription allowances started to deplete much faster as soon as I introduced Switchyard. I let it run for a few days and the results were very telling.
During those few days, Astra High accounted for 25% of the token usage and the tasks were not architecture-level work, but tasks that Sol Medium would have been perfectly capable of solving. Then I applied current pricing to my token usage to see how much I would pay with the metered API. As it turned out, Astra contributed to 75% of that estimated cost.
While OpenAI doesn’t show exact token usage for its subscription API, the local telemetry results were able to explain why my tokens were running out much faster than the previous week. When I switched from Astra to Sol Medium as a backup and escalation model, my token usage returned to the previous rate. The idea of sending a request to a small LLM sounds very simple. But in fact, there is a lot to consider. First, LLMs often struggle to estimate the complexity of the tasks and the effort required to solve them. There are multiple research papers about this, and I explored this topic myself in the What I learned trying to forecast coding-agent spend article.
Ideally, you need to check how well your classifier actually works — whether it classifies tasks correctly and what its success rate is. There are some specialized models, but they are not the ones provided by default with an OpenAI subscription or NeMo Switchyard. So now you need to either find a good match or train your own, which gets us back to the ROI question — is it really worth it when something like Sol Medium can solve most of your issues reasonably well?
Then there’s your default workflow. I personally often send screenshots of errors or UI bugs to my coding agent. But the classifier model doesn’t necessarily work with images. Another issue I ran into is the content of the initial prompt. For example, I often write up .md files that contain descriptions of the work to be done. Then my prompt is “Implement task.md”. The naive classifier call won’t see the content of the file and very likely will use your default model, not the most appropriate one. I had to change my default workflow to include the content of the task.md file in the initial classification call to work around the issue, but it’s just one of the many potential gotchas here.
Yet another thing to figure out is how often to reclassify. Should it be per LLM call and increase the overall call count? Or per user turn? Or per session? Everything can be tested and answered, but the time and effort spent on configuration keep adding up.
And finally, the coding agent harness and the IDEs often have their own configurations and optimizations to work with OpenAI and Anthropic subscriptions and APIs. Adding one more piece into this puzzle again can make the situation worse, not better. For example, I ran into issues when I added Switchyard to my OpenCode Desktop, where the classifier correctly chose Terra Medium, but cache misses increased token usage and resulted in a higher estimated cost than my Sol Medium baseline.
Switchyard provides a Stage Router that avoids the LLM classifier altogether, but it needs tuning as well. As a reminder, in my setup it defaults to a weaker model and escalates to a stronger model when the confidence threshold is reached.
With the two tests I chose to run, I got to experience two different failures — one due to a high threshold and another one due to a low one:
What makes matters worse is the potential savings impact. Letting Qwen finish a simpler task would result in $0 estimated cost. But escalating from Qwen to Sol Medium for the harder task turned out to be more expensive than simply solving it with Sol Medium in the first place — almost 2x more — most likely due to the increased context size that now included all of Qwen’s errors as well.
There is a general issue of handoff between the models while working on a task within one session, especially if it doesn’t involve subagents. My local Qwen3.6–35B is running on a three-year-old MacBook Pro. I like it because it lets me explore local AI capabilities, but its context window is only 128K (compare this to 1M for OpenAI’s GPT-6 models). The compaction and handoff issues were very noticeable when I used Switchyard for my normal coding tasks. In a longer thread, Qwen would often just fail to pick the work handed off to it.
You can expect other potential issues if you start mixing models with different capabilities and configurations, beyond the context window size. Support for different languages and images, along with privacy and security concerns — all of this can turn into problems during handoff between different models.
Is this fixable? I am sure it is. The question again is the ROI of the work you have to do before you get to any savings and the maintenance of it all as AI models keep evolving.
If your goal for model routing is saving tokens, whether to stretch your subscription allowance or reduce your metered API bill, this is what I would recommend:
For myself, I ended up with a simple conclusion — for day-to-day work, Sol Medium, the Toyota Corolla of LLMs, is good enough for me, and the headache of choosing a different method of transportation for each trip is not worth the effort, at least not today.
Model Routing with NVIDIA’s NeMo Switchyard was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.