You have built the chatbot. You have tuned the prompt, chosen the frontier model, and wired everything into production. The responses are fluent, helpful, and impressively human. Then you look at the bill.
A third of your API calls are asking for business hours. Another chunk is saying hello, goodbye, or requesting a password reset. These are not open-ended reasoning tasks. They are simple, predictable intents with well-known answers. Yet your most expensive model is generating them token by token, one by one, thousands of times per day.
The problem is not your model. It is your routing.
Every query is treated as a full LLM problem. There is no cheap layer deciding whether a message even needs a large model. The result is predictable: latency that frustrates users, costs that scale linearly with traffic, and a system that burns money on the digital equivalent of saying “Hi there.”
A semantic router is the layer that fixes this. It is one of the simplest yet highest-return upgrades you can add to an LLM-powered product.
A semantic router is a lightweight intent classification layer that sits between the user and your LLM infrastructure. It uses a fast, cheap embedding model to compare the meaning of a query against known route definitions, then sends the query to the cheapest destination that can handle it.
The word “semantic” matters. Traditional routing relies on keywords, regular expressions, or exact matches. That works until someone asks “What time do you open?” instead of “What are your opening hours?” or “When does the store close?” The sentences share no keywords, but they share the same intent.
A semantic router captures meaning. It embeds the incoming query and the example utterances for each route into the same vector space. Then it measures cosine similarity. If the query is close enough to a known route, it goes there. If not, it falls back to the general LLM.
This makes the router both flexible and cheap. It does not need an LLM call to make a routing decision. A small sentence transformer on a CPU can classify thousands of queries per second for essentially zero cost.
Most teams start with one of two approaches. They either use keyword rules or they call an LLM to classify intent. Both have problems.
Keyword rules are brittle. The moment a user rephrases a greeting, the rule breaks. You end up with an ever-growing list of regexes that nobody wants to maintain.
LLM-as-classifier is more robust, but it defeats the purpose. You are still paying for a model call before you decide whether to call a model. The classification itself adds one to three seconds of latency and another round of token cost.
The semantic router sits in the middle. It is flexible enough to handle rephrasing, because it understands meaning. It is cheap enough to run on every query, because it uses an embedding model rather than a generative model. And it is fast enough that the routing decision itself is measured in milliseconds.
The core idea is simple, but the implementation has a clear structure. Here is the flow from route definition to final destination.
A route is a destination with a set of example utterances. Static routes return hardcoded answers. Agent routes forward to a specialized model or system. You usually want a fallback route for anything uncertain.
static routes: greeting: ["hello", "hi", "hey there"] hours: ["what time do you open?", "when do you close?"] farewell: ["bye", "goodbye", "see you"]agent routes: billing: ["i was charged twice", "refund my subscription"] technical: ["my integration is failing", "api error 500"]fallback: general LLM
At startup, encode every example utterance for every route using a small sentence transformer. The result is a matrix of normalized embedding vectors. This is done once, so the cost is amortized across every query.
When a user sends a message, encode that single query with the same model. The embedding should be normalized so cosine similarity is just a dot product.
Compare the query vector against every route vector. The highest score is the winning route. You can use the maximum score across utterances, the average score, or a weighted blend. Maximum is the simplest starting point and tends to be most decisive.
If the best score is above your threshold, send the query to that route. If not, send it to the fallback. The threshold is the most important hyperparameter in the system. Too low and queries go to the wrong route. Too high and the LLM is doing work a static route could have handled. This one is kinda tricky to be honest.
Let me make the economics concrete. Imagine a customer support bot that handles one hundred thousand messages per month.
Roughly thirty-five percent of those messages are greetings, farewells, business hours, password resets, and other simple FAQs. You almost definitely use the top tier model for this Customer Support Agent since of course you don’t want less intelligence model ruin your potential customer experience. At a frontier model price of fifteen dollars per million tokens and an average of one hundred fifty tokens per query, that trivial traffic alone costs around fifteen dollars per day.
That is over five thousand dollars per year spent generating answers that were already known.
And cost is only half the story. Each LLM call adds one to three seconds of latency. A user asking for your opening hours has to wait for a model to slowly type out “We are open Monday to Friday, 9 AM to 5 PM.” That is a frustrating experience for a question that deserves an instant answer.
A semantic router changes the equation. The embedding model is local and runs in under ten milliseconds. Static routes return in microseconds. The expensive model only sees the queries that actually need it.
With the router handling thirty-five percent of traffic as static routes, daily LLM spend drops from roughly forty-five dollars to twenty-nine dollars. Responses return in under ten milliseconds instead of one to three seconds. Over a year, the savings on a single bot approach six thousand dollars. And the user experience on simple queries improves dramatically.
Routes are not all the same. The most useful systems distinguish between static routes and agent routes.
The key insight is that the router decides which destination fits without calling an LLM to make that decision. It is classification, not generation. That distinction is what makes the cost and latency numbers work.
You can also add a safety route before the main router. This route catches off-topic, abusive, or policy-violating inputs before they reach any LLM. Because it is cheap, you can run it on every message without worrying about the bill.
Let’s make it into a simple code implementation. You need a sentence transformer, some route definitions, cosine similarity, and a threshold. Here is a complete example you can adapt.
import numpy as npfrom sentence_transformers import SentenceTransformerclass SemanticRouter: def __init__(self, routes, model_name="all-MiniLM-L6-v2", threshold=0.8): self.routes = routes self.threshold = threshold self.encoder = SentenceTransformer(model_name) # Pre-compute normalized embeddings for every route utterance. # This is done once at startup so each query only needs one embedding. for route in self.routes: route["emb"] = self.encoder.encode( route["utterances"], normalize_embeddings=True, ) def route(self, query): # Encode the incoming query with the same model. q_emb = self.encoder.encode(query, normalize_embeddings=True) # Compare against each route using cosine similarity. # Because vectors are normalized, cosine similarity is just a dot product. # We take the maximum score across all utterances in a route. scores = [ (route["name"], np.max(route["emb"] @ q_emb)) for route in self.routes ] name, score = max(scores, key=lambda x: x[1]) return name if score >= self.threshold else "fallback"if __name__ == "__main__": routes = [ {"name": "greeting", "utterances": ["hello", "hi", "hey", "hi there"]}, {"name": "farewell", "utterances": ["bye", "goodbye", "see you", "it's time to go"]}, { "name": "hours", "utterances": [ "what are your hours", "when do you open", "when does the store close", ], }, ] router = SemanticRouter(routes, threshold=0.75) queries = [ "Hi there!", "It's time to go.", "What time do you open?", "Who won the game?", ] for q in queries: print(f"Query: '{q}' -> Routed to: {router.route(q)}")
The code is intentionally simple. There is no vector database, no LangChain, no external API. The model is twenty-two megabytes, runs on CPU, and the only dependency beyond NumPy is the sentence transformer library.
When you run this, the output looks like this:
Query: 'Hi there!' -> Routed to: greetingQuery: 'It's time to go.' -> Routed to: farewellQuery: 'What time do you open?' -> Routed to: hoursQuery: 'Who won the game?' -> Routed to: fallback
Notice that “What time do you open?” routes to the hours intent even though it does not share a single keyword with the example “When does the store close?” That is the semantic part in action.
For larger systems, one router is not enough. The most robust architectures use stacked routers and guardrails.
The most common mistake is overthinking the model. For most production routing tasks, all-MiniLM-L6-v2 is the right choice. It is twenty-two megabytes, runs on CPU, costs nothing, and classifies intent well enough to be useful. It has only 384 dimensions, so similarity computation is fast.
Only upgrade to a larger or API-based embedding model if you can measure a meaningful accuracy gap on your labeled evaluation set. The extra dimensions and quality usually do not matter for routing, and they add latency and cost.
The threshold is more important than the model. It controls the trade-off between routing precision and LLM call rate.
Tune the threshold on a labeled set of real queries. Plot the precision and the LLM call rate as you sweep the threshold. The goal is to maximize the percentage of trivial queries handled by static routes while keeping the misroute rate below your tolerance.
You should also measure the router in production. Log the query, the winning route, the score, and the final outcome. Every week, review low-confidence queries and ambiguous cases. Add new example utterances to the routes that are underperforming. A semantic router is not a set-and-forget component; it improves with feedback.
Let me put this together in a concrete scenario. Imagine a SaaS company with a support chatbot that currently routes everything to a frontier model. A semantic router is deployed with five routes:
The static routes save the company forty-five percent of LLM calls immediately. The agent routes improve quality because each one gets a tailored system prompt and the right tools. The general LLM only handles the ten percent of queries that actually need it.
Latency drops because static responses return in under ten milliseconds. The billing and technical agents are faster because they receive shorter, more focused prompts rather than a long general system prompt.
The implementation cost is small. The model is open source. The code is less than a hundred lines. The deployment can be a single Python service or a lightweight function inside your existing API.
The customer support bot example has one more route worth adding: a safety route. Some inputs should never reach any LLM, no matter how cheap the routing step is. A semantic router can block off-topic, abusive, or policy-violating queries before they reach the billing or technical agents.
The guardrail route works exactly like the others. You define example utterances for the kinds of inputs you want to reject: prompts that ask for medical advice, legal opinions, competitors’ proprietary data, or anything outside your product’s scope. When the router matches the query to this route, it returns a standard refusal instead of forwarding the request.
The advantage is speed and cost. A safety check that runs inside a small embedding model is fast enough to run on every message and cheap enough that you never think about the bill. A false positive just sends a user to a human fallback. A false negative sends a risky query to your model.
In practice, the safety router runs first. The query is embedded, scored against the safety route, and only released to the domain router if it passes. This is the same pattern as layered routing: classify cheaply, decide early, and keep the expensive model away from work it should never do.
A semantic router is not a universal solution. It shines when you have predictable, high-volume intents mixed with genuinely open-ended queries. It is less useful when every query is unique and requires reasoning.
Use it when:
Skip it when:
The best LLM architecture is not the one that uses the biggest model for every query. It is the one that knows exactly when not to use it.
A semantic router turns your LLM pipeline from a flat, expensive firehose into a targeted, efficient system. It intercepts trivial queries before they reach the model. It sends complex queries to the right specialized agent. It leaves the frontier model for the work that genuinely needs it.
This is the pattern that makes LLM products economically viable at scale. It saves money. It improves latency. It makes the user experience better. And it does all of this with a small, open-source embedding model and a few lines of Python.
Here are several key takeaways from this article:
Thank you for reading this article! I hope you found it helpful.
#AIEngineering #LLM #SemanticRouter #MachineLearning #MLOps #GenerativeAI #CostOptimization #NaturalLanguageProcessing
Semantic Router: The AI Traffic Controller was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.