Semantic Routing, Assembled: agentgateway and vLLM Semantic Router in Practice

Agentgateway and vLLM Semantic Router (vSR) integrate via external processing (ExtProc) to enable semantic routing of prompts to the most cost-effective LLM model, reducing workload costs by approximately 40% without requiring the consumer to explicitly select a model. vSR acts as a classifier that extracts signals from the prompt, computes a score, maps it to a band, and applies boolean rules to select a model, while agentgateway owns the data path, forwards the request to the chosen provider (e.g., OpenAI, Anthropic, Gemini), and enforces policies such as authentication and rate limiting.

In our previous post https://agentgateway.dev/blog/2026-07-17-semantic-routing-llm-costs/ on semantic routing with agentgateway and vLLM Semantic Router we focussed on the benefits model selection can bring to agentic systems in terms of costs. We demonstrated that by analyzing and classifying each prompt, we can send routine work to cheap models, while keeping the frontier models doing the hard stuff. As a result, workloads can be processed ~40% cheaper, without the model consumer ever having to explicitly pick a model. In that post, we treated the router as a black box: agentgateway sends the received prompt to the router, the router picks a model, agentgateway routes the request to the selected model, processes the response, and records the conversation’s metrics. This post opens the box. Before we extend the idea of semantic routing to concepts like cross-provider model selection and routing, we need to have a better understanding of the underlying machinery: how agentgateway integrates with vLLM Semantic Router vSR , how vSR selects the model to route, how that information is communicated back to agentgateway, and finally how agentgateway uses this information to route the request to the correct LLM provider such as OpenAI, Anthropic, and Gemini and model such as gpt-5.5, claude-haiku-4-5, and gemini-3.5-flash . This blog uses agentgateway and the llm-semantic-routing example https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing , a solid starting point for building your own solution. The agentgateway docs give a concise The first thing to understand is what vLLM Semantic Router is not . It is not a model. It is not a proxy that sits in your data path forwarding tokens. vSR is a classifier that emits a routing decision . When agentgateway or any other system for that matter hands it a prompt, it extracts various signals from that prompt, such as keywords, complexity score, and context length. It combines these signals into a score using a concept called projection , which maps the score onto a band a named range of that score such as “low-cost” or “expensive” . Finally, the decision layer applies boolean rules over signals and bands, resolving to a model via the configured selection strategy priority, confidence, or tier . Agentgateway owns and controls the data path. It receives the client request and applies routing logic and policies like authentication, authorization, or rate limiting if configured and applicable. Next, it calls the model provider, streams the response, and, optionally, prices the request against its model-cost catalog and emits telemetry. The router decides, while the gateway enforces and observes. So how does a request on the gateway get a decision from the router mid-flight? Through external processing ExtProc https://agentgateway.dev/docs/kubernetes/latest/traffic-management/extproc/ about-external-processing : agentgateway streams a request’s headers and body, as well as its response headers and body if enabled, to an external gRPC service mid-flight. It holds the request until that service answers. The service isn’t just consulted . It receives the real headers and body and can mutate them, rewrite headers, replace the request body, or short-circuit the call entirely. Agentgateway applies those changes before further processing the request or response. vSR’s routing decision happens on the request side of that hook. It receives the prompt in the request body, classifies it, and enriches the request with its decision. It rewrites the model field in the body to the model it chose and adds an x-vsr-selected-model header plus diagnostic x-vsr- headers to the request. Agentgateway then forwards the request to the provider with that selected model. vSR speaks this ExtProc protocol on port 50051 , wired up with a single AgentgatewayPolicy https://agentgateway.dev/docs/kubernetes/latest/reference/api-kubespec/policies/ . Here is the actual request lifecycle for the single-provider setup: The response also passes back through vSR — for caching, safety, and usage accounting. It is omitted here to keep the routing path clear. The client sends model: "auto" . Agentgateway sends the request to vSR via ExtProc. vSR classifies, rewrites model in the request body to the model it chose, and adds an x-vsr-selected-model header to the request. Agentgateway then forwards the request to the LLM provider, OpenAI, with the chosen model. The client never picked a model, and the gateway never had to understand the prompt. The example https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing consists of three agentgateway objects plus a vSR Helm release https://vllm-sr.ai/docs/installation/k8s/agentgateway/ , which deploys the vSR. It assumes you already have a working agentgateway control plane and agentgateway proxy, an openai-secret , and optionally, a model-cost catalog https://agentgateway.dev/docs/kubernetes/main/llm/cost-controls/costs/ , and an OpenTelemetry stack https://agentgateway.dev/docs/kubernetes/main/observability/otel-stack/ all covered in the agentgateway docs https://agentgateway.dev/docs/kubernetes/main/llm/providers/openai/ . With that in place, the routing is small. For a complete, runnable end-to-end setup, see the upstream llm-semantic-routing example https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing . apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayBackend metadata: name: openai-router-selected namespace: agentgateway-system spec: ai: provider: No model is pinned here on purpose. vSR selects it per request; agentgateway forwards whatever model ended up in the request body. openai: {} policies: auth: secretRef: name: openai-secret The important detail is the empty openai: {} , which means that no specific model is set. In a normal LLM backend, you’d name the model. Here the model is a runtime decision , so the backend deliberately leaves it empty. apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: openai-semantic-routing namespace: agentgateway-system spec: parentRefs: - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway-proxy, namespace: agentgateway-system } rules: - matches: - path: type: PathPrefix value: /v1/chat/completions - path: type: PathPrefix value: /v1/responses backendRefs: - group: agentgateway.dev kind: AgentgatewayBackend name: openai-router-selected namespace: agentgateway-system A plain HTTPRoute : OpenAI-compatible chat and responses paths, one backend. Nothing about it knows a router exists. AgentgatewayBackends , such as to route requests across LLM providers, add multiple rules to the HTTPRoute , with at least one rule per LLM provider. We’ll cover dynamic cross-provider LLM routing in a follow-up blog post. apiVersion: agentgateway.dev/v1alpha1 kind: AgentgatewayPolicy metadata: name: semantic-router-extproc namespace: agentgateway-system spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute attached to the route above name: openai-semantic-routing traffic: extProc: backendRef: name: semantic-router vSR's gRPC service namespace: agentgateway-system port: 50051 processingOptions: requestHeaderMode: Send requestBodyMode: FullDuplexStreamed vSR needs the prompt body to classify responseHeaderMode: Send responseBodyMode: Buffered allowModeOverride: true This example shows the whole integration configuration in agentgateway. Note that vSR also needs to be correctly configured for ExtProc communication, as shown in the code repo’s example https://github.com/agentgateway/agentgateway/blob/main/examples/llm-semantic-routing/k8s/semantic-router-values.yaml L347-L362 . For this demo, we assume vSR has already been setup with this configuration . The AgentgatewayPolicy https://agentgateway.dev/docs/kubernetes/main/about/policies/ , targets the HTTPRoute and points at vSR’s ExtProc server. Note what is absent : there is no traffic.phase configuration defined in the policy so the policy is applied in the default PostRouting phase. This means that vSR runs after agentgateway has already selected the backend. For a single provider that is completely fine because there is only one backend, so the LLM provider routing decision is never in question. The only thing vSR changes is the model field inside the request body. Hold that thought, as it is the exact hinge that the next blog post, which discusses cross LLM provider model routing, turns on. vSR’s behavior lives in its Helm values semantic-router-values.yaml https://github.com/agentgateway/agentgateway/blob/main/examples/llm-semantic-routing/k8s/semantic-router-values.yaml . It reads like a small rules engine, and it’s worth understanding as a pipeline, because vSR traces its own work with a span per stage semantic router.signal. for signal evaluation, semantic router.decision.evaluation for the decision, and semantic router.plugin.execution for the plugin chain . A trace reads back as that same pipeline. 1. The models it may choose. vSR keeps its own view of the candidate models, their IDs, pricing, and the backend to reach them: providers: models: - name: gpt-5.4-nano provider model id: gpt-5.4-nano pricing: { prompt per 1m: 0.20, completion per 1m: 1.25 } - name: gpt-5.5 provider model id: gpt-5.5 pricing: { prompt per 1m: 5.00, completion per 1m: 30.00 } 2. Signals: the classification inputs. The example is tuned for coding prompts and combines several signal types: a high-specificity keyword list distributed rate limiter , linearizability , TLA+ , race condition , … , embedding similarity to example intents routine vs. advanced , a complexity score with hard/easy prototype phrases, a context -length signal, and structure signals many questions, dense constraints : routing: signals: keywords: - { name: advanced markers, operator: OR, keywords: distributed systems, consensus, linearizability, "TLA+", race condition, network partition, ... } embeddings: - name: advanced reasoning intent threshold: 0.68 candidates: - Design a fault-tolerant distributed system and analyze tradeoffs. - Find the root cause of a subtle concurrency bug from incomplete logs. complexity: - { name: needs advanced reasoning, threshold: 0.70, hard: {...}, easy: {...} } 3. Projection: collapse signals to a lane. A weighted sum turns all those signals into one advanced need score , then a threshold splits it into two lanes: routing: projections: scores: - name: advanced need score method: weighted sum inputs: - { type: keyword, name: advanced markers, weight: 0.45 } - { type: embedding, name: advanced reasoning intent, weight: 0.35 } - { type: complexity, name: needs advanced reasoning:hard, weight: 0.35 } - { type: keyword, name: routine coding markers, weight: -0.10 } pushes back toward cheap mappings: - name: advanced need band source: advanced need score outputs: - { name: low cost lane, lt: 0.35 } - { name: expensive lane, gte: 0.35 } 4. Decisions: a lane becomes a model. Finally, each lane maps to a concrete model, by priority: routing: decisions: - name: route to expensive priority 200 rules: { conditions: { type: projection, name: expensive lane } } modelRefs: { model: gpt-5.5 } - name: route to low cost priority 100 default rules: { conditions: { type: projection, name: low cost lane } } modelRefs: { model: gpt-5.4-nano } That is the reasoning the router does on every request: extract signals, score, band, decide. Here each lane maps to a single fixed model. Remember that, too In our cross-provider follow-up blog post, we will replace the “one model per lane” concept with “a set of candidate models per lane, cheapest capable one wins.” With model: "auto" and the debug header, the selected model shows up in the response headers: curl -sS -i "$GW/v1/chat/completions" -H 'Content-Type: application/json' -H 'X-VSR-Debug: true' \ -d '{"model":"auto","messages": {"role":"user","content":"Implement a small Go helper and one table-driven test."} ,"max tokens":64}' ... x-vsr-selected-model: gpt-5.4-nano Swap the prompt for a complex coding question and the same request comes back x-vsr-selected-model: gpt-5.5 . Two prompts, one endpoint. There was no model named by the client, and the decision was made entirely from the prompt’s content. Because agentgateway owns the data path, an OpenTelemetry stack shows the concept end to end. Agentgateway traces the request and the LLM call, which is priced against the model cost catalog. vSR emits its own spans for signal evaluation, the decision, and the plugin chain. When trace context is propagated across the ExtProc hop, those vSR spans join agentgateway’s trace, so a single trace reads as the request walking through the whole pipeline as previously described. Look again at what made the single-provider case simple: there is only one backend, so vSR never has to influence routing and only rewrites model. The ExtProc policy runs PostRouting, Add a second provider and that stops working. If the router decides a prompt should go to Anthropic, rewriting model isn’t enough. The request has to be routed to the Anthropic backend , and agentgateway makes its routing decision before ExtProc runs. The decision now has to be available before the route is selected, and the route has to be able to match on it. That is exactly what we will cover in our next post, the ability to make routing decisions and route requests across LLM providers, effectively opening the door to the whole model market. A working semantic-routing setup: a classifier that reads prompts and emits decisions, attached to agentgateway as an ExtProc, steering one provider’s model tiers with nothing in application code. That’s the foundation. For the reference version of this integration, see the vLLM Semantic Router integration page https://agentgateway.dev/docs/kubernetes/latest/integrations/vllm-semantic-router/ in the agentgateway docs. In our next post, we make the same decision choose a provider , not just a tier.