In our previous post on semantic routing with agentgateway and vLLM Semantic Router we focussed on the benefits model selection can bring to agentic systems in terms of costs. We demonstrated that by analyzing and classifying each prompt, we can send routine work to cheap models, while keeping the frontier models doing the hard stuff. As a result, workloads can be processed ~40% cheaper, without the model consumer ever having to explicitly pick a model. In that post, we treated the router as a black box: agentgateway sends the received prompt to the router, the router picks a model, agentgateway routes the request to the selected model, processes the response, and records the conversation’s metrics.
This post opens the box. Before we extend the idea of semantic routing to concepts like cross-provider model selection and routing, we need to have a better understanding of the underlying machinery: how agentgateway integrates with vLLM Semantic Router (vSR), how vSR selects the model to route, how that information is communicated back to agentgateway, and finally how agentgateway uses this information to route the request to the correct LLM provider (such as OpenAI, Anthropic, and Gemini) and model (such as gpt-5.5, claude-haiku-4-5, and gemini-3.5-flash).
This blog uses agentgateway and the llm-semantic-routing example, a solid starting point for building your own solution. The agentgateway docs give a concise
The first thing to understand is what vLLM Semantic Router is not. It is not a model. It is not a proxy that sits in your data path forwarding tokens. vSR is a classifier that emits a routing decision. When agentgateway (or any other system for that matter) hands it a prompt, it extracts various signals from that prompt, such as keywords, complexity score, and context length. It combines these signals into a score using a concept called projection, which maps the score onto a band (a named range of that score such as “low-cost” or “expensive”). Finally, the decision layer applies boolean rules over signals and bands, resolving to a model via the configured selection strategy (priority, confidence, or tier).
Agentgateway owns and controls the data path. It receives the client request and applies routing logic and policies like authentication, authorization, or rate limiting if configured and applicable. Next, it calls the model provider, streams the response, and, optionally, prices the request against its model-cost catalog and emits telemetry. The router decides, while the gateway enforces and observes.
So how does a request on the gateway get a decision from the router mid-flight? Through external processing (ExtProc): agentgateway streams a request’s headers and body, as well as its response headers and body if enabled, to an external gRPC service mid-flight. It holds the request until that service answers. The service isn’t just consulted. It receives the real headers and body and can mutate them, rewrite headers, replace the request body, or short-circuit the call entirely. Agentgateway applies those changes before further processing the request or response.
vSR’s routing decision happens on the request side of that hook. It receives the prompt in the request body, classifies it, and enriches the request with its decision. It rewrites the model
field in the body to the model it chose and adds an x-vsr-selected-model
header (plus diagnostic x-vsr-*
headers) to the request. Agentgateway then forwards the request to the provider with that selected model. vSR speaks this ExtProc protocol on port 50051
, wired up with a single AgentgatewayPolicy.
Here is the actual request lifecycle for the single-provider setup:
(The response also passes back through vSR — for caching, safety, and usage accounting. It is omitted here to keep the routing path clear.)
The client sends model: "auto"
. Agentgateway sends the request to vSR via ExtProc. vSR classifies, rewrites model
in the request body to the model it chose, and adds an x-vsr-selected-model
header to the request. Agentgateway then forwards the request to the LLM provider, OpenAI, with the chosen model. The client never picked a model, and the gateway never had to understand the prompt.
The example consists of three agentgateway objects plus a vSR Helm release, which deploys the vSR. It assumes you already have a working agentgateway control plane and agentgateway proxy, an openai-secret
, and optionally, a model-cost catalog, and an OpenTelemetry stack (all covered in the agentgateway docs). With that in place, the routing is small. For a complete, runnable end-to-end setup, see the upstream llm-semantic-routing example.
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
name: openai-router-selected
namespace: agentgateway-system
spec:
ai:
provider:
openai: {}
policies:
auth:
secretRef:
name: openai-secret
The important detail is the empty openai: {}
, which means that no specific model is set. In a normal LLM backend, you’d name the model. Here the model is a runtime decision, so the backend deliberately leaves it empty.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: openai-semantic-routing
namespace: agentgateway-system
spec:
parentRefs:
- { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway-proxy, namespace: agentgateway-system }
rules:
- matches:
- path:
type: PathPrefix
value: /v1/chat/completions
- path:
type: PathPrefix
value: /v1/responses
backendRefs:
- group: agentgateway.dev
kind: AgentgatewayBackend
name: openai-router-selected
namespace: agentgateway-system
A plain HTTPRoute
: OpenAI-compatible chat and responses paths, one backend. Nothing about it knows a router exists.
AgentgatewayBackends
, such as to route requests across LLM providers, add multiple rules to the HTTPRoute
, with at least one rule per LLM provider. We’ll cover dynamic cross-provider LLM routing in a follow-up blog post.
apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
name: semantic-router-extproc
namespace: agentgateway-system
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute # attached to the route above
name: openai-semantic-routing
traffic:
extProc:
backendRef:
name: semantic-router # vSR's gRPC service
namespace: agentgateway-system
port: 50051
processingOptions:
requestHeaderMode: Send
requestBodyMode: FullDuplexStreamed # vSR needs the prompt body to classify
responseHeaderMode: Send
responseBodyMode: Buffered
allowModeOverride: true
This example shows the whole integration configuration in agentgateway. (Note that vSR also needs to be correctly configured for ExtProc communication, as shown in the code repo’s example. For this demo, we assume vSR has already been setup with this configuration).
The AgentgatewayPolicy, targets the HTTPRoute and points at vSR’s ExtProc server. Note what is absent: there is no traffic.phase
configuration defined in the policy so the policy is applied in the default PostRouting phase. This means that vSR runs after agentgateway has already selected the backend. For a single provider that is completely fine because there is only one backend, so the LLM provider routing decision is never in question. The only thing vSR changes is the model
field inside the request body. Hold that thought, as it is the exact hinge that the next blog post, which discusses cross LLM provider model routing, turns on.
vSR’s behavior lives in its Helm values ( semantic-router-values.yaml). It reads like a small rules engine, and it’s worth understanding as a pipeline, because vSR traces its own work with a span per stage (
semantic_router.signal.*
for signal evaluation, semantic_router.decision.evaluation
for the decision, and semantic_router.plugin.execution
for the plugin chain). A trace reads back as that same pipeline.1. The models it may choose. vSR keeps its own view of the candidate models, their IDs, pricing, and the backend to reach them:
providers:
models:
- name: gpt-5.4-nano
provider_model_id: gpt-5.4-nano
pricing: { prompt_per_1m: 0.20, completion_per_1m: 1.25 }
- name: gpt-5.5
provider_model_id: gpt-5.5
pricing: { prompt_per_1m: 5.00, completion_per_1m: 30.00 }
2. Signals: the classification inputs. The example is tuned for coding prompts and combines several signal types: a high-specificity keyword list (distributed rate limiter
, linearizability
, TLA+
, race condition
, …), embedding similarity to example intents (routine vs. advanced), a complexity score with hard/easy prototype phrases, a context-length signal, and structure signals (many questions, dense constraints):
routing:
signals:
keywords:
- { name: advanced_markers, operator: OR, keywords: [distributed systems, consensus, linearizability, "TLA+", race condition, network partition, ...] }
embeddings:
- name: advanced_reasoning_intent
threshold: 0.68
candidates:
- Design a fault-tolerant distributed system and analyze tradeoffs.
- Find the root cause of a subtle concurrency bug from incomplete logs.
complexity:
- { name: needs_advanced_reasoning, threshold: 0.70, hard: {...}, easy: {...} }
3. Projection: collapse signals to a lane. A weighted sum turns all those signals into one advanced_need_score
, then a threshold splits it into two lanes:
routing:
projections:
scores:
- name: advanced_need_score
method: weighted_sum
inputs:
- { type: keyword, name: advanced_markers, weight: 0.45 }
- { type: embedding, name: advanced_reasoning_intent, weight: 0.35 }
- { type: complexity, name: needs_advanced_reasoning:hard, weight: 0.35 }
- { type: keyword, name: routine_coding_markers, weight: -0.10 } # pushes back toward cheap
mappings:
- name: advanced_need_band
source: advanced_need_score
outputs:
- { name: low_cost_lane, lt: 0.35 }
- { name: expensive_lane, gte: 0.35 }
4. Decisions: a lane becomes a model. Finally, each lane maps to a concrete model, by priority:
routing:
decisions:
- name: route_to_expensive # priority 200
rules: { conditions: [{ type: projection, name: expensive_lane }] }
modelRefs: [{ model: gpt-5.5 }]
- name: route_to_low_cost # priority 100 (default)
rules: { conditions: [{ type: projection, name: low_cost_lane }] }
modelRefs: [{ model: gpt-5.4-nano }]
That is the reasoning the router does on every request: extract signals, score, band, decide. Here each lane maps to a single fixed model. Remember that, too! In our cross-provider follow-up blog post, we will replace the “one model per lane” concept with “a set of candidate models per lane, cheapest capable one wins.”
With model: "auto"
and the debug header, the selected model shows up in the response headers:
curl -sS -i "$GW/v1/chat/completions" -H 'Content-Type: application/json' -H 'X-VSR-Debug: true' \
-d '{"model":"auto","messages":[{"role":"user","content":"Implement a small Go helper and one table-driven test."}],"max_tokens":64}'
Swap the prompt for a complex coding question and the same request comes back x-vsr-selected-model: gpt-5.5
. Two prompts, one endpoint. There was no model named by the client, and the decision was made entirely from the prompt’s content.
Because agentgateway owns the data path, an OpenTelemetry stack shows the concept end to end. Agentgateway traces the request and the LLM call, which is priced against the model cost catalog. vSR emits its own spans for signal evaluation, the decision, and the plugin chain. When trace context is propagated across the ExtProc hop, those vSR spans join agentgateway’s trace, so a single trace reads as the request walking through the whole pipeline as previously described.
Look again at what made the single-provider case simple: there is only one backend, so vSR never has to influence routing and only rewrites model. The ExtProc policy runs PostRouting,
Add a second provider and that stops working. If the router decides a prompt should go to Anthropic, rewriting model
isn’t enough. The request has to be routed to the Anthropic backend, and agentgateway makes its routing decision before ExtProc runs. The decision now has to be available before the route is selected, and the route has to be able to match on it.
That is exactly what we will cover in our next post, the ability to make routing decisions and route requests across LLM providers, effectively opening the door to the whole model market.
A working semantic-routing setup: a classifier that reads prompts and emits decisions, attached to agentgateway as an ExtProc, steering one provider’s model tiers with nothing in application code. That’s the foundation. For the reference version of this integration, see the vLLM Semantic Router integration page in the agentgateway docs. In our next post, we make the same decision choose a provider, not just a tier.