{"slug": "semantic-routing-assembled-agentgateway-and-vllm-semantic-router-in-practice", "title": "Semantic Routing, Assembled: agentgateway and vLLM Semantic Router in Practice", "summary": "Agentgateway and vLLM Semantic Router (vSR) integrate via external processing (ExtProc) to enable semantic routing of prompts to the most cost-effective LLM model, reducing workload costs by approximately 40% without requiring the consumer to explicitly select a model. vSR acts as a classifier that extracts signals from the prompt, computes a score, maps it to a band, and applies boolean rules to select a model, while agentgateway owns the data path, forwards the request to the chosen provider (e.g., OpenAI, Anthropic, Gemini), and enforces policies such as authentication and rate limiting.", "body_md": "In our [previous post](https://agentgateway.dev/blog/2026-07-17-semantic-routing-llm-costs/) on semantic routing with agentgateway and vLLM Semantic Router we focussed on the benefits model selection can bring to agentic systems in terms of costs. We demonstrated that by analyzing and classifying each prompt, we can send routine work to cheap models, while keeping the frontier models doing the hard stuff. As a result, workloads can be processed ~40% cheaper, without the model consumer ever having to explicitly pick a model. In that post, we treated the router as a black box: agentgateway sends the received prompt to the router, the router picks a model, agentgateway routes the request to the selected model, processes the response, and records the conversation’s metrics.\n\nThis post opens the box. Before we extend the idea of semantic routing to concepts like cross-provider model selection and routing, we need to have a better understanding of the underlying machinery: how agentgateway integrates with vLLM Semantic Router (vSR), how vSR selects the model to route, how that information is communicated back to agentgateway, and finally how agentgateway uses this information to route the request to the correct LLM provider (such as OpenAI, Anthropic, and Gemini) and model (such as gpt-5.5, claude-haiku-4-5, and gemini-3.5-flash).\n\nThis blog uses agentgateway and the [ llm-semantic-routing example](https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing), a solid starting point for building your own solution. The agentgateway docs give a concise\n\nThe first thing to understand is what vLLM Semantic Router is *not*. It is not a model. It is not a proxy that sits in your data path forwarding tokens. vSR is a **classifier that emits a routing decision**. When agentgateway (or any other system for that matter) hands it a prompt, it extracts various signals from that prompt, such as keywords, complexity score, and context length. It combines these signals into a score using a concept called *projection*, which maps the score onto a band (a named range of that score such as “low-cost” or “expensive”). Finally, the decision layer applies boolean rules over signals and bands, resolving to a model via the configured selection strategy (priority, confidence, or tier).\n\nAgentgateway owns and controls the data path. It receives the client request and applies routing logic and policies like authentication, authorization, or rate limiting if configured and applicable. Next, it calls the model provider, streams the response, and, optionally, prices the request against its model-cost catalog and emits telemetry. The router decides, while the gateway enforces and observes.\n\nSo how does a request on the gateway get a decision from the router mid-flight? Through [external processing (ExtProc)](https://agentgateway.dev/docs/kubernetes/latest/traffic-management/extproc/#about-external-processing): agentgateway streams a request’s headers and body, as well as its response headers and body if enabled, to an external gRPC service mid-flight. It holds the request until that service answers. The service isn’t just *consulted*. It receives the real headers and body and can **mutate** them, rewrite headers, replace the request body, or short-circuit the call entirely. Agentgateway applies those changes before further processing the request or response.\n\nvSR’s *routing decision* happens on the request side of that hook. It receives the prompt in the request body, classifies it, and enriches the request with its decision. It rewrites the `model`\n\nfield in the body to the model it chose and adds an `x-vsr-selected-model`\n\nheader (plus diagnostic `x-vsr-*`\n\nheaders) to the request. Agentgateway then forwards the request to the provider with that selected model. vSR speaks this ExtProc protocol on port `50051`\n\n, wired up with a single [ AgentgatewayPolicy](https://agentgateway.dev/docs/kubernetes/latest/reference/api-kubespec/policies/).\n\nHere is the actual request lifecycle for the single-provider setup:\n\n*(The response also passes back through vSR — for caching, safety, and usage accounting. It is omitted here to keep the routing path clear.)*\n\nThe client sends `model: \"auto\"`\n\n. Agentgateway sends the request to vSR via ExtProc. vSR classifies, rewrites `model`\n\nin the request body to the model it chose, and adds an `x-vsr-selected-model`\n\nheader to the request. Agentgateway then forwards the request to the LLM provider, OpenAI, with the chosen model. The client never picked a model, and the gateway never had to understand the prompt.\n\n[The example](https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing) consists of three agentgateway objects plus a [vSR Helm release](https://vllm-sr.ai/docs/installation/k8s/agentgateway/), which deploys the vSR. It assumes you already have a working agentgateway control plane and agentgateway proxy, an `openai-secret`\n\n, and optionally, a [model-cost catalog](https://agentgateway.dev/docs/kubernetes/main/llm/cost-controls/costs/), and an [OpenTelemetry stack](https://agentgateway.dev/docs/kubernetes/main/observability/otel-stack/) (all covered in the [agentgateway docs](https://agentgateway.dev/docs/kubernetes/main/llm/providers/openai/)). With that in place, the routing is small. For a complete, runnable end-to-end setup, see the upstream [ llm-semantic-routing example](https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing).\n\n```\napiVersion: agentgateway.dev/v1alpha1\nkind: AgentgatewayBackend\nmetadata:\n name: openai-router-selected\n namespace: agentgateway-system\nspec:\n ai:\n provider:\n # No model is pinned here on purpose. vSR selects it per request;\n # agentgateway forwards whatever model ended up in the request body.\n openai: {}\n policies:\n auth:\n secretRef:\n name: openai-secret\n```\n\nThe important detail is the empty `openai: {}`\n\n, which means that no specific model is set. In a normal LLM backend, you’d name the model. Here the model is a *runtime decision*, so the backend deliberately leaves it empty.\n\n```\napiVersion: gateway.networking.k8s.io/v1\nkind: HTTPRoute\nmetadata:\n name: openai-semantic-routing\n namespace: agentgateway-system\nspec:\n parentRefs:\n - { group: gateway.networking.k8s.io, kind: Gateway, name: agentgateway-proxy, namespace: agentgateway-system }\n rules:\n - matches:\n - path:\n type: PathPrefix\n value: /v1/chat/completions\n - path:\n type: PathPrefix\n value: /v1/responses\n backendRefs:\n - group: agentgateway.dev\n kind: AgentgatewayBackend\n name: openai-router-selected\n namespace: agentgateway-system\n```\n\nA plain `HTTPRoute`\n\n: OpenAI-compatible chat and responses paths, one backend. Nothing about it knows a router exists.\n\n`AgentgatewayBackends`\n\n, such as to route requests across LLM providers, add multiple rules to the `HTTPRoute`\n\n, with at least one rule per LLM provider. We’ll cover dynamic cross-provider LLM routing in a follow-up blog post.\n\n```\napiVersion: agentgateway.dev/v1alpha1\nkind: AgentgatewayPolicy\nmetadata:\n name: semantic-router-extproc\n namespace: agentgateway-system\nspec:\n targetRefs:\n - group: gateway.networking.k8s.io\n kind: HTTPRoute  # attached to the route above\n name: openai-semantic-routing\n traffic:\n extProc:\n backendRef:\n name: semantic-router  # vSR's gRPC service\n namespace: agentgateway-system\n port: 50051\n processingOptions:\n requestHeaderMode: Send\n requestBodyMode: FullDuplexStreamed  # vSR needs the prompt body to classify\n responseHeaderMode: Send\n responseBodyMode: Buffered\n allowModeOverride: true\n```\n\nThis example shows the whole integration configuration in agentgateway. (Note that vSR also needs to be correctly configured for ExtProc communication, as shown [in the code repo’s example](https://github.com/agentgateway/agentgateway/blob/main/examples/llm-semantic-routing/k8s/semantic-router-values.yaml#L347-L362). For this demo, we assume vSR has already been setup with this configuration).\n\nThe [AgentgatewayPolicy](https://agentgateway.dev/docs/kubernetes/main/about/policies/), targets the HTTPRoute and points at vSR’s ExtProc server. Note what is *absent*: there is no `traffic.phase`\n\nconfiguration defined in the policy so the policy is applied in the default **PostRouting** phase. This means that vSR runs *after* agentgateway has already selected the backend. For a single provider that is completely fine because there is only one backend, so the LLM provider routing decision is never in question. The only thing vSR changes is the `model`\n\nfield inside the request body. Hold that thought, as it is the exact hinge that the next blog post, which discusses cross LLM provider model routing, turns on.\n\nvSR’s behavior lives in its Helm values ([ semantic-router-values.yaml](https://github.com/agentgateway/agentgateway/blob/main/examples/llm-semantic-routing/k8s/semantic-router-values.yaml)). It reads like a small rules engine, and it’s worth understanding as a pipeline, because vSR traces its own work with a span per stage (\n\n`semantic_router.signal.*`\n\nfor signal evaluation, `semantic_router.decision.evaluation`\n\nfor the decision, and `semantic_router.plugin.execution`\n\nfor the plugin chain). A trace reads back as that same pipeline.**1. The models it may choose.** vSR keeps its own view of the candidate models, their IDs, pricing, and the backend to reach them:\n\n```\nproviders:\n models:\n - name: gpt-5.4-nano\n provider_model_id: gpt-5.4-nano\n pricing: { prompt_per_1m: 0.20, completion_per_1m: 1.25 }\n - name: gpt-5.5\n provider_model_id: gpt-5.5\n pricing: { prompt_per_1m: 5.00, completion_per_1m: 30.00 }\n```\n\n**2. Signals: the classification inputs.** The example is tuned for coding prompts and combines several signal types: a high-specificity **keyword** list (`distributed rate limiter`\n\n, `linearizability`\n\n, `TLA+`\n\n, `race condition`\n\n, …), **embedding** similarity to example intents (routine vs. advanced), a **complexity** score with hard/easy prototype phrases, a **context**-length signal, and **structure** signals (many questions, dense constraints):\n\n```\nrouting:\n signals:\n keywords:\n - { name: advanced_markers, operator: OR, keywords: [distributed systems, consensus, linearizability, \"TLA+\", race condition, network partition, ...] }\n embeddings:\n - name: advanced_reasoning_intent\n threshold: 0.68\n candidates:\n - Design a fault-tolerant distributed system and analyze tradeoffs.\n - Find the root cause of a subtle concurrency bug from incomplete logs.\n complexity:\n - { name: needs_advanced_reasoning, threshold: 0.70, hard: {...}, easy: {...} }\n```\n\n**3. Projection: collapse signals to a lane.** A weighted sum turns all those signals into one `advanced_need_score`\n\n, then a threshold splits it into two lanes:\n\n```\nrouting:\n projections:\n scores:\n - name: advanced_need_score\n method: weighted_sum\n inputs:\n - { type: keyword, name: advanced_markers, weight: 0.45 }\n - { type: embedding, name: advanced_reasoning_intent, weight: 0.35 }\n - { type: complexity, name: needs_advanced_reasoning:hard, weight: 0.35 }\n - { type: keyword, name: routine_coding_markers, weight: -0.10 } # pushes back toward cheap\n mappings:\n - name: advanced_need_band\n source: advanced_need_score\n outputs:\n - { name: low_cost_lane, lt: 0.35 }\n - { name: expensive_lane, gte: 0.35 }\n```\n\n**4. Decisions: a lane becomes a model.** Finally, each lane maps to a concrete model, by priority:\n\n```\nrouting:\n decisions:\n - name: route_to_expensive  # priority 200\n rules: { conditions: [{ type: projection, name: expensive_lane }] }\n modelRefs: [{ model: gpt-5.5 }]\n - name: route_to_low_cost  # priority 100 (default)\n rules: { conditions: [{ type: projection, name: low_cost_lane }] }\n modelRefs: [{ model: gpt-5.4-nano }]\n```\n\nThat is the reasoning the router does on every request: extract signals, score, band, decide. Here each lane maps to a single fixed model. Remember that, too! In our cross-provider follow-up blog post, we will replace the “one model per lane” concept with “a set of candidate models per lane, cheapest capable one wins.”\n\nWith `model: \"auto\"`\n\nand the debug header, the selected model shows up in the response headers:\n\n```\ncurl -sS -i \"$GW/v1/chat/completions\" -H 'Content-Type: application/json' -H 'X-VSR-Debug: true' \\\n -d '{\"model\":\"auto\",\"messages\":[{\"role\":\"user\",\"content\":\"Implement a small Go helper and one table-driven test.\"}],\"max_tokens\":64}'\n# ...\n# x-vsr-selected-model: gpt-5.4-nano\n```\n\nSwap the prompt for a complex coding question and the same request comes back `x-vsr-selected-model: gpt-5.5`\n\n. Two prompts, one endpoint. There was no model named by the client, and the decision was made entirely from the prompt’s content.\n\nBecause agentgateway owns the data path, an OpenTelemetry stack shows the concept end to end. Agentgateway traces the request and the LLM call, which is priced against the model cost catalog. vSR emits its own spans for signal evaluation, the decision, and the plugin chain. When trace context is propagated across the ExtProc hop, those vSR spans join agentgateway’s trace, so a single trace reads as the request walking through the whole pipeline as previously described.\n\nLook again at what made the single-provider case simple: **there is only one backend, so vSR never has to influence routing and only rewrites model.** The ExtProc policy runs PostRouting,\n\nAdd a second provider and that stops working. If the router decides a prompt should go to Anthropic, rewriting `model`\n\nisn’t enough. The request has to be *routed to the Anthropic backend*, and agentgateway makes its routing decision before ExtProc runs. The decision now has to be available *before* the route is selected, and the route has to be able to match on it.\n\nThat is exactly what we will cover in our next post, the ability to make routing decisions and route requests across LLM providers, effectively opening the door to the whole model market.\n\nA working semantic-routing setup: a classifier that reads prompts and emits decisions, attached to agentgateway as an ExtProc, steering one provider’s model tiers with nothing in application code. That’s the foundation. For the reference version of this integration, see the [vLLM Semantic Router integration page](https://agentgateway.dev/docs/kubernetes/latest/integrations/vllm-semantic-router/) in the agentgateway docs. In our next post, we make the same decision choose a *provider*, not just a tier.", "url": "https://wpnews.pro/news/semantic-routing-assembled-agentgateway-and-vllm-semantic-router-in-practice", "canonical_source": "https://agentgateway.dev/blog/2026-07-28-agentgateway-semantic-router-integration/", "published_at": "2026-07-28 00:00:00+00:00", "updated_at": "2026-07-28 16:26:19.710563+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "ai-agents", "machine-learning", "large-language-models"], "entities": ["agentgateway", "vLLM Semantic Router", "OpenAI", "Anthropic", "Gemini", "gpt-5.5", "claude-haiku-4-5", "gemini-3.5-flash"], "alternates": {"html": "https://wpnews.pro/news/semantic-routing-assembled-agentgateway-and-vllm-semantic-router-in-practice", "markdown": "https://wpnews.pro/news/semantic-routing-assembled-agentgateway-and-vllm-semantic-router-in-practice.md", "text": "https://wpnews.pro/news/semantic-routing-assembled-agentgateway-and-vllm-semantic-router-in-practice.txt", "jsonld": "https://wpnews.pro/news/semantic-routing-assembled-agentgateway-and-vllm-semantic-router-in-practice.jsonld"}}