{"slug": "reduce-llm-spend-with-semantic-routing", "title": "Reduce LLM Spend with Semantic Routing", "summary": "A cost-based semantic routing demo combining vLLM Semantic Router (vSR) with agentgateway reduced LLM spend by 35.4% on a 50-prompt Go and Rust coding dataset, according to a developer evaluation published July 16, 2026. The routed lane cost $0.591444 versus $0.914840 for the always-expensive lane, with 90% expected-tier agreement (45 of 50 prompts) and lower p50 latency (5.22 vs 7.55 seconds).", "body_md": "I spend a lot of time in Go and Rust, where a quick “one more thing” in a long coding chat can turn a tidy refactor into an hour of debugging. After the hard part is over, routine tests and follow-up questions can still burn premium-model credits. Sending every request to a cheaper model is not the answer: distributed-systems design, correctness work, and difficult debugging need more capable help.\n\nThis example combines [vLLM Semantic Router (vSR)](https://vllm-sr.ai/)\nwith [agentgateway](https://agentgateway.dev/) to send routine Go and Rust\ncoding tasks to `gpt-5.4-nano`\n\nand escalate complex correctness,\ndistributed-systems, and deep-debugging work to `gpt-5.5`\n\n. The goal is a\nreproducible cost reduction without quietly routing everything to the cheapest\ntier.\n\nThe client sends an OpenAI-compatible request with `model: \"auto\"`\n\n. An\nagentgateway policy sends it to vSR, which evaluates semantic, complexity,\nkeyword, context, and structure signals before selecting a model. Agentgateway\nforwards the request and records the outcome. Cost is not a classifier input;\nit is measured after the request, keeping the policy explainable.\n\n`auto`\n\nopts into that automatic selection. A client can instead request\n`gpt-5.4-nano`\n\nor `gpt-5.5`\n\ndirectly to force a tier and bypass automatic\nrouting; the request still passes through agentgateway. Organizations\nthat require automatic routing can use a [request-body\ntransformation](https://agentgateway.dev/docs/kubernetes/latest/traffic-management/transformations/validate/)\nto rewrite every request to `auto`\n\n.\n\nMy company needs prices, not just token counts, to make a budget decision.\nAgentgateway loads a model cost catalog and exposes realized request cost in\nmetrics, logs, and traces. The [model cost catalog\nguide](https://agentgateway.dev/docs/kubernetes/latest/llm/cost-controls/costs/) explains how\nto generate and load the catalog with `agctl`\n\n.\n\nFor this evaluation, I use the integrated [OpenTelemetry stack](https://agentgateway.dev/docs/kubernetes/main/observability/otel-stack/).\nPrometheus, Loki, Tempo, the OpenTelemetry Collector, and Grafana let\nengineering and finance inspect the same model-selection event.\n\nI built the [cost-based semantic routing demo](https://github.com/danehans/agentgateway-demos/tree/main/cost-based-semantic-routing)\naround a compact 50-prompt Go and Rust dataset. Half of the prompts are routine\nimplementation or test work; the other half involve correctness or\ndistributed-systems problems. I send every prompt through two lanes:\n\n| Lane | Purpose |\n|---|---|\n`routed` |\nvSR selects `gpt-5.4-nano` or `gpt-5.5` . |\n`always_expensive` |\nEvery request uses `gpt-5.5` ; the cost baseline. |\n\nEach run has an ID on every request, so its cost query excludes unrelated cluster traffic. I deliberately leave out a forced-cheap lane: the useful comparison is whether routing saves money while still escalating complex work.\n\nEach prompt has an expected tier in the checked-in dataset: 25 routine prompts\nexpect `gpt-5.4-nano`\n\n, and 25 complex prompts expect `gpt-5.5`\n\n. Expected-tier\nagreement is the share of requests where vSR selected that labelled tier. It\nchecks the routing policy, not the quality of the model’s answer.\n\nIn a clean run on July 16, 2026, all 100 primary requests completed with exact\nmodel-catalog lookups. vSR selected `gpt-5.4-nano`\n\nfor 30 requests and\n`gpt-5.5`\n\nfor 20. Catalog-priced agentgateway metrics reported `$0.591444`\n\nfor\nthe routed lane versus `$0.914840`\n\nfor the always-expensive lane, a **35.4%\ncost reduction**. Routed p50 latency was **5.22 seconds**, compared with\n**7.55 seconds** for the always-expensive lane. P95 was 19.09 seconds for\nrouted traffic and 19.61 seconds for the always-expensive lane.\n\nThe chart reached **90% expected-tier agreement (45 of 50 prompts)**. All 25\nroutine prompts stayed on nano, and 20 of 25 complex prompts escalated to\nGPT-5.5. This is not a claim of perfect model selection, but it shows the\nsavings did not come from a blanket downgrade: 40% of routed requests still\nused the expensive model. It is a policy sanity check, not an answer-quality\nbenchmark or a substitute for application task-success and user-feedback data.\n\nThe default dataset uses independent requests. For long-running coding agents, the optional cache-transition evaluation uses two ordered ten-turn Go and Rust conversations with a stable prefix. It reports provider-observed cache reads by model transition without mixing cache behavior into the primary evaluation.\n\nThe demo automates the setup work I would otherwise repeat by hand. It uses the [agentgateway semantic-routing\nexample](https://github.com/agentgateway/agentgateway/tree/main/examples/llm-semantic-routing),\ncreates or reuses a kind cluster, and installs agentgateway, vSR, a model cost\ncatalog, and the OpenTelemetry stack. It checks readiness, routing,\ncatalog-priced metrics, logs, and traces before the primary requests run.\n\nBefore running it, review the [demo requirements](https://github.com/danehans/agentgateway-demos/tree/main/cost-based-semantic-routing#requirements):\n12 GiB of Docker memory, 30 GiB of free disk, the required CLIs, an OpenAI API\nkey, and access to both models.\n\n```\ngit clone https://github.com/danehans/agentgateway-demos.git\ncd agentgateway-demos/cost-based-semantic-routing\n\nexport OPENAI_API_KEY='sk-...'\n./demo.sh all --yes\n```\n\nThe default run sends 106 billable requests: two routing probes, four smoke-test\nrequests, and 100 primary requests. It writes request-level JSONL, an\nevaluation-scoped Prometheus report, and an SVG chart under `results/`\n\n.\n\nWhen I want to change the policy, I edit the fetched vSR values, redeploy with\n`./demo.sh router`\n\n, and run `./demo.sh eval --yes`\n\nagain.\n\nCodex lets me choose a model manually, but a user-level profile can instead\nsend every request to a gateway such as agentgateway with the stable `auto`\n\nmodel name:\n\n```\n# ~/.codex/agentgateway.config.toml\nmodel = \"auto\"\nmodel_provider = \"agentgateway\"\n\n[model_providers.agentgateway]\nname = \"Corporate agentgateway\"\nbase_url = \"https://my.corp.agentgateway.com/v1\"\nwire_api = \"responses\"\n```\n\nI start Codex with `codex --profile agentgateway`\n\n. It sends OpenAI Responses\nAPI traffic to agentgateway; vSR selects the tier and agentgateway records the\ndecision and realized cost.\n\nCurrent Codex emits an `unknown model auto`\n\nfallback-metadata warning. Routing\nstill works, but Codex warns that the fallback metadata may degrade behavior.\n\nThe profile keeps credentials out of the configuration file. Codex documents\n[custom model providers](https://learn.chatgpt.com/docs/config-file/config-advanced#custom-model-providers)\nand user-level [configuration profiles](https://learn.chatgpt.com/docs/config-file/config-advanced#profiles).\n\nSemantic routing is not a magic cost switch. Start with an explainable policy, compare it with always using the expensive model, confirm it uses both tiers, and tune it against task completion, feedback, retries, and escalation rates. Agentgateway supplies the accounting and observability; vSR makes the semantic decision.\n\nThis example is intentionally static: it uses configured keywords, semantic candidates, weights, and thresholds rather than learning from application outcomes. Those signals need periodic review as traffic changes. The agentgateway community and vSR project are continuing work to make semantic routing easier to tune, operate, and measure against real application traffic, so stay tuned.\n\nAfter validating a tiering policy, use [agentgateway cost controls](https://agentgateway.dev/docs/kubernetes/main/llm/cost-controls/)\nto attribute usage with virtual keys, observe realized spend by model and\nconsumer, and enforce budget or spend limits.", "url": "https://wpnews.pro/news/reduce-llm-spend-with-semantic-routing", "canonical_source": "https://agentgateway.dev/blog/2026-07-17-semantic-routing-llm-costs/", "published_at": "2026-07-20 00:00:00+00:00", "updated_at": "2026-07-21 14:58:01.939936+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools"], "entities": ["vLLM Semantic Router", "agentgateway", "gpt-5.4-nano", "gpt-5.5", "OpenTelemetry", "Prometheus", "Grafana", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/reduce-llm-spend-with-semantic-routing", "markdown": "https://wpnews.pro/news/reduce-llm-spend-with-semantic-routing.md", "text": "https://wpnews.pro/news/reduce-llm-spend-with-semantic-routing.txt", "jsonld": "https://wpnews.pro/news/reduce-llm-spend-with-semantic-routing.jsonld"}}