{"slug": "measuring-agentgateway-s-overhead-as-an-inference-gateway", "title": "Measuring Agentgateway's Overhead as an Inference Gateway", "summary": "A Google Summer of Code 2026 project by Abhay Chaurasiya, mentored by Nina Polshakova and Daneyon Hansen of CNCF, measured the overhead of agentgateway as an inference gateway and found it dramatically outperforms plain round-robin Kubernetes Service under load. On 16 H100 GPUs with Qwen/Qwen3-32B across 8 vLLM servers, agentgateway standalone achieved 16,178 peak output tokens/s (+134.1%) and 16.52 requests/sec (+146.5%) versus 6,910 tokens/s and 6.70 req/s for the Kubernetes Service, while cutting TTFT p50 from 62.9s to 0.1s at 60 QPS. The project built on llm-d-benchmark tooling and is tracked in agentgateway/agentgateway#85.", "body_md": "*A GSoC 2026 project summary — Abhay Chaurasiya, mentored by Nina Polshakova\nand Daneyon Hansen, CNCF*\n\nThis work originally started as an attempt to benchmark the inference\nrouting extension in kgateway ([kgateway-dev/kgateway#12289](https://github.com/kgateway-dev/kgateway/issues/12289)).\nAs we got further into it, though, it made more sense to focus on\nagentgateway instead. Since agentgateway is the component acting as the\nsidecar proxy for EPP, benchmarking there gives us a more direct picture of\nthe impact. The work has been tracked in\n[agentgateway/agentgateway#85](https://github.com/agentgateway/agentgateway/issues/85)\nsince then.\n\nFor the benchmarking itself, we built on top of\n[llm-d-benchmark](https://github.com/llm-d/llm-d-benchmark) rather than\nputting together a separate harness. We reused its CLI and comparison\ntemplates, which let us get started quickly and kept the benchmarking\napproach aligned with the existing tooling. Working with it also surfaced\na few issues in the shared tools, some of which we were able to fix\nupstream.\n\nAgentgateway can run as EPP’s (Endpoint Picker’s) proxy sidecar, routing inference traffic to model servers based on live signals like KV-cache utilization and queue depth, instead of just round-robin. That kind of routing takes more work per request than plain round-robin does, but until now nobody had actually measured how much.\n\nThis project’s goal was to build a real, repeatable way to measure that — how much latency agentgateway adds over a plain Kubernetes Service with no gateway at all — and to make that measurement something the project can keep running over time, not a one-off number.\n\nThe numbers below come from GPU benchmark runs on 16 x H100 GPUs (Qwen/Qwen3-32B across 8 vLLM model servers with TP=2), comparing three setup options on the exact same cluster and hardware:\n\nHere is how the three setups compare under heavy load (60 QPS request-rate stage, zero request failures):\n\n| Metric | Kubernetes Service (RR) | Agentgateway Standalone | Agentgateway on Kubernetes |\n|---|---|---|---|\n| Peak output tokens/s | 6,910 | 16,178 (+134.1%) | 14,241 (+106.1%) |\n| Requests/sec | 6.70 | 16.52 (+146.5%) | 13.96 (+108.3%) |\n| TTFT p50 | 62.9s | 0.1s (-99.8%) | 0.2s (-99.7%) |\n| TTFT p90 | 135.6s | 0.2s (-99.8%) | 0.2s (-99.8%) |\n\nNotice that the latency gap isn’t a constant “proxy tax” — it opens up as request load increases. At light load (3 QPS), all three setups perform almost identically. The difference only shows up once traffic ramps up and the plain Kubernetes Service starts piling requests behind whichever pod round-robin happens to hit (its TTFT p50 jumps from 0.5s to 62.9s). By contrast, agentgateway with EPP continuously routes traffic to pods with available capacity.\n\nThe one metric where round-robin looks lower on paper — inter-token latency (30.3ms vs ~50ms p50 at 60 QPS) — is actually expected behavior. Because vLLM uses continuous batching, taking on more concurrent requests trades slightly higher per-token generation time for vastly higher total throughput. In short, higher inter-token latency under load is just vLLM keeping the GPUs fully saturated.\n\nRunning agentgateway as an in-cluster Kubernetes Gateway (via Gateway API HTTPRoute + InferencePool) shows the same pattern: slightly lower peak throughput than standalone sidecar mode (14,241 vs 16,178 tokens/s, but still double plain round-robin) while keeping TTFT sub-second under load:\n\nInstead of writing and maintaining custom benchmarking scripts, this uses [llm-d-benchmark](https://github.com/llm-d/llm-d-benchmark)’s tooling. A single command (`make -C controller benchmark`\n\n) spins up the test targets on a cluster, runs the load test through `inference-perf`\n\n, and generates comparison reports. It manages its own `llm-d-benchmark`\n\ncheckout automatically so there is no manual setup needed.\n\nWorking on this also turned up a few real upstream issues — a broken image tag in the `llm-d-router`\n\nrelease chart, and a missing CLI flag in `llm-d-benchmark`\n\n’s `standup`\n\ncommand (both filed, second one merged). While testing the GPU benchmark workflow locally, I caught another edge case: two guard clauses in the runner script were exiting with code 1 instead of returning 0 on the default Kind provider path, silently killing local runs. Because the upstream unit tests only covered the GKE branch, the issue went unnoticed — we patched it and verified the local Kind path end-to-end before merging.\n\nEverything below is real, merged or open work, not a summary of intentions:\n\n**agentgateway/agentgateway**\n\n**llm-d/llm-d-router**\n\n**llm-d/llm-d-benchmark**\n\n**llm-d/llm-d**\n\nThe core comparison between a plain Service and agentgateway is built, automated, and tested end-to-end. Daneyon Hansen expanded this into a full campaign-based benchmark system covering three treatments (plain Service, agentgateway standalone, and agentgateway on Kubernetes), complete with automated GKE cluster provisioning/teardown and the H100 GPU results above. That campaign system is merged into PR [#2526](https://github.com/agentgateway/agentgateway/pull/2526), along with the local Kind-path fix, which we verified on a real cluster before merging.\n\n**Future work: automated regression detection.** Right now, this benchmark runs on demand when triggered. The next step is a CI gate that compares a new run’s tail latency against a stored baseline and fails the build if it crosses a threshold — the same fail-hard limit pattern I used on `MaxSearchResults`\n\nin Jaeger’s MCP server earlier this year. I built and verified the core scripts (`check_regression.py`\n\n) locally against real project data, but keeping it off CI for now while we decide where the baseline data should live long-term (in-repo vs. cloud storage).\n\nThe thing that came up more than once was simple: check the current state before assuming you know what’s going on. I got burned by a stale local git clone twice this summer. One time, I filed an issue for something that had already been fixed upstream. Another time, I almost brought back a bug that had already been patched. Both times, the problem was trusting what I remembered or what my local code showed instead of checking what was actually happening right now.\n\nAnother thing I learned was not to treat changing direction as a failure. The benchmarking approach changed a few times during the project. We started with custom scripts, then moved to llm-d-benchmark’s tooling. The comparison went from three setups to two, and we eventually moved from a CPU simulator run to real GPU hardware. Those changes came out of feedback and what we were seeing as the work progressed. Nothing was really wasted, though. Each change built on what we’d already done and left us with something useful.\n\nThanks to my mentors, Nina Polshakova and Daneyon Hansen, for pointing this in the right direction through a few scope changes along the way, and to the llm-d-benchmark maintainers for tooling that meant I didn’t have to build a benchmarking harness from scratch.", "url": "https://wpnews.pro/news/measuring-agentgateway-s-overhead-as-an-inference-gateway", "canonical_source": "/blog/2026-08-20-benchmarking-agentgateway-epp-proxy-overhead/", "published_at": "2026-08-20 00:00:00+00:00", "updated_at": "2026-08-20 14:14:56.868107+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure"], "entities": ["Abhay Chaurasiya", "Nina Polshakova", "Daneyon Hansen", "CNCF", "agentgateway", "kgateway", "llm-d-benchmark", "vLLM"], "alternates": {"html": "https://wpnews.pro/news/measuring-agentgateway-s-overhead-as-an-inference-gateway", "markdown": "https://wpnews.pro/news/measuring-agentgateway-s-overhead-as-an-inference-gateway.md", "text": "https://wpnews.pro/news/measuring-agentgateway-s-overhead-as-an-inference-gateway.txt", "jsonld": "https://wpnews.pro/news/measuring-agentgateway-s-overhead-as-an-inference-gateway.jsonld"}}