{"slug": "benchmarking-agentgateway-vs-litellm-part-2-fixed-throughput", "title": "Benchmarking Agentgateway vs LiteLLM Part 2: Fixed Throughput", "summary": "AgentGateway outperformed LiteLLM in a fixed-throughput benchmark at 3,000 QPS, sustaining 2,998.94 QPS with sub-millisecond P99 latency while LiteLLM achieved only 2,465.89 QPS with 30.6 ms P99 latency. AgentGateway used 13% average CPU and 13 MB memory versus LiteLLM's 345% CPU and 11.67 GB memory, demonstrating significantly higher efficiency for high-throughput AI infrastructure.", "body_md": "In [Part 1](https://agentgateway.dev/blog/2026-06-26-benchmarking-agentgateway-vs-litellm/), I pushed both agentgateway and LiteLLM to their maximum throughput and compared latency, CPU, and memory usage. The downside of that benchmark was that each gateway was processing a very different workload. agentgateway handled roughly 10× the requests of LiteLLM so it wasn’t exactly an apples-to-apples comparison.\n\nIn this post, I use a **fixed target throughput of 3,000 QPS** for both gateways. This allows me to compare how efficiently each proxy handles the same traffic level.\n\nThe benchmark uses a very simple architecture. A mock LLM server immediately returns a fixed response so the benchmark measures **proxy overhead** rather than model inference time.\n\n[Fortio](https://fortio.org/) generates traffic against each gateway at the configured rate.\n\n```\nfortio (bt) ──► litellm :4000 ───────┐\n├──► mock-server (hyper-server) :8081\nfortio (bt) ──► agentgateway :4001 ──┘\n```\n\nI ran the benchmark with:\n\n```\n./scripts/run-benchmark.sh -q 3000 -d 30\n```\n\nThe benchmark uses:\n\n| Gateway | Actual Throughput | P50 | P90 | P99 |\n|---|---|---|---|---|\n| agentgateway | 2998.94 QPS |\n0.227 ms |\n0.249 ms |\n0.436 ms |\n| LiteLLM | 2465.89 QPS | 12.318 ms | 19.739 ms | 30.626 ms |\n\nA few observations immediately stand out.\n\nFirst, agentgateway sustained almost exactly the requested throughput of 3,000 QPS, processing 89,984 successful requests over the 30-second benchmark.\n\nLiteLLM never reached the target rate. It averaged 2,466 QPS, about 18% below the requested throughput, completing 74,008 requests during the same period.\n\nLatency was also dramatically different. agentgateway maintained a P99 latency below half a millisecond, while LiteLLM’s P99 exceeded 30 ms. Even median latency (P50) was over 50× lower with agentgateway.\n\n| Gateway | Avg CPU | Peak CPU | Avg Memory | Peak Memory |\n|---|---|---|---|---|\n| agentgateway | 13.4% | 29.5% | 13 MiB |\n17 MiB |\n| LiteLLM | 345.5% | 1158.5% | 11.67 GiB |\n11.69 GiB |\n\nThe resource utilization is arguably even more interesting than the latency numbers.\n\nAt essentially the same request rate, agentgateway used only 13% average CPU, while LiteLLM averaged 345% CPU, roughly 26× higher.\n\nMemory usage showed an even larger gap. agentgateway stayed around 13 MB throughout the test, whereas LiteLLM consumed nearly 12 GB of RAM.\n\nThis means agentgateway handled a higher request rate while using only a tiny fraction of the system resources.\n\n``` js\n./scripts/run-benchmark.sh -q 3000 -d 30\n==> Run ID: 20260626-165414\n==> LiteLLM workers: 18\n\nRunning fortio to litellm at 3000 QPS for 30s and 32 connections...\nqps: 2465.89qps p50: 12.318ms p90: 19.739ms p99: 30.626ms\n\nRunning fortio to agentgateway at 3000 QPS for 30s and 32 connections...\nqps: 2998.94qps p50: 0.227ms p90: 0.249ms p99: 0.436ms\n\nDEST,CLIENT,QPS,CONS,DUR,PAYLOAD,SUCCESS,THROUGHPUT,P50,P90,P99\nlitellm,fortio,3000,32,30,1104,74008,2465.89qps,12.318ms,19.739ms,30.626ms\nagentgateway,fortio,3000,32,30,1104,89984,2998.94qps,0.227ms,0.249ms,0.436ms\n\nCONTAINER,SAMPLES,AVG_CPU%,PEAK_CPU%,AVG_MEM,PEAK_MEM\nperf-agentgateway,21,13.42%,29.47%,13.15MiB,17.07MiB\nperf-litellm,21,345.51%,1158.47%,11.67GiB,11.69GiB\nperf-mock-server,21,6.23%,8.59%,3.06MiB,3.17MiB\n```\n\nI asked Cursor to turn the raw benchmark data into a few charts:\n\nCompared with the “maximum throughput” benchmark in Part 1, this test removes one important variable by targeting the same request rate for both gateways.\n\nEven under this controlled workload:\n\nLike the first benchmark, this test intentionally isolates proxy overhead by using a mock backend. It does **not** evaluate model inference latency or compare gateway features. If your application spends hundreds of milliseconds waiting for an LLM response, proxy latency becomes less significant.\n\nHowever, if you’re building high-throughput AI infrastructure, serving many concurrent requests, or simply want a lightweight local gateway to manage all of your LLM providers, proxy efficiency matters. In this benchmark, agentgateway consistently delivered lower latency while using substantially fewer CPU and memory resources.\n\nThe complete benchmark scripts and raw results are available in the GitHub [repository](https://github.com/linsun/litellm-agw-perf) if you’d like to reproduce the numbers yourself.", "url": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-part-2-fixed-throughput", "canonical_source": "https://agentgateway.dev/blog/2026-06-26-benchmarking-agentgateway-vs-litellm-part-2/", "published_at": "2026-06-26 01:00:00+00:00", "updated_at": "2026-07-13 13:25:29.551209+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["AgentGateway", "LiteLLM", "Fortio"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-part-2-fixed-throughput", "markdown": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-part-2-fixed-throughput.md", "text": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-part-2-fixed-throughput.txt", "jsonld": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-part-2-fixed-throughput.jsonld"}}