{"slug": "benchmarking-agentgateway-vs-litellm", "title": "Benchmarking Agentgateway vs LiteLLM", "summary": "A benchmark comparing agentgateway and LiteLLM found agentgateway handled over 11× more requests per second (36,933 QPS vs 3,198 QPS) with sub-2 ms P99 latency while consuming under 30 MB of RAM, versus LiteLLM's nearly 12 GB. The test used a mock LLM server to isolate proxy overhead, and agentgateway's virtual models feature enables local management of LLM API keys with usage visibility.", "body_md": "One of the coolest features in the agentgateway 1.3 release is **Virtual Models**. It made me realize I could use agentgateway locally to manage all of my LLM API keys while also getting visibility into my LLM usage and costs.\n\nI’ve heard good things about LiteLLM for quite a while, so I decided to compare its performance with agentgateway. My goal is to find a lightweight gateway that can handle very high throughput with minimal latency. (I occasionally ask LLMs to generate videos, which results in a lot of API traffic.)\n\nInstead of comparing features, I wanted to answer a simple question:\n\nHow much overhead does each proxy introduce under load?\n\nSpecifically, I measured:\n\nThe benchmark uses a very simple architecture. A mock LLM server immediately returns a fixed response so the benchmark measures **proxy overhead** rather than model inference time.\n\n[Fortio](https://fortio.org/) generates as much traffic as possible against each gateway.\n\n```\nfortio (bt) ──► litellm :4000 ───────┐\n├──► mock-server (hyper-server) :8081\nfortio (bt) ──► agentgateway :4001 ──┘\n```\n\nI ran the benchmark using the default configuration:\n\n```\n./scripts/run-benchmark.sh\n```\n\nThe benchmark uses:\n\n| Gateway | Throughput | P50 | P90 | P99 |\n|---|---|---|---|---|\n| agentgateway | 36,933 QPS |\n0.831 ms |\n1.533 ms |\n1.970 ms |\n| LiteLLM | 3,198 QPS | 7.076 ms | 17.986 ms | 32.192 ms |\n\nagentgateway handled **over 11× more requests per second** while maintaining sub-2 ms P99 latency.\n\n| Gateway | Avg CPU | Peak CPU | Avg Memory | Peak Memory |\n|---|---|---|---|---|\n| agentgateway | 105% | 243% | 22 MB |\n29 MB |\n| LiteLLM | 331% | 1075% | 11.8 GB |\n11.8 GB |\n\nThe memory difference was the biggest surprise. During this benchmark, LiteLLM consumed nearly **12 GB of RAM**, while agentgateway stayed below **30 MB**.\n\n``` js\n./scripts/run-benchmark.sh\n==> Run ID: 20260626-120716\n==> LiteLLM workers: 18\n...\nRunning fortio to litellm at 0 QPS for 3s and 32 connections...\nqps: 3198.48qps p50: 7.076ms p90: 17.986ms p99: 32.192ms\n...\nRunning fortio to agentgateway at 0 QPS for 3s and 32 connections...\nqps: 36933.62qps p50: 0.831ms p90: 1.533ms p99: 1.970ms\n...\nDEST,CLIENT,QPS,CONS,DUR,PAYLOAD,SUCCESS,THROUGHPUT,P50,P90,P99\nlitellm,fortio,0,32,3,1104,10186,3198.48qps,7.076ms,17.986ms,32.192ms\nagentgateway,fortio,0,32,3,1104,110831,36933.62qps,0.831ms,1.533ms,1.970ms\n...\nCONTAINER,SAMPLES,AVG_CPU%,PEAK_CPU%,AVG_MEM,PEAK_MEM\nperf-agentgateway,4,104.80%,243.02%,22.38MiB,28.79MiB\nperf-litellm,4,330.79%,1075.38%,11.81GiB,11.81GiB\nperf-mock-server,4,24.52%,81.05%,3.43MiB,3.84MiB\n```\n\nI asked Cursor to turn the raw benchmark data into a few charts:\n\nFor this benchmark, agentgateway introduced significantly less proxy overhead than LiteLLM:\n\nThis benchmark intentionally isolates proxy performance by using a mock backend, so it doesn’t measure real LLM inference latency or feature completeness. If your workload is dominated by model inference, the differences will be less noticeable. However, if you’re building high-throughput AI services or running a local gateway that handles many concurrent requests, proxy overhead becomes much more important.\n\nBased on these results, I’m going to switch my local setup to agentgateway and use it to manage all of my LLM traffic.\n\nBecause each gateway handled a different workload in this test, I followed up with [Part 2](https://agentgateway.dev/blog/2026-06-26-benchmarking-agentgateway-vs-litellm-part-2/) using a fixed 3,000 QPS target for both gateways to make the comparison more apples-to-apples.\n\nThe complete benchmark scripts and raw results are available in the GitHub [repository](https://github.com/linsun/litellm-agw-perf) if you’d like to reproduce the numbers yourself.", "url": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm", "canonical_source": "https://agentgateway.dev/blog/2026-06-26-benchmarking-agentgateway-vs-litellm/", "published_at": "2026-06-26 00:00:00+00:00", "updated_at": "2026-07-13 13:25:35.777013+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "ai-tools"], "entities": ["agentgateway", "LiteLLM", "Fortio"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm", "markdown": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm.md", "text": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm.txt", "jsonld": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm.jsonld"}}