{"slug": "benchmarking-agentgateway-vs-litellm-s-rust-mode", "title": "Benchmarking Agentgateway vs LiteLLM's Rust Mode", "summary": "In benchmarks comparing proxy overhead, agentgateway 1.4.0 achieved 35,502 QPS with 0.863 ms P50 latency, while LiteLLM 1.98.0's Rust mode achieved 984 QPS with 32.139 ms P50 latency, making agentgateway over 36 times faster. The tests used a mock Anthropic server and Fortio traffic generator, with agentgateway consuming 26 MiB average memory versus LiteLLM's 2.15 GiB.", "body_md": "Since I published my benchmark of agentgateway vs. LiteLLM ([Part 1](/blog/2026-06-26-benchmarking-agentgateway-vs-litellm/) and [Part 2](/blog/2026-06-26-benchmarking-agentgateway-vs-litellm-part-2/)), I’ve received quite a few questions about LiteLLM’s newly added Rust mode. So I decided to compare its performance with agentgateway.\n\nRather than comparing features, I wanted to answer a simple question:\n\nHow much proxy overhead does LiteLLM’s Rust mode introduce compared with agentgateway?\n\nSimilar to my previous benchmarks, I wanted to measure:\n\nThe goal is to isolate the performance overhead introduced by each proxy.\n\nFollowing the [LiteLLM Rust mode documentation](https://docs.litellm.ai/docs/proxy/rust_gateway), the recommended way to enable Rust mode is to add `rust: true`\n\nto the LiteLLM configuration.\n\nAt the time of this benchmark, Rust mode did not support the OpenAI API path I was using, so I switched to a mock Anthropic model. Here is the relevant LiteLLM configuration:\n\n```\nmodel_list:\n - model_name: claude-mock  # for test litellm rust\n litellm_params:\n model: anthropic/claude-3-5-haiku-20241022\n api_base: http://mock-server:8081  # Anthropic-style base\n api_key: dummy\n rust: true\n```\n\nI also made sure I was using the latest LiteLLM build available for the test, LiteLLM `1.98.0`\n\n, and the new agentgateway version, `1.4.0`\n\n.\n\nFor the Anthropic benchmark, I used the `/v1/messages`\n\nAPI path, which was the supported path for the Rust mode configuration I tested.\n\nI then manually verified that Rust mode was actually enabled for the requests used in the benchmark.\n\nFor example:\n\n```\ncurl -sD - -o /dev/null http://127.0.0.1:4000/v1/messages \\\n -H \"Content-Type: application/json\" \\\n -d @payloads/req-anthropic-1024.json\n```\n\nThe response included:\n\n```\nx-litellm-version: 1.98.0\nx-litellm-rust: true\n```\n\nThe `x-litellm-rust: true`\n\nheader confirms that the request was handled by LiteLLM’s Rust mode.\n\nThe benchmark uses a very simple architecture. A mock LLM server immediately returns a fixed response so the benchmark measures **proxy overhead** rather than model inference time.\n\nI used [Fortio](https://fortio.org/) to generate traffic against each gateway.\n\n```\nfortio (bt) ──► litellm(rust:true):4000 ──┐\n├──► mock-server (hyper-server) :8081\nfortio (bt) ──► agentgateway:4001 ────────┘\n```\n\nFollow the [benchmark instructions](https://github.com/linsun/litellm-agw-perf#optional-anthropic-messages-mode-mock-server) to update the LiteLLM and agentgateway configurations for Anthropic and generate the corresponding request and response payloads.\n\nI first ran the benchmark using the default configuration while specifying the Anthropic API format:\n\n```\n./scripts/run-benchmark.sh -a anthropic\n```\n\nThe benchmark uses:\n\n`POST /v1/messages`\n\n(both gateways)`rust: true`\n\n, 2 workers, image with 1.98.0| Gateway | Throughput | P50 | P90 | P99 |\n|---|---|---|---|---|\n| agentgateway | 35,502 QPS |\n0.863 ms |\n1.644 ms |\n1.972 ms |\n| LiteLLM (rust: true) | 984 QPS | 32.139 ms | 48.528 ms | 71.451 ms |\n\nAgentgateway handled **over 36× more requests per second** while maintaining sub-2 ms P99 latency.\n\n| Gateway | Avg CPU | Peak CPU | Avg Memory | Peak Memory |\n|---|---|---|---|---|\n| agentgateway | 199% | 482% | 26 MiB |\n34 MiB |\n| LiteLLM (rust: true) | 69% | 204% | 2.15 GiB |\n2.15 GiB |\n\n``` js\n==> Run ID: 20260812-215223\n==> LiteLLM workers: 2\n==> API format: anthropic\n==> Checking LiteLLM Rust header\n x-litellm-rust: true\n...\nRunning fortio to litellm at 0 QPS for 3s and 32 connections...\nqps: 983.56qps p50: 32.139ms p90: 48.528ms p99: 71.451ms\nRunning fortio to agentgateway at 0 QPS for 3s and 32 connections...\nqps: 35501.62qps p50: 0.863ms p90: 1.644ms p99: 1.972ms\n\nDEST,CLIENT,QPS,CONS,DUR,PAYLOAD,SUCCESS,THROUGHPUT,P50,P90,P99\nlitellm,fortio,0,32,3,1114,2982,983.56qps,32.139ms,48.528ms,71.451ms\nagentgateway,fortio,0,32,3,1114,106525,35501.62qps,0.863ms,1.644ms,1.972ms\n\n==> CPU / memory\nPAYLOAD CONTAINER SAMPLES AVG_CPU% PEAK_CPU% AVG_MEM PEAK_MEM\n1024 perf-agentgateway 3 198.90% 482.13% 25.80MiB 34.31MiB\n1024 perf-litellm 3 68.53% 204.44% 2.15GiB 2.15GiB\n1024 perf-mock-server 3 18.18% 52.30% 3.05MiB 3.56MiB\n\n==> Checking LiteLLM Rust header\n x-litellm-rust: true\n```\n\nFull results: [github.com/linsun/litellm-agw-perf/results/20260812-215223](https://github.com/linsun/litellm-agw-perf/tree/main/results/20260812-215223)\n\nI asked Cursor to turn the raw benchmark data into charts:\n\nMaximum-throughput tests show the upper limit of each gateway, but they don’t provide an apples-to-apples comparison at the same request rate.\n\nSince LiteLLM reached approximately 983 QPS in the maximum-throughput test, I ran a second benchmark at a fixed target of **900 QPS**.\n\n```\n./scripts/run-benchmark.sh -a anthropic -q 900 -d 30\n```\n\nThe benchmark uses:\n\n`POST /v1/messages`\n\n(both gateways)`rust: true`\n\n, 2 workers, image with 1.98.0| Gateway | Actual Throughput | P50 | P90 | P99 |\n|---|---|---|---|---|\n| agentgateway | 898.95 QPS |\n0.474 ms |\n0.671 ms |\n1.447 ms |\n| LiteLLM (rust: true) | 898.42 QPS | 17.200 ms | 31.040 ms | 46.598 ms |\n\nBoth gateways sustained the target rate. Latency remained dramatically different: agentgateway’s P99 was **1.45 ms** versus **46.60 ms** for LiteLLM.\n\n| Gateway | Avg CPU | Peak CPU | Avg Memory | Peak Memory |\n|---|---|---|---|---|\n| agentgateway | 10.3% | 26.6% | 13 MiB |\n17 MiB |\n| LiteLLM (rust: true) | 97.2% | 204.9% | 2.14 GiB |\n2.15 GiB |\n\n``` js\n./scripts/run-benchmark.sh -a anthropic -q 900 -d 30\n==> Run ID: 20260813-115938\n==> LiteLLM workers: 2\n==> API format: anthropic\n==> Checking LiteLLM Rust header\n x-litellm-rust: true\n...\nRunning fortio to litellm at 900 QPS for 30s and 32 connections...\nqps: 898.42qps p50: 17.200ms p90: 31.040ms p99: 46.598ms\nRunning fortio to agentgateway at 900 QPS for 30s and 32 connections...\nqps: 898.95qps p50: 0.474ms p90: 0.671ms p99: 1.447ms\n\nDEST,CLIENT,QPS,CONS,DUR,PAYLOAD,SUCCESS,THROUGHPUT,P50,P90,P99\nlitellm,fortio,900,32,30,1114,26976,898.42qps,17.200ms,31.040ms,46.598ms\nagentgateway,fortio,900,32,30,1114,26976,898.95qps,0.474ms,0.671ms,1.447ms\n\n==> CPU / memory\nPAYLOAD CONTAINER SAMPLES AVG_CPU% PEAK_CPU% AVG_MEM PEAK_MEM\n1024 perf-agentgateway 21 10.34% 26.64% 13.15MiB 16.57MiB\n1024 perf-litellm 21 97.19% 204.85% 2.14GiB 2.15GiB\n1024 perf-mock-server 21 2.70% 4.23% 2.08MiB 2.24MiB\n```\n\nFull results: [github.com/linsun/litellm-agw-perf/results/20260813-115938](https://github.com/linsun/litellm-agw-perf/tree/main/results/20260813-115938)\n\nFor this benchmark, agentgateway introduced significantly less proxy overhead than LiteLLM’s Rust mode.\n\nAt maximum throughput, agentgateway delivered approximately:\n\nAt a fixed 900 QPS:\n\nThe key point is that the difference isn’t simply about maximum throughput. Even when both gateways are handling the **same 900 QPS workload**, the proxy overhead is substantially different, particularly in latency and memory consumption.\n\nThis benchmark intentionally isolates proxy performance by using a mock backend, so it doesn’t measure real LLM inference latency or feature completeness. If your workload is dominated by model inference, the differences will be less noticeable. However, if you’re building high-throughput AI services or running a local gateway that handles many concurrent requests, proxy overhead becomes much more important.\n\nThe complete benchmark scripts, configurations, and raw results are available in the GitHub [repository](https://github.com/linsun/litellm-agw-perf).\n\nIf you’d like to reproduce the numbers yourself, follow the instructions in the repository to run both the maximum-throughput and fixed-throughput benchmarks.", "url": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-s-rust-mode", "canonical_source": "/blog/2026-08-13-benchmarking-agentgateway-vs-litellm-rust-mode/", "published_at": "2026-08-17 00:00:00+00:00", "updated_at": "2026-08-17 17:40:51.631393+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools"], "entities": ["agentgateway", "LiteLLM", "Fortio", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-s-rust-mode", "markdown": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-s-rust-mode.md", "text": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-s-rust-mode.txt", "jsonld": "https://wpnews.pro/news/benchmarking-agentgateway-vs-litellm-s-rust-mode.jsonld"}}