{"slug": "is-litellm-rust-really-7x-faster-a-closer-look-at-the-numbers", "title": "Is LiteLLM Rust Really 7x Faster? A Closer Look at the Numbers", "summary": "The Bifrost team re-examined LiteLLM's published benchmark files for its Rust AI gateway and concluded the raw data does not support LiteLLM's headline claims of roughly 7x lower overhead and 9x less memory than Bifrost. The review found the 4.5 ms Bifrost overhead figure behind the \"7x\" claim does not appear in LiteLLM's latency sweep, that the \"memory at rest\" chart actually reports peak RSS using the widest of three memory pairs, and that Bifrost's logging plugin may have been left enabled during testing. LiteLLM's own post acknowledges the benchmark is vendor-run and that streaming and full features were still landing.", "body_md": "Benchmark headlines are easy to share and hard to check.\n\nIn July, LiteLLM published benchmarks for its new Rust AI gateway. The headline numbers were big: **roughly 7x lower overhead and 9x less memory than Bifrost**, plus about **4.5x more requests per dollar**. For anyone choosing an LLM gateway, numbers like that make the decision sound easy.\n\nThe Bifrost team then went through LiteLLM's raw CSV and JSON files and published a response: [Debunking LiteLLM Rust Benchmarking, Part 1](https://www.getmaxim.ai/bifrost/blog/debunking-litellm-rust-benchmarking-part-1). Their conclusion is that the raw data tells a much less dramatic story than the headlines.\n\nI've written before about [why I switched from LiteLLM to Bifrost](https://dev.to/therealmrmumba/why-i-switched-from-litellm-to-bifrost-for-production-llm-systems-45kf) for production systems, so I was curious. In this article, I'll walk through what LiteLLM claimed, what the raw numbers actually show, where each side has a fair point, and how you can check the results yourself.\n\nI'll cite the numbers as published and point out the caveats on both sides.\n\nLiteLLM's [benchmark post](https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks), published in July 2026, compared four gateways: the LiteLLM Rust beta, LiteLLM Python v1, Bifrost v1.6.4, and Portkey OSS. All of them pointed at the same local mock provider, so the numbers reflect each gateway's own overhead rather than real model latency.\n\nThe headline results looked like this:\n\n| Metric | LiteLLM Rust | Bifrost | Portkey | LiteLLM Python | \n|---|---|---|---|---|\n| p99 added latency | ~0.7 ms | 4.5 ms | 2.3 ms | 257.7 ms | \n| Peak memory | 21.8 MB | 199.1 MB | 90.4 MB | 329.5 MB | \n| Est. cost per 1M requests | ~$0.000175 | $0.001008 | $0.001042 | $0.015354 | \n| Requests per dollar-hour | ~283,833 | 63,246 | n/a | n/a | \n\nFrom those, LiteLLM's site and blog drew four big claims:\n\nTo be fair to LiteLLM, its own post includes some caveats. It says the benchmark is vendor-run, that streaming and the full feature set are still landing, and that readers should treat the differences as order-of-magnitude.\n\nInstead of running a new benchmark first, the Bifrost team went back to LiteLLM's own published files in the `ai-gateway-bench` repo and checked whether the headlines matched the data. Here are the main issues they raised.\n\nThe 4.5 ms Bifrost overhead figure used for the \"7x\" claim doesn't show up in any row of LiteLLM's latency sweep. In the sweep itself, Bifrost and LiteLLM Rust trade places depending on concurrency (more on that below).\n\nThe homepage line about \"3.5x lower overhead than the next AI gateway\" compares against Portkey (2.29 ms vs 0.66 ms). Easy to miss, and easy to misread as a Bifrost comparison.\n\nThe chart is labeled \"memory at rest\" and \"idle,\" but the underlying column is `peak_rss_mb`. LiteLLM's files also contain three different memory pairs, and the headline uses the widest one (199.1 MB vs 21.8 MB). The other pairs are 144 MB vs 26.1 MB, and 282 to 298 MB vs 35 to 36 MB.\n\nLiteLLM says logging, spend tracking, and persistence were disabled for every gateway. But according to the Bifrost team's reading, the Bifrost config in the repo doesn't actually set [`enable_logging`](https://docs.getbifrost.ai/deployment-guides/config-json/client) to false, and Bifrost's [logging plugin is enabled by default in gateway mode](https://docs.getbifrost.ai/features/observability/default). So Bifrost was carrying logging overhead (including SQLite memory) that the Rust gateway wasn't. The Bifrost team notes this reading is based on newer code and not yet confirmed on v1.6.4.\n\nThe benchmark used **Bifrost v1.6.4**. Since then, Bifrost has shipped v1.6.5 to v1.6.11 and v2.0.0 to v2.4.0, and the [Bifrost changelog](https://docs.getbifrost.ai/changelogs) now lists v2.5.1. The Bifrost team says 2.x substantially reduced memory use, though this post doesn't measure that yet.\n\nThe hardware, core counts, worker counts, and mock latency aren't published, so \"identical hardware\" can't be verified. Each result also appears to be a single run with no error bars, and the Bifrost team says rankings flip between files.\n\nThe Bifrost post points out that LiteLLM's Rust migration is still in progress. By source bytes, the repo is 3.11% Rust, and LiteLLM's own tracker reports 24% of major APIs built as of v1.107.0-rc.2. In the Bifrost team's words, the benchmark measures \"a narrower Rust beta, not the full LiteLLM product rewritten in Rust.\"\n\nThese figures come straight from LiteLLM's own [`results/latency_vs_concurrency.csv`](https://github.com/BerriAI/ai-gateway-bench/blob/main/results/latency_vs_concurrency.csv) in the `ai-gateway-bench` repo. I checked every value below against it. They match the figures in the [Bifrost post](https://www.getmaxim.ai/bifrost/blog/debunking-litellm-rust-benchmarking-part-1), and every run had **zero errors**.\n\n| Concurrency | LiteLLM Rust | Bifrost | LiteLLM Python | \n|---|---|---|---|\n| 1 | +0.89 | **+0.75** | +9.79 | \n| 4 | **+0.61** | +0.63 | +26.95 | \n| 16 | **+0.33** | +0.89 | +251.34 | \n| 64 | +40.28 | **+28.68** | +546.04 | \n\nThe two trade places: Bifrost is lower at concurrency 1 and 64, LiteLLM Rust at 4 and 16. At concurrency 4, the gap is just 0.02 ms. That is a long way from \"7x.\"\n\n| Concurrency | Direct | LiteLLM Rust | Bifrost | LiteLLM Python | \n|---|---|---|---|---|\n| 1 | 46.2 | 45.2 | 44.8 | 34.4 | \n| 4 | 184.5 | 181.2 | 180.2 | 92.7 | \n| 16 | 735.6 | 721.8 | 722.7 | 158.4 | \n| 64 | 2,855.7 | 2,722.2 | 2,719.1 | 179.8 | \n\nAt the highest tested concurrency, Rust and Bifrost are separated by about **0.1%**. For practical purposes, that's a tie.\n\nOne caveat before the numbers: they come from a setup where Bifrost was likely running with logging on and Rust wasn’t, on a Bifrost version that’s several releases old.\n\nThe Bifrost post reports these results openly, and I think they’re worth repeating:\n\nThe 4.5x requests-per-dollar figure and the cost per 1M requests are **calculated**, not measured end to end. They're based on an assumed 4 vCPU / 16 GB instance and the CPU and memory numbers above. So if the memory comparison isn't like-for-like, the cost comparison inherits the same problem.\n\nThe best way to settle a benchmark argument is to run it on your own hardware. LiteLLM's benchmark lives in the [`BerriAI/ai-gateway-bench`](https://github.com/BerriAI/ai-gateway-bench) repo, and its blog outlines the basic flow:\n\n```\n# 1. Clone the repo and set up the environment\ngit clone https://github.com/BerriAI/ai-gateway-bench.git\ncd ai-gateway-bench\npython -m venv .venv && source .venv/bin/activate\npip install -r requirements.txt\n\n# 2. Start the mock upstream provider\ncargo run --release -p mock-upstream\n\n# 3. Start each gateway pointed at the mock\n#    (follow gateways/<name>/README.md for each one)\n\n# 4. Run a scenario, then regenerate the charts\npython analyze/make_chart.py\n```\n\nIf you run it, these are the changes that make the comparison fair, based on the gaps the Bifrost team found:\n\nThat's also exactly what the Bifrost team says it will do in Part 2: same benchmarks, current build, logging on and off, repeated trials, and full hardware disclosure.\n\nThis back-and-forth is a useful reminder that a benchmark is only as good as its setup. Whatever gateway you're evaluating, these are the questions I'd ask before trusting a chart:\n\nAnd most importantly: **benchmark the features you'll actually use.** A gateway with logging, budgets, fallbacks, and caching turned on behaves very differently from a stripped-down proxy forwarding requests to a mock.\n\nLiteLLM's Rust gateway is a real step forward from LiteLLM Python. The jump from hundreds of milliseconds of added latency to under a millisecond is impressive, and its memory and CPU footprint is genuinely small.\n\nBut the headline comparisons against Bifrost don't hold up well against LiteLLM's own raw data. On latency, the two trade places depending on concurrency. On throughput, they're within about 0.1% at the top tested load. The bigger memory and cost gaps come from a setup that compared an old Bifrost version, likely with logging on, against a Rust beta with logging off.\n\nSo instead of \"7x\" and \"9x,\" the honest summary is closer to this: **on the tested setup, Bifrost and LiteLLM Rust perform about the same on latency and throughput, and Rust uses less memory and CPU in a configuration that wasn't like-for-like.**\n\nI'm looking forward to Part 2, where the Bifrost team plans to rerun everything with a current build and full disclosure. Until then, the best advice is the simplest one: don't pick a gateway from a headline. Run the benchmark on your own workload, with the features you'll actually use.\n\nIf you want to try Bifrost yourself, it's open source and you can find it at [getmaxim.ai/bifrost](https://www.getmaxim.ai/bifrost).", "url": "https://wpnews.pro/news/is-litellm-rust-really-7x-faster-a-closer-look-at-the-numbers", "canonical_source": "https://dev.to/therealmrmumba/is-litellm-rust-really-7x-faster-a-closer-look-at-the-numbers-497m", "published_at": "2026-11-11 12:05:26+00:00", "updated_at": "2026-11-11 12:24:09.328633+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "ai-agents"], "entities": ["LiteLLM", "Bifrost", "Maxim AI", "Portkey"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/is-litellm-rust-really-7x-faster-a-closer-look-at-the-numbers", "markdown": "https://wpnews.pro/news/is-litellm-rust-really-7x-faster-a-closer-look-at-the-numbers.md", "text": "https://wpnews.pro/news/is-litellm-rust-really-7x-faster-a-closer-look-at-the-numbers.txt", "jsonld": "https://wpnews.pro/news/is-litellm-rust-really-7x-faster-a-closer-look-at-the-numbers.jsonld"}}