Is LiteLLM Rust Really 7x Faster? A Closer Look at the Numbers

The Bifrost team re-examined LiteLLM's published benchmark files for its Rust AI gateway and concluded the raw data does not support LiteLLM's headline claims of roughly 7x lower overhead and 9x less memory than Bifrost. The review found the 4.5 ms Bifrost overhead figure behind the "7x" claim does not appear in LiteLLM's latency sweep, that the "memory at rest" chart actually reports peak RSS using the widest of three memory pairs, and that Bifrost's logging plugin may have been left enabled during testing. LiteLLM's own post acknowledges the benchmark is vendor-run and that streaming and full features were still landing.

Benchmark headlines are easy to share and hard to check. In July, LiteLLM published benchmarks for its new Rust AI gateway. The headline numbers were big: roughly 7x lower overhead and 9x less memory than Bifrost , plus about 4.5x more requests per dollar . For anyone choosing an LLM gateway, numbers like that make the decision sound easy. The Bifrost team then went through LiteLLM's raw CSV and JSON files and published a response: Debunking LiteLLM Rust Benchmarking, Part 1 https://www.getmaxim.ai/bifrost/blog/debunking-litellm-rust-benchmarking-part-1 . Their conclusion is that the raw data tells a much less dramatic story than the headlines. I've written before about why I switched from LiteLLM to Bifrost https://dev.to/therealmrmumba/why-i-switched-from-litellm-to-bifrost-for-production-llm-systems-45kf for production systems, so I was curious. In this article, I'll walk through what LiteLLM claimed, what the raw numbers actually show, where each side has a fair point, and how you can check the results yourself. I'll cite the numbers as published and point out the caveats on both sides. LiteLLM's benchmark post https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks , published in July 2026, compared four gateways: the LiteLLM Rust beta, LiteLLM Python v1, Bifrost v1.6.4, and Portkey OSS. All of them pointed at the same local mock provider, so the numbers reflect each gateway's own overhead rather than real model latency. The headline results looked like this: | Metric | LiteLLM Rust | Bifrost | Portkey | LiteLLM Python | |---|---|---|---|---| | p99 added latency | ~0.7 ms | 4.5 ms | 2.3 ms | 257.7 ms | | Peak memory | 21.8 MB | 199.1 MB | 90.4 MB | 329.5 MB | | Est. cost per 1M requests | ~$0.000175 | $0.001008 | $0.001042 | $0.015354 | | Requests per dollar-hour | ~283,833 | 63,246 | n/a | n/a | From those, LiteLLM's site and blog drew four big claims: To be fair to LiteLLM, its own post includes some caveats. It says the benchmark is vendor-run, that streaming and the full feature set are still landing, and that readers should treat the differences as order-of-magnitude. Instead of running a new benchmark first, the Bifrost team went back to LiteLLM's own published files in the ai-gateway-bench repo and checked whether the headlines matched the data. Here are the main issues they raised. The 4.5 ms Bifrost overhead figure used for the "7x" claim doesn't show up in any row of LiteLLM's latency sweep. In the sweep itself, Bifrost and LiteLLM Rust trade places depending on concurrency more on that below . The homepage line about "3.5x lower overhead than the next AI gateway" compares against Portkey 2.29 ms vs 0.66 ms . Easy to miss, and easy to misread as a Bifrost comparison. The chart is labeled "memory at rest" and "idle," but the underlying column is peak rss mb . LiteLLM's files also contain three different memory pairs, and the headline uses the widest one 199.1 MB vs 21.8 MB . The other pairs are 144 MB vs 26.1 MB, and 282 to 298 MB vs 35 to 36 MB. LiteLLM says logging, spend tracking, and persistence were disabled for every gateway. But according to the Bifrost team's reading, the Bifrost config in the repo doesn't actually set enable logging https://docs.getbifrost.ai/deployment-guides/config-json/client to false, and Bifrost's logging plugin is enabled by default in gateway mode https://docs.getbifrost.ai/features/observability/default . So Bifrost was carrying logging overhead including SQLite memory that the Rust gateway wasn't. The Bifrost team notes this reading is based on newer code and not yet confirmed on v1.6.4. The benchmark used Bifrost v1.6.4 . Since then, Bifrost has shipped v1.6.5 to v1.6.11 and v2.0.0 to v2.4.0, and the Bifrost changelog https://docs.getbifrost.ai/changelogs now lists v2.5.1. The Bifrost team says 2.x substantially reduced memory use, though this post doesn't measure that yet. The hardware, core counts, worker counts, and mock latency aren't published, so "identical hardware" can't be verified. Each result also appears to be a single run with no error bars, and the Bifrost team says rankings flip between files. The Bifrost post points out that LiteLLM's Rust migration is still in progress. By source bytes, the repo is 3.11% Rust, and LiteLLM's own tracker reports 24% of major APIs built as of v1.107.0-rc.2. In the Bifrost team's words, the benchmark measures "a narrower Rust beta, not the full LiteLLM product rewritten in Rust." These figures come straight from LiteLLM's own results/latency vs concurrency.csv https://github.com/BerriAI/ai-gateway-bench/blob/main/results/latency vs concurrency.csv in the ai-gateway-bench repo. I checked every value below against it. They match the figures in the Bifrost post https://www.getmaxim.ai/bifrost/blog/debunking-litellm-rust-benchmarking-part-1 , and every run had zero errors . | Concurrency | LiteLLM Rust | Bifrost | LiteLLM Python | |---|---|---|---| | 1 | +0.89 | +0.75 | +9.79 | | 4 | +0.61 | +0.63 | +26.95 | | 16 | +0.33 | +0.89 | +251.34 | | 64 | +40.28 | +28.68 | +546.04 | The two trade places: Bifrost is lower at concurrency 1 and 64, LiteLLM Rust at 4 and 16. At concurrency 4, the gap is just 0.02 ms. That is a long way from "7x." | Concurrency | Direct | LiteLLM Rust | Bifrost | LiteLLM Python | |---|---|---|---|---| | 1 | 46.2 | 45.2 | 44.8 | 34.4 | | 4 | 184.5 | 181.2 | 180.2 | 92.7 | | 16 | 735.6 | 721.8 | 722.7 | 158.4 | | 64 | 2,855.7 | 2,722.2 | 2,719.1 | 179.8 | At the highest tested concurrency, Rust and Bifrost are separated by about 0.1% . For practical purposes, that's a tie. One caveat before the numbers: they come from a setup where Bifrost was likely running with logging on and Rust wasn’t, on a Bifrost version that’s several releases old. The Bifrost post reports these results openly, and I think they’re worth repeating: The 4.5x requests-per-dollar figure and the cost per 1M requests are calculated , not measured end to end. They're based on an assumed 4 vCPU / 16 GB instance and the CPU and memory numbers above. So if the memory comparison isn't like-for-like, the cost comparison inherits the same problem. The best way to settle a benchmark argument is to run it on your own hardware. LiteLLM's benchmark lives in the BerriAI/ai-gateway-bench https://github.com/BerriAI/ai-gateway-bench repo, and its blog outlines the basic flow: 1. Clone the repo and set up the environment git clone https://github.com/BerriAI/ai-gateway-bench.git cd ai-gateway-bench python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt 2. Start the mock upstream provider cargo run --release -p mock-upstream 3. Start each gateway pointed at the mock follow gateways/<name /README.md for each one 4. Run a scenario, then regenerate the charts python analyze/make chart.py If you run it, these are the changes that make the comparison fair, based on the gaps the Bifrost team found: That's also exactly what the Bifrost team says it will do in Part 2: same benchmarks, current build, logging on and off, repeated trials, and full hardware disclosure. This back-and-forth is a useful reminder that a benchmark is only as good as its setup. Whatever gateway you're evaluating, these are the questions I'd ask before trusting a chart: And most importantly: benchmark the features you'll actually use. A gateway with logging, budgets, fallbacks, and caching turned on behaves very differently from a stripped-down proxy forwarding requests to a mock. LiteLLM's Rust gateway is a real step forward from LiteLLM Python. The jump from hundreds of milliseconds of added latency to under a millisecond is impressive, and its memory and CPU footprint is genuinely small. But the headline comparisons against Bifrost don't hold up well against LiteLLM's own raw data. On latency, the two trade places depending on concurrency. On throughput, they're within about 0.1% at the top tested load. The bigger memory and cost gaps come from a setup that compared an old Bifrost version, likely with logging on, against a Rust beta with logging off. So instead of "7x" and "9x," the honest summary is closer to this: on the tested setup, Bifrost and LiteLLM Rust perform about the same on latency and throughput, and Rust uses less memory and CPU in a configuration that wasn't like-for-like. I'm looking forward to Part 2, where the Bifrost team plans to rerun everything with a current build and full disclosure. Until then, the best advice is the simplest one: don't pick a gateway from a headline. Run the benchmark on your own workload, with the features you'll actually use. If you want to try Bifrost yourself, it's open source and you can find it at getmaxim.ai/bifrost https://www.getmaxim.ai/bifrost .