If you've been using ray-project/llmperf, you may have noticed it's now in archive mode. No new updates, no fixes, no responses to issues. If you're evaluating it for the first time, that's worth knowing before you build anything on top of it. This page is part of my LLM benchmarking guide, which covers the metrics themselves (TTFT, ITL, throughput). Here I want to focus on the tools — what's out there now that llmperf is effectively done, and what I ended up building.
Credit where it's due — llmperf was the go-to open-source option for benchmarking OpenAI-compatible endpoints. It measured the metrics that matter (TTFT, ITL, throughput), handled concurrency, and came out of the Anyscale/Ray team, so it had credibility. For a lot of teams it did the job well.
Two things pushed me to look at alternatives, and both are about fit rather than flaws:
ITL aggregation. llmperf computes Inter-Token Latency by averaging within each request first, then aggregating those per-request averages. That's a reasonable choice and works well for many use cases. But I was specifically trying to catch latency spikes during the decode phase, and per-request averaging smooths exactly those out. I needed the raw distribution.
Startup overhead. Runs spawn Ray workers, so there's a meaningful spin-up cost before the first request fires. When I just want to poke at an endpoint quickly, that's more ceremony than I want — I was after something closer to curl than a cluster setup.
Neither is a flaw. They're design decisions that matched llmperf's goals and didn't match mine.
| Tool | Single binary / low deps | GPU-level metrics | Distributed | Notes |
|---|---|---|---|---|
There's also a note for genai-perf: NVIDIA sunsetted it and moved development to aiperf.
That last one is the gap I was trying to fill — something I could drop onto a box and run in seconds, that preserved raw ITL values so spikes weren't hidden.
llmperf-rs is a single Rust binary that benchmarks any OpenAI-compatible endpoint (vLLM, Ollama, local APIs). It:
usage
field when available, falling back to a tokenizer you specify. The original llmperf used one tokenizer for everything, which gets inaccurate across model families.It's not trying to compete with the GPU-deep tools. I think of it as one level above curl — fast to start, low dependency, good enough for most "how's this endpoint doing" questions.
If that trade-off sounds right for you: grab it from the releases page, or cargo install --git https://github.com/wheynelau/llmperf-rs
.
llmperf being archived doesn't mean the tooling disappeared — aiperf, vllm-bench, GuideLLM, and others are all actively maintained. The choice mostly comes down to how deep you need to go (GPU metrics vs endpoint metrics) and how much setup you're willing to tolerate.
You can find this post and more on my blog.