Sending a few curl requests or writing a quick asyncio script isn't enough to tell if a deployment is actually fast. The problem with those "vibe checks" is that you hit single-process limits or Python's GIL long before you actually stress the model, meaning your benchmarks are lying to you. I started using AIPerf to get actual numbers because it handles the concurrency side without the usual bottlenecks.
Why my manual scripts failed #
I tried to build a custom load generator to test my setup, but I kept running into a wall where the client-side overhead was higher than the model's latency. If you are just using a basic loop, you aren't measuring the model; you're measuring how fast your local machine can push strings over HTTP. AIPerf solves this by being designed for scale, meaning it can actually saturate the inference engine to find the real breaking point.
How to run a basic benchmark #
If you want to see where your system actually peaks, you need to move away from one-off scripts. Here is the general workflow for getting a baseline:
-
Install the tool and point it at your endpoint.
-
Define your request load—don't just send one prompt; send a stream of them to see when the tokens per second (TPS) start to drop.
-
Monitor the time to first token (TTFT) and the overall throughput.
The goal is to find the saturation point. Once the TPS plateaus while the request rate increases, you've found your hardware limit.
Common bottlenecks I noticed #
While running these tests, I realized that "fast" is relative. If you're only looking at total request time, you're missing the most important part of the LLM experience: the perceived speed. AIPerf breaks this down so you can see if a slow response is due to a massive TTFT (the model thinking before it starts) or a slow generation speed (the model typing slowly).
If you're seeing a spike in latency but the TPS is steady, it's usually a queuing issue at the server level, not a compute limit. This is something you'd never catch with a simple curl command because you aren't putting enough concurrent pressure on the system to trigger the queue.
The takeaway for deployment #
Stop relying on "it feels fast" or a few manual prompts. You need a tool that can simulate actual production traffic to know if your instance size is correct or if you're overpaying for compute you aren't even saturating. For anyone deploying at scale, the only metric that matters is the point where latency degrades under load, and that requires a dedicated benchmarking tool rather than a hand-rolled Python script.
Next Can Dario Amodei actually pace the AI frontier with these proposals? →
All Replies （3） #
Finally, some sanity. I wasted a week using Locust and still missed the bottleneck. Does this actually handle KV cache spikes?
This burned me during my last scale test. You didn't mention how it handles request concurrency vs. queue depth for 4096 tokens.
I want to try this tonight. Does it support vLLM specifically, or are you using Triton for the load?