{"slug": "aiperf-is-the-only-way-i-ve-found-to-get-real-llm-inference-benchmarks", "title": "AIPerf is the only way I've found to get real LLM inference benchmarks", "summary": "AIPerf is a dedicated benchmarking tool that measures LLM inference performance by generating concurrent request loads, according to a first-person account of using it to test deployments. The account states that manual curl requests and asyncio scripts hit single-process limits and Python's GIL before stressing the model, and that AIPerf instead reports time to first token (TTFT) and tokens per second (TPS) to identify the saturation point where TPS plateaus as request rate rises. The author recommends the tool over hand-rolled Python scripts for determining whether an instance size is correct or compute is being overpaid for.", "body_md": "# AIPerf is the only way I've found to get real LLM inference benchmarks\n\nSending a few curl requests or writing a quick asyncio script isn't enough to tell if a deployment is actually fast. The problem with those \"vibe checks\" is that you hit single-process limits or Python's GIL long before you actually stress the model, meaning your benchmarks are lying to you. I started using AIPerf to get actual numbers because it handles the concurrency side without the usual bottlenecks.\n\n## Why my manual scripts failed\n\nI tried to build a custom load generator to test my setup, but I kept running into a wall where the client-side overhead was higher than the model's latency. If you are just using a basic loop, you aren't measuring the model; you're measuring how fast your local machine can push strings over HTTP. AIPerf solves this by being designed for scale, meaning it can actually saturate the inference engine to find the real breaking point.\n\n## How to run a basic benchmark\n\nIf you want to see where your system actually peaks, you need to move away from one-off scripts. Here is the general workflow for getting a baseline:\n\n1. Install the tool and point it at your endpoint.\n\n2. Define your request load—don't just send one prompt; send a stream of them to see when the tokens per second (TPS) start to drop.\n\n3. Monitor the time to first token (TTFT) and the overall throughput.\n\nThe goal is to find the saturation point. Once the TPS plateaus while the request rate increases, you've found your hardware limit.\n\n## Common bottlenecks I noticed\n\nWhile running these tests, I realized that \"fast\" is relative. If you're only looking at total request time, you're missing the most important part of the LLM experience: the perceived speed. AIPerf breaks this down so you can see if a slow response is due to a massive TTFT (the model thinking before it starts) or a slow generation speed (the model typing slowly).\n\nIf you're seeing a spike in latency but the TPS is steady, it's usually a queuing issue at the server level, not a compute limit. This is something you'd never catch with a simple curl command because you aren't putting enough concurrent pressure on the system to trigger the queue.\n\n## The takeaway for deployment\n\nStop relying on \"it feels fast\" or a few manual prompts. You need a tool that can simulate actual production traffic to know if your instance size is correct or if you're overpaying for compute you aren't even saturating. For anyone deploying at scale, the only metric that matters is the point where latency degrades under load, and that requires a dedicated benchmarking tool rather than a hand-rolled Python script.\n\n[Next Can Dario Amodei actually pace the AI frontier with these proposals? →](https://promptcube3.com/en/threads/9511/)\n\n## All Replies （3）\n\nFinally, some sanity. I wasted a week using Locust and still missed the bottleneck. Does this actually handle KV cache spikes?\n\nThis burned me during my last scale test. You didn't mention how it handles request concurrency vs. queue depth for 4096 tokens.\n\nI want to try this tonight. Does it support vLLM specifically, or are you using Triton for the load?", "url": "https://wpnews.pro/news/aiperf-is-the-only-way-i-ve-found-to-get-real-llm-inference-benchmarks", "canonical_source": "https://promptcube3.com/en/threads/9524/", "published_at": "2026-09-19 17:01:54+00:00", "updated_at": "2026-09-19 17:22:57.437805+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["AIPerf", "Python", "Locust", "vLLM", "Triton"], "alternates": {"html": "https://wpnews.pro/news/aiperf-is-the-only-way-i-ve-found-to-get-real-llm-inference-benchmarks", "markdown": "https://wpnews.pro/news/aiperf-is-the-only-way-i-ve-found-to-get-real-llm-inference-benchmarks.md", "text": "https://wpnews.pro/news/aiperf-is-the-only-way-i-ve-found-to-get-real-llm-inference-benchmarks.txt", "jsonld": "https://wpnews.pro/news/aiperf-is-the-only-way-i-ve-found-to-get-real-llm-inference-benchmarks.jsonld"}}