{"slug": "tuning-vllm-what-every-setting-does-to-the-arithmetic", "title": "Tuning vLLM: What Every Setting Does to the Arithmetic", "summary": "VLLM v0.27.1, the open-source inference engine, has unified its scheduler around a fixed token budget per step, making chunked prefill, prefix caching, and speculative decoding stackable by default, according to a technical series on tuning the engine. The article explains that --max-num-seqs is a ceiling, not a batch, and that performance-mode defaults to balanced, with throughput mode doubling card-dependent numbers. The series is part five of seven, focusing on command-line arguments for production users.", "body_md": "*Part five of seven. Where the previous parts turn into command-line arguments.*\n\nThe first parts of this series were physics. This one is the command line.\n\nIf you run language models in production, there is a good chance you are on vLLM, and a good chance you have barely touched the config. That used to be negligence. It is now a reasonable position, the engine was rebuilt around the goal of needing no configuration, so most of the tuning advice you will find describes switches that are already flipped.\n\nSo this is not twenty things to turn on. That article is written, several times over. The question is which defaults are wrong for *your* traffic, and how to tell.\n\nThe scheduler used to treat reading the prompt and writing the answer as separate phases. It no longer does. It keeps one map of request to token count, {request_id: num_tokens} in [vLLM’s own description of the V1 scheduler](https://vllm.ai/blog/2025-01-27-v1-alpha-release), and hands out a **fixed token budget each step**. Prompt tokens and output tokens count against that budget identically. Whatever fits runs, and that is the whole policy.\n\nThat sounds like an internal refactor. It is the thing that makes everything else possible.\n\nA long prompt arriving in pieces is just “a prompt that took part of the budget.” Reusing a cached prefix is just “these tokens cost nothing.”\n\nThe obvious guess is **continuous batching**, and it is wrong. Continuous batching refills a finished slot on the next step instead of waiting for the whole batch to drain, and it has been in vLLM since 2023. The idea is older: iteration-level scheduling comes from [the Orca paper](https://www.usenix.org/conference/osdi22/presentation/yu) in 2022. Under static batching a request that finished in twenty tokens left its slot empty while another ground through two thousand. vLLM has never worked that way.\n\nWhat the unified budget bought is composition. Chunked prefill is the clearest case. It splits one long prompt across several steps, and it is on by default now. That is why a very long prompt no longer freezes the server for everyone already generating. Chunked prefill used to fight prefix caching and speculative decoding, because each of the three wanted to special-case a phase the scheduler still believed in. Once a step is just a token budget, the three stop being special cases and start stacking.\n\nI think this is the most underrated thing in the engine. It is not a kernel and it is not a format, and it is worth more than either.\n\nTwo of these six depend on the card. Read them against parts one and two and they stop looking like independent settings.\n\nEvery number below is **vLLM v0.27.1**, the release current as of August 2026. They move between releases, and the last paragraph names the two files to read them out of. v0.27.1 also ships --performance-mode, which defaults to balanced. Set it to throughput and both card-dependent numbers double, so the defaults printed here stop describing your server. The doubling applies only where you have not set a value yourself. The third value, interactivity, leaves both numbers alone. It captures a CUDA graph at every batch size up to 32, on top of the ladder balanced already uses, so it is a knob on padding waste at small batch rather than on these two numbers.\n\n**--max-num-seqs is a ceiling, not a batch.** The name keeps inviting the other reading. Your arithmetic intensity is the batch you actually run, vllm:num_requests_running. Part two’s capacity numbers sit under the 1024 ceiling at every context length it measured, and under the 256 ceiling from 8K upward: 504 conversations at 4K, 252 at 8K, 63 at 32K, 15 at 128K. Raising the ceiling usually leaves your intensity where it was, because from 8K up something else is already binding.\n\nThe exception is short contexts on a card carrying the 256 default, where the cache can hold more conversations than the ceiling allows. Do not reach for part two’s 504 there. That number came off a two-H100 pool, and the tier is chosen per device, so that pool carries 1024. Work the number out for your own card at your own context length before you decide the knob does nothing. [Part two’s calculator](https://netsatsawat.github.io/llm-inference-arithmetic/pens/part-2-kv-cache.html) does that in the browser.\n\nOutside that short-context case, lowering the ceiling is the only direction that moves your arithmetic intensity at all, and it moves it down. You are trading intensity for a bounded batch. That trade is why it is still the first thing to reach for on an interactive product. A ceiling permissive enough to fill the card admits a new user straight away, then decodes them alongside a thousand others, every token slower for it. The wait does not disappear. It moves inside the batch.\n\nThe default is 1024 on an H100, an H200 or a B200, and **256 on an A100, an L40S, or anything under 70 GiB**. The rule is device memory, and the A100 is the one exception, pulled into the lower tier by name rather than by size. On the A100, vLLM measured a large token budget costing throughput, and the ceiling came down with the budget.\n\n**--gpu-memory-utilization is a fence, not a reservation.** Context length sets the batch you can hold; this sets the pool it comes out of. The intuitive reading is backwards. At 0.92, vLLM will not allocate more than 92% of the card once it has taken its memory snapshot, and it takes that snapshot after the CUDA context and the NCCL buffers are already resident. Its real footprint is that 92% plus whatever those two cost you. Weights, the profiled activation peak and the CUDA graph buffers come out of *inside* that 92%, and what survives becomes KV cache. The other 8% is left alone for the CUDA context, the NCCL buffers, fragmentation and any co-tenant. Pushing to 0.95 buys real capacity, and it is the cheapest capacity you can get: three points of the card, all of which land in the cache. The biggest win is elsewhere. The 8-bit cache flag further down halves what every conversation costs. What 0.95 costs you is the headroom itself, so what you are courting is fragmentation or a co-tenant rather than a long request. The cache is allocated at startup, and a request that cannot get blocks is preempted rather than crashing. Watch vllm:num_preemptions_total, not your error log.\n\n**--max-num-batched-tokens is the latency dial.** It bounds the whole step, and the requests already generating are served off the top of it, one token each, before any prompt gets a share. So your prefill allowance is that budget minus your running batch. The same mechanism puts a floor under the budget: the engine will not start if the budget is smaller than --max-num-seqs. The one-token-each part holds until you turn on speculative decoding, and then every running request reserves room for its drafts too. Small values interleave finely. They protect the per-token speed of everyone already generating, at the cost of the newcomer’s first token. Large values do the reverse. There is no correct setting, only which of your two targets you would rather miss. On an H100, an H200 or a B200 the default is 16,384 offline but **8,192 through the API server**, and on an A100 or L40S it is 8,192 offline and **2,048 through the API server**. Same two tiers as the ceiling, resolved from the same branch, but this one also splits on how you launched: offline in Python, or serving behind the API. Most people are serving, and almost nobody knows which of the two they are on.\n\n**--enable-prefix-caching is already on, and it is the setting most likely to make your benchmark lie.** Part two’s mechanism, as one flag: reuse is exact-match from the first token, so it pays enormously on a shared system prompt or a multi-turn chat, and nothing at all on traffic that varies at the front. You do not turn this on. You check that nobody turned it off. Then you check the hit rate before you trust a number, and you check it as a rate over your benchmark window: vllm:prefix_cache_hits_total and vllm:prefix_cache_queries_total are counters since server start, so dividing the raw values gives you warmup and every previous run, not this one. Use sum(rate(vllm:prefix_cache_hits_total[5m])) / sum(rate(vllm:prefix_cache_queries_total[5m])), and keep the sum(): every vLLM metric carries model_name and engine labels. A run with a warm cache and a run with a cold one are measuring two different products.\n\n**--kv-cache-dtype fp8 is the cache problem solved with one flag.** Halving the cache roughly doubles the conversations that fit. Whether that raises your arithmetic intensity is a different question, and the answer splits. Decode attention now reads half the bytes, so its arithmetic intensity roughly doubles, which is the claim in part four, [ Quantization Is Four Decisions, Not One](https://medium.com/towards-artificial-intelligence/quantization-is-four-decisions-not-one-c52a0b3f296e). The aggregate is a different story. At the same batch it barely moves, and filling the extra capacity does not rescue it either. The reason for that is a mixture-of-experts reason, so if you serve a dense model it is not yours: part two put gpt-oss-120b’s expert layers at 93% of the bytes fetched per step, firing four of its 128 experts, and they stay bandwidth-bound at any batch you can reach.\n\nTake the flag for the capacity, then run two checks. Read the startup line that names which attention backend was chosen out of the potential backends. A backend that cannot serve an FP8 cache is dropped from the candidate list, so this does not fail quietly: name one explicitly and you get a hard error, name none and vLLM either promotes a different backend or refuses to start. Your capacity arithmetic is safe. What you can lose without noticing is the backend you benchmarked on. And re-run whatever evaluation you have. The evidence on quality is already in, and it is good: vLLM’s [April 2026 study](https://vllm.ai/blog/2026-04-22-fp8-kvcache) tested the FP8 cache on models from Llama-3.1–8B to Kimi-K2.5, on needle-in-a-haystack at 128k, MRCR out to 1M tokens, and four reasoning benchmarks. It recovered 97 to 98% of baseline AUC@128k while giving up a point or two on the reasoning tasks, deliberately using uncalibrated per-tensor scales because that is what the plain flag hands you. It is also someone else’s models at someone else’s context lengths, which is the same reason you are not taking anyone else’s defaults on trust.\n\n**--enforce-eager is a startup-time trade.** It skips both torch.compile and CUDA graph capture, which cost time at startup and pay it back in steady state. You get the fastest start and slower decode after that. That is the right trade while iterating, and a surprising amount of “vLLM is slow” turns out to be this flag, left on after debugging. Keep it in development and nowhere else, with one exception: the graph buffers it skips go back to the KV cache, so where capacity is the binding constraint, measure both ways rather than assuming.\n\nFor some readers the answer is none of the six. That is a real answer, not a polite one. For everyone else it depends on which wall you are against. There are two walls, and one mistake worth ruling out before you look at either.\n\nIf you are memory-constrained, the 8-bit cache is the biggest single win here. The price is not throughput. It is two checks: that your backend actually supports it, and that your own evaluation survives it.\n\nIf you are chasing tail latency on an interactive product, the concurrency ceiling is the one to move, and the one with a real price. A lower ceiling means a smaller running batch and steadier per-token speed for the requests that got in. You pay for that twice: in arithmetic intensity and throughput per card, and in a longer queue. The scheduler stops admitting once the running batch reaches the ceiling. You are buying the ninety-ninth percentile of the token rate with money.\n\nIf you inherited the deployment from somebody who was debugging, check that eager mode is off before you conclude anything about the engine.\n\nThose three are not the same three as the sweep order below, and that is deliberate. The sweep order is the order you measure in. This list is what you change before you measure anything. If you want the whole set on one page, [the cheat sheet is here](https://netsatsawat.github.io/llm-inference-arithmetic/vllm-tuning-cheatsheet.html): six settings, both card tiers, the sweep order, and which wall each number points at. There is [a printable version](https://netsatsawat.github.io/llm-inference-arithmetic/vllm-tuning-cheatsheet.pdf) as well.\n\nFour numbers, and you need them together, because three of them trade against each other. **Time to first token** (TTFT) comes down to prompt reading and queueing. **Time per output token** (TPOT) comes down to memory bandwidth and batch size. **Throughput** is the aggregate. **Goodput** is throughput of requests that met your target, and it is the only one that maps to money: a server at full throughput and 40% goodput is failing while looking busy. It is also the only one of the four you have to declare before you can measure it. vllm bench serve takes the targets as --goodput ttft:500 tpot:50, in milliseconds, your numbers rather than mine. Until you give it targets, it prints nothing under that heading.\n\nThe test data matters more than any setting, because all four numbers move with the shape of the incoming requests. A benchmark with uniform 512-token prompts and no shared prefix tells you something true about a workload nobody has.\n\nNo published number transfers here, and one of my own would not either. The whole argument of this part is that a default is wrong for a particular workload, which makes the measurement workload-specific too: a sweep on someone else’s traffic tells you about their traffic. What does transfer is the procedure, and vllm bench serve is the benchmark. v0.27.1 wraps it in a sweep driver that starts and stops the server for you at every point. Set MODEL to whatever you are serving, and replace every number in the bench command with one of yours:\n\n```\nvllm bench sweep serve \\  --serve-cmd \"vllm serve $MODEL\" \\  --bench-cmd \"vllm bench serve --dataset-name random \\    --random-input-len 3000 --random-output-len 300 --random-range-ratio 0.4 \\    --random-prefix-len 1000 --request-rate 12 --goodput ttft:500 tpot:50\" \\  --serve-params sweep.json \\  -e my-sweep -o results/vllm bench sweep serve \\  --serve-cmd \"vllm serve $MODEL\" \\  --bench-cmd \"vllm bench serve --dataset-name random\" \\  --serve-params sweep.json \\  -o results/\n```\n\nEvery flag spelled out there is replacing a default, and the defaults rebuild the exact benchmark this piece argues against. --dataset-name random on its own is 1024-token prompts, 128-token outputs, zero length variance, no shared prefix, and --request-rate inf, which sends every request at time zero. Take that and your time to first token is queueing delay against one dump, and the ninety-ninth percentile of a dump is not a property of the setting you moved. --goodput is in the string for a different reason. The sweep driver appends --percentile-metrics, --save-result and the result paths to your --bench-cmd, and nothing else, so unless you declare the targets yourself, request_goodput comes back null at every point. The form is milliseconds: ttft:500 tpot:50.\n\nsweep.json holds the values to walk, a list of objects with one entry per point, [{\"max_num_seqs\": 256}, {\"max_num_seqs\": 1024}]. Dashes or underscores, both parse. Each point runs three times by default, and the driver clears the prefix cache between runs, so what you collect is cold-cache numbers. vllm bench sweep plot_pareto then puts tokens per second per user against tokens per second per GPU, which is the shape you are reading rather than the peak. It takes the experiment directory as a positional argument, not the output root, so with -e my-sweep above it is vllm bench sweep plot_pareto results/my-sweep. Name the experiment or it gets a timestamp and you go hunting for it, and re-running the same name without --resume stops with “Cannot overwrite existing experiment_dir”.\n\n**Pin your request shapes first.** Four things: the prompt length distribution, the output length distribution, how much your prompts share a prefix, and the rate requests arrive at. The fourth is the one people drop, and a sweep without it is a saturation test. If you cannot state all four, stop. That is the real task, and the sweep can wait for it.\n\n**Then move one setting at a time, in this order.** --gpu-memory-utilization first, because it sets the size of the pool. Then --max-num-seqs, which decides whether a newcomer waits. Then --max-num-batched-tokens, where you choose which latency target to miss. Move two together and you learn nothing, because the second one landed on a machine you had already changed.\n\n**Then read three lines out of the startup log.** Chunked prefill is enabled with max_num_batched_tokens=... is printed on every boot, since chunked prefill is on by default, and it settles outright which column of the defaults you are on. GPU KV cache size: ... tokens and Maximum concurrency for ... tokens per request: ...x give you the capacity number computed for your own card and your own model. Watch those two move after you change the first setting: the token budget sets the activation peak vLLM profiles, and that peak comes out of the same fence --gpu-memory-utilization set. The pool you sized is not quite the pool you finish with. Do not go hunting for Defaulting max_num_batched_tokens to ..., it is logged at debug level and you will not see it.\n\n**Record all four numbers at every point,** and record the ninety-ninth percentile next to the median. A setting that improves the median and wrecks the tail made your product worse, and the median alone hides that completely.\n\n**Read the shape, not the peak.** You are looking for where a curve bends. A bend is a constraint starting to bind. Knowing which wall you hit first is worth much more than the best single point, and the wall is usually KV capacity.\n\n**Expect most of it to do nothing.** That is the honest finding, and it is the useful one: it tells you the remaining time belongs somewhere other than the flags. Write down the settings that moved nothing, or you will sweep them again in six months.\n\nvLLM’s defaults are a good answer to a question asked on somebody else’s traffic. They are not neutral and they are not wrong. They are decisions somebody already made on evidence you did not supply.\n\nThere are not twenty independent settings here. There are a handful of knobs on physics the earlier parts already explained, and most of the job is knowing which is which.\n\nWhich knob is yours depends on which wall you hit first, and your dashboard already answers that. Watch vllm:num_preemptions_total against the vllm:num_requests_running you were serving happily last month. Preemptions climbing under a load you used to handle means capacity is binding, and the memory fence and the 8-bit cache are where the room is. Preemptions flat but a latency target still slipping means the time is going to queueing and step size rather than to capacity, so the concurrency ceiling and the token budget are the pair to move. Flat and on target means the defaults are right for your traffic, which is the outcome nobody writes an article about.\n\nPart six is what happens when one machine stops being enough: how a model splits across several, and why prefix caching, an 8-bit cache and a sensible concurrency limit are all worth exhausting first, because each of them is cheaper than a second cluster.\n\n*Every default here is from vLLM v0.27.1 and moves between releases. The per-card numbers are **get_batch_defaults in **vllm/engine/arg_utils.py, the 0.92 fence is the field default in **vllm/config/cache.py, and the unified scheduler is described in vLLM’s own V1 alpha announcement. Read them against the version you are actually running before quoting any of them.*\n\n[Tuning vLLM: What Every Setting Does to the Arithmetic](https://pub.towardsai.net/tuning-vllm-what-every-setting-does-to-the-arithmetic-28d5f65e231b) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/tuning-vllm-what-every-setting-does-to-the-arithmetic", "canonical_source": "https://pub.towardsai.net/tuning-vllm-what-every-setting-does-to-the-arithmetic-28d5f65e231b?source=rss----98111c9905da---4", "published_at": "2026-08-23 18:01:01+00:00", "updated_at": "2026-08-23 18:42:57.337867+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["vLLM", "Orca", "H100"], "alternates": {"html": "https://wpnews.pro/news/tuning-vllm-what-every-setting-does-to-the-arithmetic", "markdown": "https://wpnews.pro/news/tuning-vllm-what-every-setting-does-to-the-arithmetic.md", "text": "https://wpnews.pro/news/tuning-vllm-what-every-setting-does-to-the-arithmetic.txt", "jsonld": "https://wpnews.pro/news/tuning-vllm-what-every-setting-does-to-the-arithmetic.jsonld"}}