cd /news/ai-infrastructure/decoding-policies-in-a-serving-engin… · home › topics › ai-infrastructure › article
[ARTICLE · art-174324] src=machinelearningmastery.com ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Decoding Policies in a Serving Engine

A Machine Learning Mastery tutorial demonstrates how decoding policies configured through vLLM's SamplingParams change runtime cost in a serving engine, using the HuggingFaceTB/SmolLM2-1.7B-Instruct model to show that sampling filters, multiple outputs, beam search, stop conditions and structured outputs each add per-token work, extend request lifetime, or hold KV cache blocks longer. The post sets generation_config="vllm" so checkpoint defaults in generation_config.json cannot silently override the specified policy, and includes a standalone example for collecting decode-speed measurements and a profiling trace rather than publishing machine-specific numbers. The tutorial argues these effects matter even when model weights and prompts are unchanged, and that actual cost depends on model, hardware, batch, prompt length, output length and vLLM version.

recommendation

Pin generation_config="vllm" and profile decode speed and KV memory per policy, since sampling filters, multiple outputs, stop rules and structured outputs each add per-token work or extend request lifetime.

more in playbook: Inference and self-hosting
by read15 min views8 publishedNov 22, 2026
Decoding Policies in a Serving Engine
Image: source

Decoding Strategies and Output Control explained how greedy decoding, sampling, beam search, stopping rules, and output constraints select tokens. This post focuses on what those choices cost in an inference engine.

A decoding policy can add work after every model call, create several live sequences for one request, keep KV cache blocks alive for longer, or require separate state for each request. These effects matter even when the model weights and prompt stay the same. This post uses vLLM to make the runtime consequences concrete without repeating the algorithms from Decoding Strategies and Output Control.

In this post, you will learn about:

  • How decoding settings affect per-token work and batching
  • How multiple outputs and stop rules change request lifetime and KV use
  • How structured outputs add runtime cost

Let’s get started.

Overview #

This post is divided into six parts; they are:

  • Running Decoding Policies with vLLM
  • Sampling Filters and Per-Token Work
  • Multiple Outputs and Beam Search
  • Stop Conditions and Request Lifetime
  • Structured Outputs
  • Batching and Runtime Cost

Running Decoding Policies with vLLM #

vLLM separates model execution from token selection. The LLM object loads a model from Hugging Face Hub and runs it. SamplingParams describes how vLLM processes the logits and selects output tokens.

The following setup uses one model and prompt throughout the post:

Initialize vLLM and compare greedy decoding with one prompt.

from vllm import LLM, SamplingParams

model_name = "HuggingFaceTB/SmolLM2-1.7B-Instruct"
prompt = "Explain why KV cache memory matters during inference."

llm = LLM(
    model=model_name,
    generation_config="vllm",
)

def show_output(name, sampling_params):
    result = llm.generate(
        [prompt],
        sampling_params,
        use_tqdm=False,
    )[0]

    print(f"\n{name}")
    for completion in result.outputs:
        print(completion.text)

greedy_params = SamplingParams(
    temperature=0,
    min_tokens=16,
    max_tokens=64,
)
show_output("greedy decoding", greedy_params)

12345678910111213141516171819202122232425262728

from vllm import LLM, SamplingParams model_name = "HuggingFaceTB/SmolLM2-1.7B-Instruct"prompt = "Explain why KV cache memory matters during inference." llm = LLM(    model=model_name,    generation_config="vllm",) def show_output(name, sampling_params):    result = llm.generate(        [prompt],        sampling_params,        use_tqdm=False,    )[0]     print(f"\n{name}")    for completion in result.outputs:        print(completion.text)  greedy_params = SamplingParams(    temperature=0,    min_tokens=16,    max_tokens=64,)show_output("greedy decoding", greedy_params)

Setting generation_config="vllm" prevents values in the checkpoint’s generation_config.json from silently replacing vLLM’s defaults. Each example therefore shows the policy specified in its SamplingParams object.

The greedy_params object is an input configuration. The show_output() function passes it to llm.generate(). That method returns one RequestOutput for each prompt, so [0] selects the result for the first prompt. The result.outputs list contains the generated CompletionOutput objects. In this example, it contains one completion because greedy_params uses the default n=1. The min_tokens=16 setting keeps this short demonstration from stopping at EOS before it prints any text.

The examples print outputs so that you can confirm the policy ran. Runtime cost depends on the model, hardware, batch, prompt length, output length, and vLLM version. The following standalone example shows how to collect a decode-speed measurement and a memory trace without including machine-specific results in this post.

Measure decode speed and write a profiling trace.

from pathlib import Path
import time

import torch
from vllm import LLM, SamplingParams

profile_dir = Path("vllm_profile").resolve()
profile_prompt = "Explain why KV cache memory matters during inference."
profiled_llm = LLM(
    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
    generation_config="vllm",
    profiler_config={
        "profiler": "torch",
        "torch_profiler_dir": str(profile_dir),
        "torch_profiler_with_memory": True,
        "torch_profiler_record_shapes": True,
    },
)
profile_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=128)

profiled_llm.generate([profile_prompt], profile_params, use_tqdm=False)

torch.cuda.synchronize()
start = time.perf_counter()
result = profiled_llm.generate(
    [profile_prompt],
    profile_params,
    use_tqdm=False,
)[0]
torch.cuda.synchronize()
elapsed = time.perf_counter() - start
prompt_tokens = len(result.prompt_token_ids or [])
output_tokens = len(result.outputs[0].token_ids)

print("\nBenchmark")
print(f"prompt tokens: {prompt_tokens}")
print(f"output tokens: {output_tokens}")
print(f"offline request wall time: {elapsed:.4f} seconds")
print(f"output tokens per wall second: {output_tokens / elapsed:.1f}")

profiled_llm.start_profile("greedy_decode")
profiled_llm.generate([profile_prompt], profile_params, use_tqdm=False)
profiled_llm.stop_profile()
print(f"profile written to: {profile_dir}")

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647

from pathlib import Pathimport time import torchfrom vllm import LLM, SamplingParams profile_dir = Path("vllm_profile").resolve()profile_prompt = "Explain why KV cache memory matters during inference."profiled_llm = LLM(    model="HuggingFaceTB/SmolLM2-1.7B-Instruct",    generation_config="vllm",    profiler_config={        "profiler": "torch",        "torch_profiler_dir": str(profile_dir),        "torch_profiler_with_memory": True,        "torch_profiler_record_shapes": True,    },)profile_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=128) # Warm up model execution and CUDA Graphs before measuring speed.profiled_llm.generate([profile_prompt], profile_params, use_tqdm=False) # Measure a complete offline request without an active profiler.torch.cuda.synchronize()start = time.perf_counter()result = profiled_llm.generate(    [profile_prompt],    profile_params,    use_tqdm=False,)[0]torch.cuda.synchronize()elapsed = time.perf_counter() - startprompt_tokens = len(result.prompt_token_ids or [])output_tokens = len(result.outputs[0].token_ids) print("\nBenchmark")print(f"prompt tokens: {prompt_tokens}")print(f"output tokens: {output_tokens}")print(f"offline request wall time: {elapsed:.4f} seconds")print(f"output tokens per wall second: {output_tokens / elapsed:.1f}") # Run another request while collecting operator and memory events.profiled_llm.start_profile("greedy_decode")profiled_llm.generate([profile_prompt], profile_params, use_tqdm=False)profiled_llm.stop_profile()print(f"profile written to: {profile_dir}")

The benchmark summary is printed before profiling begins. This wall time includes tokenization, prefill, all decode steps, and Python overhead. It is not time to first token or decode-only speed. The optional RequestOutput.metrics field may be None in vLLM V1, so this example does not rely on it. For TTFT and TPOT, stream responses from a server or use vllm bench serve as shown in vLLM Quick Start.

Run this standalone example separately from the earlier example so that only one vLLM engine owns the GPU. The profiled request writes PyTorch Profiler traces containing CPU and GPU operations, tensor shapes, and memory allocation events. Open the trace in Perfetto or TensorBoard to inspect which operators allocate memory and consume execution time. Profiling changes performance, so do not use the profiled request to report request speed.

vLLM allocates model weights and most of its KV cache pool when the engine starts, before start_profile() is called. The trace therefore shows memory activity during generation, not the engine’s complete reserved GPU footprint. You can use vLLM’s kv_cache_usage_perc metric or a device-level tool such as nvidia-smi when the total reservation or live KV cache occupancy is the quantity of interest.

Sampling Filters and Per-Token Work #

You can configure different decoding policies with SamplingParams. Greedy decoding selects the largest logit. Temperature sampling rescales logits before sampling. Top-$k$ and top-$p$ restrict the candidates, while a repetition penalty adjusts logits based on tokens that have already appeared.

Run this block after the llm and show_output() setup above:

Apply temperature, top-k, top-p, and repetition policies to one prompt.

policies = {
    "greedy": SamplingParams(
        temperature=0, max_tokens=64,
    ),
    "temperature sampling": SamplingParams(
        temperature=0.8, seed=42, max_tokens=64,
    ),
    "top-k sampling": SamplingParams(
        temperature=0.8, top_k=40, seed=42, max_tokens=64,
    ),
    "nucleus sampling": SamplingParams(
        temperature=0.8, top_p=0.9, seed=42, max_tokens=64,
    ),
    "repetition penalty": SamplingParams(
        temperature=0.8,
        top_p=0.9,
        repetition_penalty=1.1,
        seed=42,
        max_tokens=64,
    ),
}

for name, params in policies.items():
    show_output(name, params)

123456789101112131415161718192021222324

policies = {    "greedy": SamplingParams(        temperature=0, max_tokens=64,    ),    "temperature sampling": SamplingParams(        temperature=0.8, seed=42, max_tokens=64,    ),    "top-k sampling": SamplingParams(        temperature=0.8, top_k=40, seed=42, max_tokens=64,    ),    "nucleus sampling": SamplingParams(        temperature=0.8, top_p=0.9, seed=42, max_tokens=64,    ),    "repetition penalty": SamplingParams(        temperature=0.8,        top_p=0.9,        repetition_penalty=1.1,        seed=42,        max_tokens=64,    ),} for name, params in policies.items():    show_output(name, params)

All five policies keep one live continuation. Their KV cache requirements are therefore similar when they produce the same number of tokens. Their logit-processing work differs.

The model still computes a logit for every vocabulary item. Top-$k$ does not make the final vocabulary projection produce only $k$ values, and top-$p$ does not avoid that projection either. These filters operate on the logits after the model has produced them.

Greedy selection needs a maximum. Sampling additionally rescales logits, constructs a probability distribution, and draws a token. Top-$k$ needs to identify high-scoring candidates. Top-$p$ needs enough ordering information to find the smallest candidate set whose cumulative probability reaches $p$. Repetition penalties need information about earlier tokens and modify affected logits before selection.

For the same model, batch, and output length, the extra operations are:

  • Greedy decoding needs a maximum over logits.
  • Temperature sampling constructs and samples from a probability distribution.
  • Top-$k$ sampling identifies the highest-scoring candidates.
  • Top-$p$ sampling finds a cumulative-probability cutoff; implementations may use sorting or specialized kernels.
  • Repetition penalties update logits according to token history.

This processing happens at every decode step. It is usually smaller than the model forward pass, but it is not free. Its relative importance grows when the model is small, the vocabulary or batch is large, or several processors are combined.

The largest cost difference may still come from output length. A policy that generates more tokens invokes the model more times and extends the KV cache further, even if its sampler is inexpensive.

The n parameter asks vLLM to return several sampled continuations for one prompt:

Request several independent sampled continuations.

parallel_samples = SamplingParams(
    temperature=0.8,
    top_p=0.9,
    n=4,
    seed=42,
    max_tokens=64,
)

show_output("four sampled continuations", parallel_samples)

123456789

parallel_samples = SamplingParams(    temperature=0.8,    top_p=0.9,    n=4,    seed=42,    max_tokens=64,) show_output("four sampled continuations", parallel_samples)

Parallel sampling produces independent continuations from the same prompt. Each sequence samples its own tokens and then conditions on its own generated history. The sequences do not exchange information, compete, or get ranked against one another. They may share the prompt computation and prompt KV blocks, but their state becomes private as their generated tokens diverge.

With n=4, the request creates four live output sequences. If all four generate $G$ tokens, their private generated cache is approximately four times the cache for one $G$-token continuation. The model also performs decode work for four active sequences instead of one.

Generating four continuations is therefore slower and uses more cache than generating one. It can still be more efficient than submitting four independent requests because vLLM can share prompt work and batch the active sequences.

Beam search uses several continuations differently. Its hypotheses compete using cumulative scores. At every step, beam search expands them, ranks their candidates together, and retains only the best beams. Parallel sampling returns its independent responses without choosing a winner. Current vLLM exposes beam search through a separate API:

Run beam search with the model created earlier.

from vllm.sampling_params import BeamSearchParams

beam_params = BeamSearchParams(
    beam_width=4,
    max_tokens=64,
    length_penalty=1.0,
)

beam_results = llm.beam_search([{"prompt": prompt}], beam_params)

for index, sequence in enumerate(beam_results[0].sequences):
    print(f"beam {index}: {sequence.text}")

123456789101112

from vllm.sampling_params import BeamSearchParams beam_params = BeamSearchParams(    beam_width=4,    max_tokens=64,    length_penalty=1.0,) beam_results = llm.beam_search([{"prompt": prompt}], beam_params) for index, sequence in enumerate(beam_results[0].sequences):    print(f"beam {index}: {sequence.text}")

A beam width of four creates up to four live hypotheses. In addition to their model and cache work, beam search must retrieve candidate scores, update cumulative beam scores, select surviving hypotheses, and keep their token and cache references aligned. This control flow is more expensive than independently sampling four tokens.

At the same width and output length, beam search adds cross-beam ranking and survivor selection at every step. Its measured latency also depends on how the engine batches and shares state.

Prompt blocks and common beam prefixes may be shared internally, so total memory does not have to equal four complete copies of the request. After beams diverge, however, their new KV state is private. Wider beams therefore reduce the cache capacity available for unrelated requests.

Stop Conditions and Request Lifetime #

The cost of a request depends strongly on when it finishes. Every additional output token requires another decode iteration and another token’s KV state.

vLLM can bound generation with max_tokens, delay EOS or token-based stops with min_tokens, and stop when a string or token ID is produced:

Set generation limits and a stop string for one prompt.

bounded = SamplingParams(
    temperature=0,
    min_tokens=16,
    max_tokens=96,
)
show_output("bounded generation", bounded)

stop_at_blank_line = SamplingParams(
    temperature=0,
    stop=["\n\n"],
    max_tokens=96,
)
show_output("stop at a blank line", stop_at_blank_line)

12345678910111213

bounded = SamplingParams(    temperature=0,    min_tokens=16,    max_tokens=96,)show_output("bounded generation", bounded) stop_at_blank_line = SamplingParams(    temperature=0,    stop=["\n\n"],    max_tokens=96,)show_output("stop at a blank line", stop_at_blank_line)

The parameter max_tokens is an upper bound, not a reservation that always becomes cache data. The request can finish earlier after EOS or another stop condition. A larger limit permits a request to remain active longer, consume more decode iterations, and grow a larger cache.

A smaller max_tokens can cap worst-case latency and memory growth, but it is not faster when both configurations stop naturally at the same earlier token.

The parameter min_tokens prevents EOS and IDs listed in stop_token_ids from ending the sequence too early. This can force the engine to continue a response that would otherwise finish. Stop strings are handled separately: vLLM incrementally detokenizes output and checks whether the generated text ends with a configured string. String matching therefore adds text-processing state and work after token selection.

Compared with ordinary EOS stopping, min_tokens can only leave the amount of generation work unchanged or increase it. A stop string adds matching overhead, but it reduces total cost when it ends generation sufficiently earlier.

Requests in the same batch usually finish at different lengths. vLLM removes finished sequences from later iterations and can schedule other work in their place. Until a request finishes, its KV blocks remain live. A policy that tends to generate longer answers therefore affects both its own cost and the capacity available to other requests.

Structured Outputs #

Structured output restricts generation to tokens that can continue a choice, regular expression, grammar, or JSON schema. vLLM represents these constraints with StructuredOutputsParams:

Constrain one response to a JSON schema.

import json
from vllm.sampling_params import StructuredOutputsParams

schema = {
    "type": "object",
    "properties": {
        "sentiment": {
            "type": "string",
            "enum": ["positive", "negative"],
        },
        "confidence": {
            "type": "number",
            "minimum": 0,
            "maximum": 1,
        },
    },
    "required": ["sentiment", "confidence"],
    "additionalProperties": False,
}

params = SamplingParams(
    temperature=0,
    max_tokens=32,
    structured_outputs=StructuredOutputsParams(json=schema),
)

prompt = """
Classify the sentiment of this sentence:
"vLLM makes model serving much easier."

Return the sentiment and a confidence score.
"""

result = llm.generate([prompt], params, use_tqdm=False)[0]
answer = json.loads(result.outputs[0].text)

print(answer)
print(answer["sentiment"])
print(answer["confidence"])

123456789101112131415161718192021222324252627282930313233343536373839

import jsonfrom vllm.sampling_params import StructuredOutputsParams schema = {    "type": "object",    "properties": {        "sentiment": {            "type": "string",            "enum": ["positive", "negative"],        },        "confidence": {            "type": "number",            "minimum": 0,            "maximum": 1,        },    },    "required": ["sentiment", "confidence"],    "additionalProperties": False,} params = SamplingParams(    temperature=0,    max_tokens=32,    structured_outputs=StructuredOutputsParams(json=schema),) prompt = """Classify the sentiment of this sentence:"vLLM makes model serving much easier." Return the sentiment and a confidence score.""" result = llm.generate([prompt], params, use_tqdm=False)[0]answer = json.loads(result.outputs[0].text) print(answer)print(answer["sentiment"])print(answer["confidence"])

vLLM constrains token generation so that the text follows the schema. The function json.loads() then converts the generated JSON into a Python dictionary. The schema guarantees the output structure and permitted values, but it does not guarantee that the classification or confidence is correct.

The constraint does not make the model compute only the allowed logits. The model still produces its normal vocabulary-sized output. The structured-output backend tracks the constraint state, determines which token IDs are valid next, and masks invalid logits before selection.

This work repeats after every generated token. A small fixed choice has little state. A grammar or JSON schema may require a more complicated state machine, and every active request can be at a different point in that machine. The runtime must keep these states synchronized with the corresponding token sequences.

For the same output length, constrained generation adds state updates and masking. A fixed choice usually requires less constraint work than a large grammar or JSON schema, but the measured cost depends on the backend and batch size.

Structured output can still reduce total application cost. Without it, an invalid response may need parsing, repair, or complete regeneration. The useful comparison is therefore not only constrained versus unconstrained cost per token, but also the number of failed responses and retries avoided.

Batching and Runtime Cost #

vLLM uses continuous batching to combine active requests at each model iteration. Requests do not need identical prompts, output lengths, temperatures, or top-$k$ values. The model can produce logits for a batch and apply request-specific sampling parameters to its rows.

Different policies nevertheless change the batch that vLLM must execute:

  • Greedy decoding and ordinary sampling contribute one active sequence per request.
  • Setting n=4 contributes four output sequences after the shared prompt.
  • Beam width four maintains up to four scored hypotheses and additional beam bookkeeping.
  • Longer outputs keep sequences and their cache blocks active for more iterations.
  • Repetition penalties require request-specific token history during logit processing.
  • Structured outputs require one constraint state and valid-token mask per active sequence.

The important distinction is between model work and post-processing work. Temperature, top-$k$, top-$p$, penalties, and constraints normally leave the transformer forward pass unchanged. They add work between the logits and the selected token. Branching and longer generation go further: they increase the number of sequence positions for which the model itself must run.

Heterogeneous policies can also make the sampler take a more general path than a batch in which every row uses the same simple rule. Whether that difference matters depends on the workload and implementation. The policy should therefore be chosen for required output behavior first, then checked under the intended serving workload rather than judged from its name alone.

Further Reading #

Below are some resources you may find useful:

Summary #

In this post, you used vLLM to connect decoding settings to runtime cost. Greedy decoding and ordinary sampling keep one live sequence, while filters and penalties add per-token logit processing. Parallel outputs and beam search create more live sequences, model work, and private KV state after their continuations diverge.

You also learned that stopping rules determine how long cache blocks remain active and that structured outputs maintain a constraint state and mask logits at every step. vLLM can batch many of these policies together, but the number of live sequences, their lengths, and their request-specific processing still determine the work.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @vllm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/decoding-policies-in…] indexed:0 read:15min 2026-11-22 · —