{"slug": "decoding-policies-in-a-serving-engine", "title": "Decoding Policies in a Serving Engine", "summary": "A Machine Learning Mastery tutorial demonstrates how decoding policies configured through vLLM's SamplingParams change runtime cost in a serving engine, using the HuggingFaceTB/SmolLM2-1.7B-Instruct model to show that sampling filters, multiple outputs, beam search, stop conditions and structured outputs each add per-token work, extend request lifetime, or hold KV cache blocks longer. The post sets generation_config=\"vllm\" so checkpoint defaults in generation_config.json cannot silently override the specified policy, and includes a standalone example for collecting decode-speed measurements and a profiling trace rather than publishing machine-specific numbers. The tutorial argues these effects matter even when model weights and prompts are unchanged, and that actual cost depends on model, hardware, batch, prompt length, output length and vLLM version.", "body_md": "[*Decoding Strategies and Output Control*](https://machinelearningmastery.com/decoding-strategies-and-output-control/) explained how greedy decoding, sampling, beam search, stopping rules, and output constraints select tokens. This post focuses on what those choices cost in an inference engine.\n\nA decoding policy can add work after every model call, create several live sequences for one request, keep KV cache blocks alive for longer, or require separate state for each request. These effects matter even when the model weights and prompt stay the same. This post uses vLLM to make the runtime consequences concrete without repeating the algorithms from [*Decoding Strategies and Output Control*](https://machinelearningmastery.com/decoding-strategies-and-output-control/).\n\nIn this post, you will learn about:\n\n- How decoding settings affect per-token work and batching\n- How multiple outputs and stop rules change request lifetime and KV use\n- How structured outputs add runtime cost\n\nLet’s get started.\n\n## Overview\n\nThis post is divided into six parts; they are:\n\n- Running Decoding Policies with vLLM\n- Sampling Filters and Per-Token Work\n- Multiple Outputs and Beam Search\n- Stop Conditions and Request Lifetime\n- Structured Outputs\n- Batching and Runtime Cost\n\n## Running Decoding Policies with vLLM\n\nvLLM separates model execution from token selection. The `LLM` object loads a model from Hugging Face Hub and runs it. `SamplingParams` describes how vLLM processes the logits and selects output tokens.\n\nThe following setup uses one model and prompt throughout the post:\n\n``` python\nInitialize vLLM and compare greedy decoding with one prompt.\n\nfrom vllm import LLM, SamplingParams\n\nmodel_name = \"HuggingFaceTB/SmolLM2-1.7B-Instruct\"\nprompt = \"Explain why KV cache memory matters during inference.\"\n\nllm = LLM(\n    model=model_name,\n    generation_config=\"vllm\",\n)\n\ndef show_output(name, sampling_params):\n    result = llm.generate(\n        [prompt],\n        sampling_params,\n        use_tqdm=False,\n    )[0]\n\n    print(f\"\\n{name}\")\n    for completion in result.outputs:\n        print(completion.text)\n\ngreedy_params = SamplingParams(\n    temperature=0,\n    min_tokens=16,\n    max_tokens=64,\n)\nshow_output(\"greedy decoding\", greedy_params)\n\n12345678910111213141516171819202122232425262728\n\nfrom vllm import LLM, SamplingParams model_name = \"HuggingFaceTB/SmolLM2-1.7B-Instruct\"prompt = \"Explain why KV cache memory matters during inference.\" llm = LLM(    model=model_name,    generation_config=\"vllm\",) def show_output(name, sampling_params):    result = llm.generate(        [prompt],        sampling_params,        use_tqdm=False,    )[0]     print(f\"\\n{name}\")    for completion in result.outputs:        print(completion.text)  greedy_params = SamplingParams(    temperature=0,    min_tokens=16,    max_tokens=64,)show_output(\"greedy decoding\", greedy_params)\n```\n\nSetting `generation_config=\"vllm\"` prevents values in the checkpoint’s `generation_config.json` from silently replacing vLLM’s defaults. Each example therefore shows the policy specified in its `SamplingParams` object.\n\nThe `greedy_params` object is an input configuration. The `show_output()` function passes it to `llm.generate()`. That method returns one `RequestOutput` for each prompt, so `[0]` selects the result for the first prompt. The `result.outputs` list contains the generated `CompletionOutput` objects. In this example, it contains one completion because `greedy_params` uses the default `n=1`. The `min_tokens=16` setting keeps this short demonstration from stopping at EOS before it prints any text.\n\nThe examples print outputs so that you can confirm the policy ran. Runtime cost depends on the model, hardware, batch, prompt length, output length, and vLLM version. The following standalone example shows how to collect a decode-speed measurement and a memory trace without including machine-specific results in this post.\n\n``` python\nMeasure decode speed and write a profiling trace.\n\nfrom pathlib import Path\nimport time\n\nimport torch\nfrom vllm import LLM, SamplingParams\n\nprofile_dir = Path(\"vllm_profile\").resolve()\nprofile_prompt = \"Explain why KV cache memory matters during inference.\"\nprofiled_llm = LLM(\n    model=\"HuggingFaceTB/SmolLM2-1.7B-Instruct\",\n    generation_config=\"vllm\",\n    profiler_config={\n        \"profiler\": \"torch\",\n        \"torch_profiler_dir\": str(profile_dir),\n        \"torch_profiler_with_memory\": True,\n        \"torch_profiler_record_shapes\": True,\n    },\n)\nprofile_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=128)\n\n# Warm up model execution and CUDA Graphs before measuring speed.\nprofiled_llm.generate([profile_prompt], profile_params, use_tqdm=False)\n\n# Measure a complete offline request without an active profiler.\ntorch.cuda.synchronize()\nstart = time.perf_counter()\nresult = profiled_llm.generate(\n    [profile_prompt],\n    profile_params,\n    use_tqdm=False,\n)[0]\ntorch.cuda.synchronize()\nelapsed = time.perf_counter() - start\nprompt_tokens = len(result.prompt_token_ids or [])\noutput_tokens = len(result.outputs[0].token_ids)\n\nprint(\"\\nBenchmark\")\nprint(f\"prompt tokens: {prompt_tokens}\")\nprint(f\"output tokens: {output_tokens}\")\nprint(f\"offline request wall time: {elapsed:.4f} seconds\")\nprint(f\"output tokens per wall second: {output_tokens / elapsed:.1f}\")\n\n# Run another request while collecting operator and memory events.\nprofiled_llm.start_profile(\"greedy_decode\")\nprofiled_llm.generate([profile_prompt], profile_params, use_tqdm=False)\nprofiled_llm.stop_profile()\nprint(f\"profile written to: {profile_dir}\")\n\n1234567891011121314151617181920212223242526272829303132333435363738394041424344454647\n\nfrom pathlib import Pathimport time import torchfrom vllm import LLM, SamplingParams profile_dir = Path(\"vllm_profile\").resolve()profile_prompt = \"Explain why KV cache memory matters during inference.\"profiled_llm = LLM(    model=\"HuggingFaceTB/SmolLM2-1.7B-Instruct\",    generation_config=\"vllm\",    profiler_config={        \"profiler\": \"torch\",        \"torch_profiler_dir\": str(profile_dir),        \"torch_profiler_with_memory\": True,        \"torch_profiler_record_shapes\": True,    },)profile_params = SamplingParams(temperature=0, min_tokens=16, max_tokens=128) # Warm up model execution and CUDA Graphs before measuring speed.profiled_llm.generate([profile_prompt], profile_params, use_tqdm=False) # Measure a complete offline request without an active profiler.torch.cuda.synchronize()start = time.perf_counter()result = profiled_llm.generate(    [profile_prompt],    profile_params,    use_tqdm=False,)[0]torch.cuda.synchronize()elapsed = time.perf_counter() - startprompt_tokens = len(result.prompt_token_ids or [])output_tokens = len(result.outputs[0].token_ids) print(\"\\nBenchmark\")print(f\"prompt tokens: {prompt_tokens}\")print(f\"output tokens: {output_tokens}\")print(f\"offline request wall time: {elapsed:.4f} seconds\")print(f\"output tokens per wall second: {output_tokens / elapsed:.1f}\") # Run another request while collecting operator and memory events.profiled_llm.start_profile(\"greedy_decode\")profiled_llm.generate([profile_prompt], profile_params, use_tqdm=False)profiled_llm.stop_profile()print(f\"profile written to: {profile_dir}\")\n```\n\nThe benchmark summary is printed before profiling begins. This wall time includes tokenization, prefill, all decode steps, and Python overhead. It is not time to first token or decode-only speed. The optional `RequestOutput.metrics` field may be `None` in vLLM V1, so this example does not rely on it. For TTFT and TPOT, stream responses from a server or use `vllm bench serve` as shown in *vLLM Quick Start*.\n\nRun this standalone example separately from the earlier example so that only one vLLM engine owns the GPU. The profiled request writes PyTorch Profiler traces containing CPU and GPU operations, tensor shapes, and memory allocation events. Open the trace in Perfetto or TensorBoard to inspect which operators allocate memory and consume execution time. Profiling changes performance, so do not use the profiled request to report request speed.\n\nvLLM allocates model weights and most of its KV cache pool when the engine starts, before `start_profile()` is called. The trace therefore shows memory activity during generation, not the engine’s complete reserved GPU footprint. You can use vLLM’s `kv_cache_usage_perc` metric or a device-level tool such as `nvidia-smi` when the total reservation or live KV cache occupancy is the quantity of interest.\n\n## Sampling Filters and Per-Token Work\n\nYou can configure different decoding policies with `SamplingParams`. Greedy decoding selects the largest logit. Temperature sampling rescales logits before sampling. Top-$k$ and top-$p$ restrict the candidates, while a repetition penalty adjusts logits based on tokens that have already appeared.\n\nRun this block after the `llm` and `show_output()` setup above:\n\n```\nApply temperature, top-k, top-p, and repetition policies to one prompt.\n\npolicies = {\n    \"greedy\": SamplingParams(\n        temperature=0, max_tokens=64,\n    ),\n    \"temperature sampling\": SamplingParams(\n        temperature=0.8, seed=42, max_tokens=64,\n    ),\n    \"top-k sampling\": SamplingParams(\n        temperature=0.8, top_k=40, seed=42, max_tokens=64,\n    ),\n    \"nucleus sampling\": SamplingParams(\n        temperature=0.8, top_p=0.9, seed=42, max_tokens=64,\n    ),\n    \"repetition penalty\": SamplingParams(\n        temperature=0.8,\n        top_p=0.9,\n        repetition_penalty=1.1,\n        seed=42,\n        max_tokens=64,\n    ),\n}\n\nfor name, params in policies.items():\n    show_output(name, params)\n\n123456789101112131415161718192021222324\n\npolicies = {    \"greedy\": SamplingParams(        temperature=0, max_tokens=64,    ),    \"temperature sampling\": SamplingParams(        temperature=0.8, seed=42, max_tokens=64,    ),    \"top-k sampling\": SamplingParams(        temperature=0.8, top_k=40, seed=42, max_tokens=64,    ),    \"nucleus sampling\": SamplingParams(        temperature=0.8, top_p=0.9, seed=42, max_tokens=64,    ),    \"repetition penalty\": SamplingParams(        temperature=0.8,        top_p=0.9,        repetition_penalty=1.1,        seed=42,        max_tokens=64,    ),} for name, params in policies.items():    show_output(name, params)\n```\n\nAll five policies keep one live continuation. Their KV cache requirements are therefore similar when they produce the same number of tokens. Their logit-processing work differs.\n\nThe model still computes a logit for every vocabulary item. Top-$k$ does not make the final vocabulary projection produce only $k$ values, and top-$p$ does not avoid that projection either. These filters operate on the logits after the model has produced them.\n\nGreedy selection needs a maximum. Sampling additionally rescales logits, constructs a probability distribution, and draws a token. Top-$k$ needs to identify high-scoring candidates. Top-$p$ needs enough ordering information to find the smallest candidate set whose cumulative probability reaches $p$. Repetition penalties need information about earlier tokens and modify affected logits before selection.\n\nFor the same model, batch, and output length, the extra operations are:\n\n- *Greedy decoding* needs a maximum over logits.\n- *Temperature sampling* constructs and samples from a probability distribution.\n- *Top-$k$ sampling* identifies the highest-scoring candidates.\n- *Top-$p$ sampling* finds a cumulative-probability cutoff; implementations may use sorting or specialized kernels.\n- *Repetition penalties* update logits according to token history.\n\nThis processing happens at every decode step. It is usually smaller than the model forward pass, but it is not free. Its relative importance grows when the model is small, the vocabulary or batch is large, or several processors are combined.\n\nThe largest cost difference may still come from output length. A policy that generates more tokens invokes the model more times and extends the KV cache further, even if its sampler is inexpensive.\n\n## Multiple Outputs and Beam Search\n\nThe `n` parameter asks vLLM to return several sampled continuations for one prompt:\n\n```\nRequest several independent sampled continuations.\n\nparallel_samples = SamplingParams(\n    temperature=0.8,\n    top_p=0.9,\n    n=4,\n    seed=42,\n    max_tokens=64,\n)\n\nshow_output(\"four sampled continuations\", parallel_samples)\n\n123456789\n\nparallel_samples = SamplingParams(    temperature=0.8,    top_p=0.9,    n=4,    seed=42,    max_tokens=64,) show_output(\"four sampled continuations\", parallel_samples)\n```\n\nParallel sampling produces independent continuations from the same prompt. Each sequence samples its own tokens and then conditions on its own generated history. The sequences do not exchange information, compete, or get ranked against one another. They may share the prompt computation and prompt KV blocks, but their state becomes private as their generated tokens diverge.\n\nWith `n=4`, the request creates four live output sequences. If all four generate $G$ tokens, their private generated cache is approximately four times the cache for one $G$-token continuation. The model also performs decode work for four active sequences instead of one.\n\nGenerating four continuations is therefore slower and uses more cache than generating one. It can still be more efficient than submitting four independent requests because vLLM can share prompt work and batch the active sequences.\n\nBeam search uses several continuations differently. Its hypotheses compete using cumulative scores. At every step, beam search expands them, ranks their candidates together, and retains only the best beams. Parallel sampling returns its independent responses without choosing a winner. Current vLLM exposes beam search through a separate API:\n\n```\nRun beam search with the model created earlier.\n\nfrom vllm.sampling_params import BeamSearchParams\n\nbeam_params = BeamSearchParams(\n    beam_width=4,\n    max_tokens=64,\n    length_penalty=1.0,\n)\n\nbeam_results = llm.beam_search([{\"prompt\": prompt}], beam_params)\n\nfor index, sequence in enumerate(beam_results[0].sequences):\n    print(f\"beam {index}: {sequence.text}\")\n\n123456789101112\n\nfrom vllm.sampling_params import BeamSearchParams beam_params = BeamSearchParams(    beam_width=4,    max_tokens=64,    length_penalty=1.0,) beam_results = llm.beam_search([{\"prompt\": prompt}], beam_params) for index, sequence in enumerate(beam_results[0].sequences):    print(f\"beam {index}: {sequence.text}\")\n```\n\nA beam width of four creates up to four live hypotheses. In addition to their model and cache work, beam search must retrieve candidate scores, update cumulative beam scores, select surviving hypotheses, and keep their token and cache references aligned. This control flow is more expensive than independently sampling four tokens.\n\nAt the same width and output length, beam search adds cross-beam ranking and survivor selection at every step. Its measured latency also depends on how the engine batches and shares state.\n\nPrompt blocks and common beam prefixes may be shared internally, so total memory does not have to equal four complete copies of the request. After beams diverge, however, their new KV state is private. Wider beams therefore reduce the cache capacity available for unrelated requests.\n\n## Stop Conditions and Request Lifetime\n\nThe cost of a request depends strongly on when it finishes. Every additional output token requires another decode iteration and another token’s KV state.\n\nvLLM can bound generation with `max_tokens`, delay EOS or token-based stops with `min_tokens`, and stop when a string or token ID is produced:\n\n```\nSet generation limits and a stop string for one prompt.\n\nbounded = SamplingParams(\n    temperature=0,\n    min_tokens=16,\n    max_tokens=96,\n)\nshow_output(\"bounded generation\", bounded)\n\nstop_at_blank_line = SamplingParams(\n    temperature=0,\n    stop=[\"\\n\\n\"],\n    max_tokens=96,\n)\nshow_output(\"stop at a blank line\", stop_at_blank_line)\n\n12345678910111213\n\nbounded = SamplingParams(    temperature=0,    min_tokens=16,    max_tokens=96,)show_output(\"bounded generation\", bounded) stop_at_blank_line = SamplingParams(    temperature=0,    stop=[\"\\n\\n\"],    max_tokens=96,)show_output(\"stop at a blank line\", stop_at_blank_line)\n```\n\nThe parameter `max_tokens` is an upper bound, not a reservation that always becomes cache data. The request can finish earlier after EOS or another stop condition. A larger limit permits a request to remain active longer, consume more decode iterations, and grow a larger cache.\n\nA smaller `max_tokens` can cap worst-case latency and memory growth, but it is not faster when both configurations stop naturally at the same earlier token.\n\nThe parameter `min_tokens` prevents EOS and IDs listed in `stop_token_ids` from ending the sequence too early. This can force the engine to continue a response that would otherwise finish. Stop strings are handled separately: vLLM incrementally detokenizes output and checks whether the generated text ends with a configured string. String matching therefore adds text-processing state and work after token selection.\n\nCompared with ordinary EOS stopping, `min_tokens` can only leave the amount of generation work unchanged or increase it. A stop string adds matching overhead, but it reduces total cost when it ends generation sufficiently earlier.\n\nRequests in the same batch usually finish at different lengths. vLLM removes finished sequences from later iterations and can schedule other work in their place. Until a request finishes, its KV blocks remain live. A policy that tends to generate longer answers therefore affects both its own cost and the capacity available to other requests.\n\n## Structured Outputs\n\nStructured output restricts generation to tokens that can continue a choice, regular expression, grammar, or JSON schema. vLLM represents these constraints with `StructuredOutputsParams`:\n\n``` python\nConstrain one response to a JSON schema.\n\nimport json\nfrom vllm.sampling_params import StructuredOutputsParams\n\nschema = {\n    \"type\": \"object\",\n    \"properties\": {\n        \"sentiment\": {\n            \"type\": \"string\",\n            \"enum\": [\"positive\", \"negative\"],\n        },\n        \"confidence\": {\n            \"type\": \"number\",\n            \"minimum\": 0,\n            \"maximum\": 1,\n        },\n    },\n    \"required\": [\"sentiment\", \"confidence\"],\n    \"additionalProperties\": False,\n}\n\nparams = SamplingParams(\n    temperature=0,\n    max_tokens=32,\n    structured_outputs=StructuredOutputsParams(json=schema),\n)\n\nprompt = \"\"\"\nClassify the sentiment of this sentence:\n\"vLLM makes model serving much easier.\"\n\nReturn the sentiment and a confidence score.\n\"\"\"\n\nresult = llm.generate([prompt], params, use_tqdm=False)[0]\nanswer = json.loads(result.outputs[0].text)\n\nprint(answer)\nprint(answer[\"sentiment\"])\nprint(answer[\"confidence\"])\n\n123456789101112131415161718192021222324252627282930313233343536373839\n\nimport jsonfrom vllm.sampling_params import StructuredOutputsParams schema = {    \"type\": \"object\",    \"properties\": {        \"sentiment\": {            \"type\": \"string\",            \"enum\": [\"positive\", \"negative\"],        },        \"confidence\": {            \"type\": \"number\",            \"minimum\": 0,            \"maximum\": 1,        },    },    \"required\": [\"sentiment\", \"confidence\"],    \"additionalProperties\": False,} params = SamplingParams(    temperature=0,    max_tokens=32,    structured_outputs=StructuredOutputsParams(json=schema),) prompt = \"\"\"Classify the sentiment of this sentence:\"vLLM makes model serving much easier.\" Return the sentiment and a confidence score.\"\"\" result = llm.generate([prompt], params, use_tqdm=False)[0]answer = json.loads(result.outputs[0].text) print(answer)print(answer[\"sentiment\"])print(answer[\"confidence\"])\n```\n\nvLLM constrains token generation so that the text follows the schema. The function `json.loads()` then converts the generated JSON into a Python dictionary. The schema guarantees the output structure and permitted values, but it does not guarantee that the classification or confidence is correct.\n\nThe constraint does not make the model compute only the allowed logits. The model still produces its normal vocabulary-sized output. The structured-output backend tracks the constraint state, determines which token IDs are valid next, and masks invalid logits before selection.\n\nThis work repeats after every generated token. A small fixed choice has little state. A grammar or JSON schema may require a more complicated state machine, and every active request can be at a different point in that machine. The runtime must keep these states synchronized with the corresponding token sequences.\n\nFor the same output length, constrained generation adds state updates and masking. A fixed choice usually requires less constraint work than a large grammar or JSON schema, but the measured cost depends on the backend and batch size.\n\nStructured output can still reduce total application cost. Without it, an invalid response may need parsing, repair, or complete regeneration. The useful comparison is therefore not only constrained versus unconstrained cost per token, but also the number of failed responses and retries avoided.\n\n## Batching and Runtime Cost\n\nvLLM uses continuous batching to combine active requests at each model iteration. Requests do not need identical prompts, output lengths, temperatures, or top-$k$ values. The model can produce logits for a batch and apply request-specific sampling parameters to its rows.\n\nDifferent policies nevertheless change the batch that vLLM must execute:\n\n- Greedy decoding and ordinary sampling contribute one active sequence per request.\n- Setting `n=4` contributes four output sequences after the shared prompt.\n- Beam width four maintains up to four scored hypotheses and additional beam bookkeeping.\n- Longer outputs keep sequences and their cache blocks active for more iterations.\n- Repetition penalties require request-specific token history during logit processing.\n- Structured outputs require one constraint state and valid-token mask per active sequence.\n\nThe important distinction is between model work and post-processing work. Temperature, top-$k$, top-$p$, penalties, and constraints normally leave the transformer forward pass unchanged. They add work between the logits and the selected token. Branching and longer generation go further: they increase the number of sequence positions for which the model itself must run.\n\nHeterogeneous policies can also make the sampler take a more general path than a batch in which every row uses the same simple rule. Whether that difference matters depends on the workload and implementation. The policy should therefore be chosen for required output behavior first, then checked under the intended serving workload rather than judged from its name alone.\n\n## Further Reading\n\nBelow are some resources you may find useful:\n\n- vLLM documentation. [Sampling Parameters](https://docs.vllm.ai/en/latest/api/vllm/sampling_params/) . This reference defines temperature, top-$k$, top-$p$, penalties, output counts, and stop conditions.\n- vLLM documentation. [Generative Models](https://docs.vllm.ai/en/latest/models/generative_models/) . This guide covers offline generation and the current beam-search API.\n- vLLM documentation. [Structured Outputs](https://docs.vllm.ai/en/latest/features/structured_outputs/) . This guide covers choices, regular expressions, grammars, and JSON schemas.\n- vLLM documentation. [Profiling vLLM](https://docs.vllm.ai/en/latest/contributing/profiling/) . This guide explains profiler configuration, memory traces, and the overhead of profiling.\n- Woosuk Kwon et al. (2023). [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180) . This paper explains vLLM’s block-based KV cache and continuous batching design.\n\n## Summary\n\nIn this post, you used vLLM to connect decoding settings to runtime cost. Greedy decoding and ordinary sampling keep one live sequence, while filters and penalties add per-token logit processing. Parallel outputs and beam search create more live sequences, model work, and private KV state after their continuations diverge.\n\nYou also learned that stopping rules determine how long cache blocks remain active and that structured outputs maintain a constraint state and mask logits at every step. vLLM can batch many of these policies together, but the number of live sequences, their lengths, and their request-specific processing still determine the work.", "url": "https://wpnews.pro/news/decoding-policies-in-a-serving-engine", "canonical_source": "https://machinelearningmastery.com/decoding-policies-in-a-serving-engine/", "published_at": "2026-11-22 19:06:51+00:00", "updated_at": "2026-11-22 20:44:40.480323+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["vLLM", "Machine Learning Mastery", "Hugging Face", "HuggingFaceTB/SmolLM2-1.7B-Instruct", "SamplingParams"], "also_reported_by": [], "takeaway": "Set generation_config=\"vllm\" in vLLM so a checkpoint's generation_config.json cannot silently override your SamplingParams, and treat sampling filters, n>1 outputs, beam search, stop rules and structured outputs as added per-token work, longer request lifetimes and longer-held KV cache blocks.", "recommendation": {"advice": "Pin generation_config=\"vllm\" and profile decode speed and KV memory per policy, since sampling filters, multiple outputs, stop rules and structured outputs each add per-token work or extend request lifetime.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/decoding-policies-in-a-serving-engine", "markdown": "https://wpnews.pro/news/decoding-policies-in-a-serving-engine.md", "text": "https://wpnews.pro/news/decoding-policies-in-a-serving-engine.txt", "jsonld": "https://wpnews.pro/news/decoding-policies-in-a-serving-engine.jsonld"}}