{"slug": "when-your-ai-agent-quietly-gets-a-half-finished-answer", "title": "When Your AI Agent Quietly Gets a Half-Finished Answer", "summary": "A new technical reference documents that AI model APIs, including Anthropic, OpenAI, and vLLM, return HTTP 200 with well-formed bodies when generation hits the max output token ceiling, making truncation indistinguishable from completion without checking the stop-reason metadata field. The survey of 18 API surfaces across 14 vendors finds six different field names and seven truncation values, and highlights a vLLM issue where a truncated tool call was labeled as completed, while frameworks like LlamaIndex and smolagents ship dangerous default ceilings.", "body_md": "Every model API has a ceiling on how many tokens it will generate in one response. When a generation reaches that ceiling, the model stops mid-sentence, mid-object, or mid-tool-call — and the API returns HTTP 200 with a well-formed body. Nothing throws. In a multi-step agent pipeline, the next step receives that half-finished answer and treats it exactly like a complete one.\n\nThree terms carry most of the weight here, so it is worth fixing them before the field names start multiplying. **Max output tokens** is the request-side ceiling on generated tokens — `max_tokens`\n\non most APIs, `max_output_tokens`\n\non OpenAI’s Responses API, `n_predict`\n\non llama.cpp’s server. **Truncation** is what happens when a generation stops because it reached that ceiling rather than because the model was finished. And the **stop reason** — also called the finish reason, the done reason, or the stop type, depending on whose API you are holding — is the metadata field that tells you which of those two things happened. It is the only signal you get.\n\nAnthropic states the premise better than any third party could. Its stop-reason guide opens with the instruction to read the field, and then draws the distinction the rest of this post depends on: *“Unlike errors, which indicate failures in processing your request, stop_reason tells you why Claude completed its response generation.”* A truncated response is not a failed request. It is a successful request whose result happens to be incomplete, and the two are indistinguishable unless you look at the metadata.\n\nThis post is a pattern reference, not a measurement. It collects the field name and the verbatim truncation value for eighteen API surfaces across fourteen vendor organisations, documents the cases where the signal is rewritten or lost between the engine and your client, and covers what detection actually requires. The per-model *numbers* — what each model’s maximum output is on each serving surface — are a companion dataset and are deliberately not built here; a later post owns that census. This one owns the failure mode.\n\n- 01Truncation is a metadata fact, not a content fact.Every surveyed API returns HTTP 200 and a well-formed body when the output ceiling is hit. The stop-reason field is the only reliable signal, and no vendor documents a content-level alternative.\n- 02One signal, six field names, seven truncation values.stop_reason, finish_reason, finishReason, stopReason, done_reason and stop_type across the surveyed surfaces, plus incomplete_details.reason nested inside OpenAI's Responses API. A single portable equality check does not exist.\n- 03A truncated tool call can arrive labelled as a completed one.vLLM issue #53269, filed and closed on August 21, 2026, documents a streaming handler that rewrote the terminal reason to finish_reason: \"tool_calls\" — a success value — for a call that max_tokens had cut off mid-arguments.\n- 04The dangerous defaults live in frameworks, not APIs.LlamaIndex core ships DEFAULT_NUM_OUTPUTS = 256, and smolagents carries two different hardcoded ceilings in one framework. Anthropic, Cohere, Ollama, llama.cpp, LangChain and the Vercel AI SDK all default to the model maximum or to unlimited.\n- 05Schema validation cannot detect a cut, and structurally it never could.Conformance machinery needs a complete document to conform. Truncated prose is still valid prose, and a truncated array can still parse into a shorter, syntactically perfect list.\n\n## 01 — The Failure ModeNothing throws, because *nothing failed*.\n\nThe mental model most engineers bring to an API call is that a problem produces an exception. Truncation defeats that model at the protocol level: the request was valid, the server processed it correctly, and the response body is exactly what the schema describes. The only thing wrong with it is that the content stops early, and the protocol has no way to express “this is fine” versus “this is a fragment” other than a small metadata field that most client code never reads.\n\nThe consequence in a single-call application is mild — a user sees a sentence end abruptly and asks again. The consequence in a chain is different in kind. A research step that was cut at 40% of its output hands a plausible-looking summary to a planning step, which hands a plan built on incomplete evidence to an execution step. Every downstream stage behaves correctly on the input it was given. Nothing in the trace is red. This is the same structural problem as the orchestration mistakes covered in our [agentic workflow anti-patterns guide](/blog/agentic-workflow-anti-patterns-orchestration-mistakes-2026), with one aggravating feature: the defect enters the chain from outside your code, carrying a success status.\n\nTwo design decisions make this worse than it needs to be. The first is that the client libraries mostly do not raise. The OpenAI Python SDK does define an exception for the case — `LengthFinishReasonError`\n\n, whose message reads *“Could not parse response content as the length limit was reached”* — but it is raised from the structured-output parse path, not from the ordinary create call. An open pull request against that SDK, [openai-python #3589](https://github.com/openai/openai-python/pull/3589) (opened August 10, 2026), states the asymmetry in the vendor’s own repository: *“The equivalent non-streaming client.chat.completions.create() never raises for the same response — it just returns a completion with finish_reason=“length”.”* Structured output gets an exception; plain text gets a value you have to go and look at.\n\nThe second is that the field you have to look at is not the same field from one API to the next, and neither is the value it carries. That is section 03, and it is the part worth bookmarking.\n\n##### A stop reason is not an error\n\nAnthropic's stop-reason guide: \"Unlike errors, which indicate failures in processing your request, stop_reason tells you why Claude completed its response generation.\" The response is a success. The content is a fragment.\n\n##### Parse path only\n\nopenai-python raises LengthFinishReasonError inside parse_chat_completion, guarded by a check for parseable input. The plain create() call returns a completion with finish_reason=\"length\" and no exception at all.\n\n##### Anthropic forces the choice\n\nThe Messages API types max_tokens as Required[int] — \"The maximum number of tokens to generate before stopping.\" There is no silent default because the caller must pick one. That is the design with no surprise in it.\n\n## 02 — The CaseA truncated tool call, reported as a *completed* one.\n\nThe strongest documented instance of this failure is [vllm-project/vllm issue #53269](https://github.com/vllm-project/vllm/issues/53269), filed and closed on August 21, 2026, and titled *“[Bug]: streaming reports finish_reason=”tool_calls“ for a tool call truncated by max_tokens, hiding ”length“”*. It matters because the signal was not merely absent. It was replaced with a value that means the opposite.\n\nThe report describes the mechanism precisely: *“In the streaming chat handler, once any tool-call delta has been emitted the terminal reason is rewritten to “tool_calls” unconditionally. That discards the engine’s actual reason, so a generation cut short by max_tokens part way through the arguments is reported to the caller as a * The engine knew. vLLM’s own\n\n**completed** tool call.”\n\n`FinishReason`\n\nenum documents `length`\n\nas *“max_tokens was consumed, or max_model_len was reached”*. The frontend overwrote it on the way out.\n\nThe caller is told the tool call is complete while holding arguments that are not valid JSON — json.loads on tool_calls[0].function.arguments raises, and a client that trusts finish_reason has no signal that anything was cut off.vllm-project/vllm issue #53269, August 21, 2026\n\nTwo details make this more than a single bug report. The first is that the same request answered differently depending only on whether it was streamed. In the reporter’s words: *“The non-streaming path does not have this problem — it only performs the translation when output.finish_reason == “stop”. So the same request answers differently depending only on stream, and the streaming answer is the wrong one: OpenAI reports “length” on truncation.”* A test suite that exercises the non-streaming path and a production deployment that streams will disagree about whether the bug exists.\n\nThe second is the reproduction. The report carries a swept reproduction — greedy decoding, temperature 0, a fixed seed, and `max_tokens`\n\nswept across the truncation point on vLLM 0.27.0 with `--enable-auto-tool-choice --tool-call-parser openai`\n\n. At`max_tokens`\n\nvalues of 64, 80, 96 and 112 the non-streaming path reported `length`\n\nwhile the streaming path reported `tool_calls`\n\n, and the streamed arguments did not parse as JSON. From 128 upward both paths reported `tool_calls`\n\nand the arguments parsed. That is one person’s reproduction on one version in one bug report, not a measurement of vLLM in general — but it is a reproduction with the version string, the flags and the sweep written down, which is more than most reports of this class carry.\n\nIt is also not an isolated incident. Two earlier pull requests attack the same bug class: [#46303](https://github.com/vllm-project/vllm/pull/46303), *“Keep length finish_reason for max_tokens-truncated** streaming tool calls”* (opened June 21, 2026), and [#47963](https://github.com/vllm-project/vllm/pull/47963), *“Report finish_reason=’length’ for tool calls truncated by max_tokens in streaming”* (opened July 8, 2026). Three dated items on one seam in ten weeks makes it a recurring class rather than a one-off, and the seam is structural: any layer that translates a terminal reason has an opportunity to lose the original.\n\nA serving framework shipped a fix for the downstream half of the same problem a week later. NVIDIA’s Dynamo, in [PR #13986](https://github.com/ai-dynamo/dynamo/pull/13986) (opened August 28, 2026), describes dispatching a tool call from a first streamed fragment holding only a prefix such as `{\"path\":\"/a/very`\n\n, then deduplicating the later fragments that would have completed the arguments. The PR’s own summary of the consequence is the cleanest sentence on the subject: *“A harness that trusts the event — which is the entire point of the event — was handed truncated JSON to execute.”* The same PR notes why bugs like this hide: *“The failure is currently masked on the vLLM path, where long string arguments happen to arrive in one frame.”* An unrelated framing coincidence upstream can keep a truncation bug invisible until the coincidence goes away.\n\n*success*reason is a different category of problem, because there is nothing left in the response to check. The stop reason was the only signal, and it now says the generation finished normally.\n\n## 03 — The ReferenceSix field names for *one* signal.\n\nThe table below is the reason this post exists. It lists the field that carries the truncation signal, the truncation value quoted verbatim from the vendor’s own documentation or source tree, and what happens to that signal when the response is streamed — for eighteen API surfaces across fourteen vendor organisations. Documentation states are current as of retrieval on August 30, 2026; source-tree reads are from each project’s default branch on the same date.\n\nThree structural facts fall out of it. Six different field names carry the signal (`stop_reason`\n\n, `finish_reason`\n\n, `finishReason`\n\n, `stopReason`\n\n, `done_reason`\n\n, `stop_type`\n\n), plus `incomplete_details.reason`\n\nnested inside a seventh shape. Seven distinct values mean truncation, and two of them do not contain the word “length” at all. And the same two words can mean opposite things across two APIs: on Anthropic’s Messages API `stop_reason`\n\nis the truncation field, while inside vLLM `stop_reason`\n\nis a different field entirely — *“The stop string or token id that caused the completion to stop”* — and `finish_reason`\n\nis the one you want.\n\n| Vendor / API | Field name | Truncation value (verbatim) | Streaming behaviour | Source |\n|---|---|---|---|---|\n| Frontier vendor APIs | ||||\n| Anthropic Messages API | `stop_reason` | `max_tokens` — “The response reached your max_tokens limit.” Also `model_context_window_exceeded` — “The response filled the model’s context window.” | null in `message_start` ; provided in `message_delta` ; not provided in any other event | docs.anthropic.com · stop reasons |\n| OpenAI Chat Completions | `finish_reason` | `length` — “if the maximum number of tokens specified in the request was reached” | same field on `chat_completion_chunk` , typed nullable; carried on the terminal chunk | platform.openai.com · chat/object |\n| OpenAI Responses API | `status` + `incomplete_details.reason` | `status: “incomplete”` with `reason: “max_output_tokens”` | dedicated terminal event `type: “response.incomplete”` | platform.openai.com · responses/object |\n| Google Gemini generateContent | `finishReason` | `MAX_TOKENS` — “The maximum number of tokens as specified in the request was reached.” | per candidate; “If empty, the model has not stopped generating tokens.” Companion `finishMessage` set only when `finishReason` is set | ai.google.dev · generate-content |\n| Google GenAI Python SDK | `finish_reason` (`FinishReason` enum) | `MAX_TOKENS` — “Token generation reached the configured maximum output tokens.” | same enum on streamed candidates | googleapis/python-genai · types.py |\n| Mistral chat completions | `finishReason` / `finish_reason` | two values — `length` and `model_length` , defined in Mistral’s official TypeScript SDK as `Length: “length”` and `ModelLength: “model_length”` | not documented on the retrieved page | mistralai/client-ts · chatcompletionchoice.ts |\n| Cohere Chat v2 | `finish_reason` | `MAX_TOKENS` — “the finish_reason field in the response will be set to ’MAX_TOKENS’”; allowed values COMPLETE, STOP_SEQUENCE, MAX_TOKENS, TOOL_CALL, ERROR, TIMEOUT | not documented on the retrieved page | docs.cohere.com · reference/chat |\n| AWS Bedrock Converse | `stopReason` | `max_tokens` — also `model_context_window_exceeded` , `malformed_model_output` and `malformed_tool_use` in the same nine-value enum | ConverseStream — not retrieved | docs.aws.amazon.com · API_Converse |\n| Gateways and OpenAI-compatible surfaces | ||||\n| OpenRouter | `finish_reason` + `native_finish_reason` | normalised `length` ; “The raw finish_reason string returned by the model is available via the native_finish_reason property.” | final chunk before `[DONE]` ; “Unlike OpenAI’s spec, this chunk contains a non-empty choices array… that repeats the finish_reason of the stream.” | openrouter.ai · api_reference |\n| Fireworks | `finish_reason` | `length` — “the message content may be partially cut off… In this case the return value might not be a valid JSON.” | metrics arrive in “the final chunk (when finish_reason is set)” | fireworks.ai · post-chatcompletions |\n| LiteLLM | `finish_reason` + `provider_specific_fields` | normalised `length` ; original kept in `native_finish_reason` when it differs | not stated on the cited page | docs.litellm.ai · completion/output |\n| Self-hosted servers | ||||\n| vLLM (engine) | `finish_reason` , plus a separate `stop_reason` | `length` — “max_tokens was consumed, or max_model_len was reached” | see issue #53269 — the streaming chat handler rewrote the terminal reason to `tool_calls` once a tool-call delta had been emitted | vllm · v1/engine/__init__.py |\n| llama.cpp server (native) | `stop_type` , plus boolean `truncated` | `limit` — “Stopped because n_predict tokens were generated before stop words or EOS was encountered” | “only content, tokens and stop will be returned until end of completion” | ggml-org/llama.cpp · server README |\n| Ollama (native API) | `done_reason` | `“length”` , from `DoneReason.String()` — not enumerated in `docs/api.md` or `api/types.go` | appears on the final `done: true` object | ollama · llm/server.go |\n| Ollama (OpenAI-compatible shim) | `finish_reason` | `“length”` passes through correctly; the connection-closed reason serialises to the empty string and `cmp.Or(r.DoneReason, “stop”)` fills in `“stop”` | `FinishChunk` emits a dedicated finish-reason chunk with an empty delta | ollama · openai/openai.go |\n| Client libraries and orchestration layers | ||||\n| Vercel AI SDK (v4 provider interface) | `finishReason.unified` + `finishReason.raw` | `’length’` — “model generated maximum number of tokens” | same object on the end-of-call callbacks | vercel/ai · language-model-v4-finish-reason.ts |\n| LangChain (ChatAnthropic) | `response_metadata[“stop_reason”]` | raw Anthropic value, unmodified; never raised as an exception | populated on the final chunk from `message_delta` | langchain · chat_models.py |\n| OpenAI Python SDK (parse path) | raises `LengthFinishReasonError` | “Could not parse response content as the length limit was reached” | raised from `get_final_completion()` ; plain `create()` “never raises… it just returns a completion with finish_reason=’length’” | openai/openai-python · PR #3589 |\n\nEmpty cells in that table are honest. Where a vendor’s retrieved documentation does not state the streaming placement, the cell says so rather than carrying an inference. One surface is missing altogether: Together AI’s `finish_reason`\n\nenum was not enumerated on the retrieved reference page, so it has no row here at all. A reference whose value is that every cell is quotable cannot fill one in from memory.\n\nThree details in the table are worth pulling out because they break assumptions rather than merely varying. Anthropic splits truncation into two reasons — your ceiling (`max_tokens`\n\n) and the model’s context window (`model_context_window_exceeded`\n\n), the second of which is *“currently typed only in the SDKs’ beta namespace”* and requires the `model-context-window-exceeded-2025-08-26`\n\nbeta header on models earlier than Sonnet 4.5. Cohere merges the same two conditions into one `MAX_TOKENS`\n\nvalue covering both “the model’s context length” and “the value specified via the max_tokens parameter”. So a normalisation layer between them cannot round-trip the distinction in either direction.\n\nAnd Gemini’s `FinishReason`\n\nenum is by far the largest surveyed, at 21 values. Exactly one of them — `MAX_TOKENS`\n\n— means you hit your ceiling. Four more (`MALFORMED_FUNCTION_CALL`\n\n, `UNEXPECTED_TOOL_CALL`\n\n, `TOO_MANY_TOOL_CALLS`\n\n, `MALFORMED_RESPONSE`\n\n) mean the output is unusable for other reasons. The common client shape — a switch on `STOP`\n\nwith everything else falling into a default branch — swallows twenty distinct conditions into one.\n\n##### OpenRouter *native_finish_reason*\n\nOpenRouter normalises each model's finish_reason to five values for portability and keeps the original in native_finish_reason. You get a stable check and a lossless record of what the provider actually said.\n\n##### LiteLLM *provider_specific_fields*\n\nSame pattern, and LiteLLM's docs state the reason it matters: \"useful for agent loops that need to distinguish between different stop conditions (e.g., Gemini's MALFORMED_FUNCTION_CALL vs a normal stop).\"\n\n##### AI SDK provider *mappers*\n\nThe Anthropic mapper maps max_tokens and model_context_window_exceeded to 'length' and everything unrecognised to 'other'. A new vendor truncation value becomes 'other' until the mapper is updated — raw is the only place it survives.\n\n##### Ollama’s *connection-closed* path\n\nOllama's max-tokens path maps correctly: DoneReasonLength returns \"length\" and the shim passes it through. It is the connection-closed reason that degrades — it falls through to an empty string, and cmp.Or(r.DoneReason, \"stop\") turns that into a clean finish_reason: \"stop\".\n\n## 04 — StreamingThe signal arrives *once*, in one event.\n\nStreaming is where consumers lose the signal without any bug being involved, because the field is not on the frames most code reads. Anthropic documents the placement exactly: *“When using streaming, stop_reason is: null in the initial message_start event; Provided in the message_delta event; Not provided in any other events.”* A consumer that subscribes to\n\n`content_block_delta`\n\nto append text to a buffer — which is the obvious way to write a streaming client — never touches the event that carries the reason.The pattern repeats with local variations. OpenAI’s Chat Completions types `finish_reason`\n\nas nullable on the streaming chunk schema, where the non-streaming object has no null: every intermediate chunk carries null and exactly one chunk carries the value. The Responses API emits a dedicated terminal event, `type: “response.incomplete”`\n\n, which means a stream consumer listening only for `response.completed`\n\nwaits for an event that is never coming. OpenRouter deliberately diverges from OpenAI’s spec by putting a content-free choice with the finish reason in the final chunk before `[DONE]`\n\n— correct behaviour that a strict OpenAI parser may discard as malformed. And llama.cpp’s native server does not send the field during a stream at all: *“In streaming mode ( stream), only content, tokens and stop will be returned until end of completion.”*\n\nOllama’s OpenAI-compatible shim deserves a precise statement, because it is easy to get backwards. The **max-tokens** path is correct: Ollama’s internal `DoneReasonLength`\n\nstringifies to `“length”`\n\n, and the shim passes that through. It is the **connection-closed** path that degrades — `DoneReasonConnectionClosed`\n\nfalls through the switch to the default branch and serialises to the empty string, and the shim’s `FinishChunk`\n\nthen applies `cmp.Or(r.DoneReason, “stop”)`\n\n, which fills the empty value with `“stop”`\n\n. So a generation cut off by a dropped connection reaches an OpenAI-shaped client as a clean, natural completion. A generation cut off by the token ceiling does not.\n\nThe practical rule for any streaming consumer is that reading the content stream is not enough. You need the terminal event, whatever it is called on your surface, and you need to record what it said — which is an observability requirement as much as a correctness one. Our [guide to trace-quality mistakes](/blog/agent-observability-anti-patterns-trace-quality-mistakes-2026) covers the span-level side of that: the stop reason belongs on the span, next to the token counts, on every model call.\n\n## 05 — ShapeWhy JSON-shaped truncation is worse than a *crash*.\n\nFireworks writes the consequence into its API reference more plainly than anyone: *“Also note that the message content may be partially cut off if finish_reason=“length”, which indicates the generation exceeded max_tokens or the conversation exceeded the max context length. In this case the return value might not be a valid JSON.”* That sentence is doing something unusual for API documentation: it is telling you that a successful response may contain a payload your parser cannot read.\n\nIt helps to order the outcomes by what the consumer can still detect. The ladder below runs from “annoying but visible” to “undetectable from the response”, and the last rung is the one worth thinking about before it happens to you.\n\n| Rung | Shape of the output | Detectable by | Sourced to |\n|---|---|---|---|\n| 1 | Truncated prose | the stop reason only | Anthropic streaming placement, stop-reason guide |\n| 2 | Truncated JSON that fails to parse | the stop reason, or a parse error | Fireworks — “might not be a valid JSON” |\n| 3 | Truncated tool call reported with a truncation reason | the stop reason plus the type of the last content block | Anthropic’s own two-field detection sample |\n| 4 | Truncated tool call reported with a success reason | nothing in the response | vllm-project/vllm #53269, Aug 21, 2026 |\n| 5 | Truncated JSON that parses into something valid but short | nothing at all | reasoning about the failure mode — no vendor documents this case |\n\nRung 5 needs its label kept on. It is reasoning about the shape of the failure, not a documented case and not a measured one: a cut inside an array of objects can leave a document that is syntactically complete and semantically short — ten results where twenty were requested, and nothing anywhere saying so. No vendor documents a rate for it and there is no number to give. The two practitioner reports that come closest describe the adjacent shape, truncated tool-call arguments being silently replaced with an empty object.\n\nThis is also the boundary with a subject that already has its own post, and the boundary is sharp. Schema conformance — grammar- constrained decoding, strict tool-use modes, the portability limits of one shared schema across vendors — is the subject of our guide to [structured output reliability in production](/blog/llm-structured-output-json-reliability-production). None of that machinery detects a cut, and structurally it cannot: conformance needs a complete document to check, and truncation is precisely the case where no complete document exists. The two problems look similar in a stack trace and have nothing in common in their causes. If you are working the parse-path angle specifically, our [OpenAI structured outputs guide](/blog/openai-structured-outputs-complete-guide) covers where the SDK does raise.\n\nFireworks documents one more wrinkle worth knowing, because it turns a truncation into a latency incident first: *“when using JSON mode, it’s crucial to also instruct the model to produce JSON via a system or user message. Without this, the model may generate an unending stream of whitespace until the generation reaches the token limit, resulting in a long-running and seemingly ’stuck’ request.”* The request that eventually returns a truncated response can spend its whole budget emitting nothing at all.\n\n## 06 — DefaultsThe low ceilings live in *frameworks*, not APIs.\n\nThe intuitive place to look for a dangerous default is the vendor API. That is the wrong place. Across the surveyed surfaces, the APIs and first-party SDKs either require you to choose a ceiling or default to the model’s own maximum. The fixed low numbers are in the frameworks that sit between you and the API — the layer you are least likely to be reading when you debug a short answer.\n\n#### Hardcoded output ceilings in agent frameworks · tokens\n\nSource: run-llama/llama_index constants.py and huggingface/smolagents models.py, default branch, read 2026-08-30LlamaIndex core ships `DEFAULT_NUM_OUTPUTS = 256 # tokens`\n\nin its constants module. Phrase that carefully: the framework-level default constant is 256, which is roughly 190 English words. Individual LLM integrations may override it, and this post did not check every one, so it is not a claim that every LlamaIndex call is capped at 256 — it is a claim about the constant the framework falls back to.\n\nsmolagents is the more surprising row, because the ceiling depends on which backend you selected. Its vLLM backend uses `max_tokens=kwargs.get(“max_tokens”, 2048)`\n\n, while its Transformers backend resolves through a chain of optional keyword arguments and falls through to `1024`\n\n. Two different silent ceilings inside one agent framework, differing by a factor of two, selected by a configuration choice that has nothing to do with output length.\n\n| SDK / framework | Default | Source |\n|---|---|---|\n| Capped — a fixed number low enough to truncate real work | ||\n| LlamaIndex core | `DEFAULT_NUM_OUTPUTS = 256 # tokens` (with `DEFAULT_CONTEXT_WINDOW = 3900` ) | run-llama/llama_index · llama-index-core constants.py |\n| smolagents — vLLM backend | `max_tokens=kwargs.get(“max_tokens”, 2048)` | huggingface/smolagents · models.py |\n| smolagents — Transformers backend | falls through the keyword chain to `1024` | huggingface/smolagents · models.py |\n| Safe — resolves to the model maximum, or to unlimited | ||\n| Anthropic Messages API | no default — `max_tokens: Required[int]` | anthropic-sdk-python · message_create_params.py |\n| Cohere Chat v2 | “If not set, max_tokens defaults to the model’s maximum output token limit.” Over-setting it silently caps at that maximum rather than erroring | docs.cohere.com · reference/chat |\n| llama.cpp server | `n_predict` — “Default: -1, where -1 is infinity” | ggml-org/llama.cpp · server README |\n| Ollama | `DefaultOptions()` sets `NumPredict: -1` | ollama · api/types.go |\n| LangChain ChatOpenAI | `max_tokens: int | None = Field(default=None, alias=“max_completion_tokens”)` | langchain-openai · chat_models/base.py |\n| LangChain ChatAnthropic | `default=None` , docstring: “If not specified, this is set dynamically using the model’s max_output_tokens” | langchain-anthropic · chat_models.py |\n| Vercel AI SDK | `maxOutputTokens` documented with no default | ai-sdk.dev · ai-sdk-core/settings |\n| Hugging Face transformers | `GenerationConfig` pops both `max_length` and `max_new_tokens` as `None` ; a per-checkpoint `generation_config.json` can still cap | huggingface/transformers · configuration_utils.py |\n| OpenAI Chat Completions | request-side default for `max_completion_tokens` — not documented on the retrieved reference | platform.openai.com · chat/object |\n| OpenAI Responses API | documented minimum for `max_output_tokens` is 16 — a minimum, not a default | platform.openai.com · responses/object |\n\nTwo entries in the safe group are worth reading as design lessons. Anthropic’s Messages API has no default because `max_tokens`\n\nis a required parameter — the design that forces an explicit choice is the design with no surprise in it. LangChain’s `ChatAnthropic`\n\nresolves the same required parameter dynamically from the model’s own `max_output_tokens`\n\nrather than pinning a constant, which is the right way to satisfy a required field on the caller’s behalf. Any advice you may have read that this integration pins a hardcoded 1024 is out of date against the current source.\n\nOne folk claim deserves killing while we are here. The widely repeated line that Hugging Face `transformers`\n\ndefaults to 20 output tokens does not hold against the current source: the `GenerationConfig`\n\nconstructor pops both `max_length`\n\nand `max_new_tokens`\n\nas `None`\n\n. A particular checkpoint’s `generation_config.json`\n\ncan still set a low value, which is a per-model fact, not a library default — and worth checking on the checkpoint you actually load.\n\nRaising a ceiling is not free, which is the one place this subject touches cost. Anthropic’s extended thinking shares the budget: *“Requires a minimum budget of 1,024 tokens and counts towards your max_tokens limit”*, so turning thinking on without raising the ceiling reduces what is left for the answer. And OpenAI’s\n\n`max_output_tokens`\n\non the Responses API is *“An upper bound for the number of tokens that can be generated for a response, including visible output tokens and reasoning tokens”*— on a reasoning model, thinking can consume the whole ceiling and leave a structurally valid, visibly empty response. The pricing side of running with generous ceilings is covered in our work on\n\n[long-context pricing thresholds](/blog/long-context-pricing-thresholds-llm-cost-cliffs).\n\n## 07 — DetectionWhat actually *works*, in order.\n\nDetection has one primary mechanism and three supporting ones, and it is worth being clear about which is which. The primary mechanism is reading the stop reason on every model call, on the surface you are actually calling, including the streaming terminal event. Everything else corroborates.\n\n##### Read the terminal reason, always\n\nAssert on the field your surface uses — and on both fields where the surface has two, as llama.cpp does with stop_type and the boolean truncated. Treat an unrecognised value as suspect rather than routing it to a default branch.\n\n##### Output-token *accounting*\n\nEvery surveyed API returns a usage object alongside the reason, so output tokens landing at the configured ceiling is a strong second signal. Use a tolerance, not equality: llama.cpp warns its limit \"may exceed the set limit slightly\" on a partial multibyte character.\n\n##### Structural validation\n\nParsing the payload catches rung 2 and nothing else. It cannot catch truncated prose, because truncated prose is still valid prose, and it cannot catch a truncated array that parses into a shorter list.\n\n##### Sentinel terminators\n\nInstruct the model to end its output with an agreed marker and check for it. This survives a normalisation layer because it does not depend on the vendor's field at all — but it is a convention your pipeline defines and enforces, not an API feature any vendor offers.\n\nAnthropic’s own recommended handling for the tool-call case is a two-field check rather than a single comparison: test whether `stop_reason`\n\nequals `“max_tokens”`\n\n, then inspect the last content block and see whether its type is `tool_use`\n\n. That shape generalises. The reason alone tells you the generation was cut; the shape of what you are holding tells you what to do about it.\n\nThe sentinel idea is worth stating carefully because it is the pattern most likely to be mistaken for a feature. No vendor documents a canonical sentinel convention. Anthropic’s sample handling does the inverse — it appends a literal notice such as `[Response truncated due to max_tokens limit]`\n\nto the text it returns to a user — which is the same idea run the other way: a marker a consumer can look for. If you adopt one, your pipeline defines it, your prompts request it, and your validation enforces it.\n\nTwo things follow for how you instrument this. First, the check belongs in the pipeline, not in the model. Truncation is one of the clearest cases in the taxonomy of [what an agent can and cannot verify about its own output](/blog/agent-self-verification-limits-by-output-modality-2026): a model that was cut off cannot report that it was cut off, because the report would have to come after the token that never arrived. Only external metadata can tell you. Second, whatever you read should be recorded. LangChain surfaces the raw Anthropic value in `response_metadata[“stop_reason”]`\n\nand never raises on it, which makes a one-line assertion in the chain step the cheapest fix available. Observability vendors are treating this as a first-class case too — LiteLLM merged test coverage for empty, whitespace and truncated tool-call arguments in its OpenTelemetry output in August 2026.\n\n## 08 — RemedyContinue the prose, *re-run* the tool call.\n\nOnce you can see truncation, the response depends on what got cut, and the vendor guidance is more specific than it first appears. Anthropic’s stop-reason table prescribes *“Raise max_tokens or continue the response”* for the `max_tokens`\n\ncase — two options for prose. For the tool-call case the same page prescribes something different: *“If Claude’s response is cut off because it hit the max_tokens limit, and the truncated response contains an incomplete tool use block, you’ll need to retry the request with a higher max_tokens value to get the full tool use.”* A full retry, not a continuation.\n\nThat distinction is the most useful operational line in this post. Continuation works on prose because prose concatenates: the second half attaches to the first and the result reads correctly. It does not work on a structured emission, because a tool call cut mid-arguments has no valid prefix to continue from — you are holding a fragment of a serialisation, and stitching a second fragment onto it produces something no schema described. The vendor’s own guidance is that you re-run it at a higher ceiling.\n\nFor the second Anthropic value, `model_context_window_exceeded`\n\n, the same table’s prescription is simply to *“Treat the response as truncated”* — raising your request ceiling does not help when the model’s own window is what filled. That is a prompt-size and context-management problem rather than a parameter problem, and it belongs with the runtime context work in our [context engineering playbook](/blog/context-engineering-agent-reliability-playbook-2026).\n\nOne more habit is worth building in early: decide what a truncated step should do to the run. Discarding the partial output and retrying costs tokens; passing it downstream costs correctness; failing the run loudly costs a restart. Whichever you choose, choose it explicitly and log the reason, because the alternative — the default in most pipelines today — is to pass the fragment along and find out three steps later.\n\n## 09 — ConclusionOne field, read on *every* call.\n\n### Truncation is a metadata fact. If nothing in your pipeline reads the metadata, a half-finished answer is indistinguishable from a finished one.\n\nThe failure mode is structural rather than incidental. Model APIs report a completed request that produced incomplete content, and the only thing separating the two is a field that most client code steps over. Eighteen documented surfaces spell that field six different ways and fill it with seven different truncation values, two of which do not contain the word “length”. A single portable equality check does not exist, which is why the table above is the useful part of this post.\n\nThe signal also degrades in transit, and there is dated first-party evidence for each step of that. Gateways like OpenRouter and LiteLLM do it correctly — normalise for portability, preserve the raw value for correctness. The AI SDK’s provider mappers collapse unrecognised values to a generic bucket. Ollama’s connection-closed path serialises to an empty string that its OpenAI-compatible shim fills in as `“stop”`\n\n, while its max-tokens path maps correctly. And vLLM’s streaming handler, in issue #53269 on August 21, 2026, rewrote the terminal reason to a success value for a tool call that had been cut mid-arguments.\n\nNone of the remedies is difficult. Read the terminal reason on every call and on the streaming path specifically; corroborate with output token counts against your configured ceiling; check where your ceiling came from, because the low ones live in frameworks rather than APIs; and remember that a truncated tool call gets re-run at a higher ceiling rather than continued. The reason this is worth writing down is not that the fix is hard — it is that nothing in the system will ever prompt you to apply it.", "url": "https://wpnews.pro/news/when-your-ai-agent-quietly-gets-a-half-finished-answer", "canonical_source": "https://www.digitalapplied.com/blog/agent-output-truncation-silent-failures", "published_at": "2026-08-29 00:00:00+00:00", "updated_at": "2026-08-30 11:23:57.409965+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["Anthropic", "OpenAI", "vLLM", "LlamaIndex", "smolagents", "Cohere", "Ollama", "Vercel AI SDK"], "alternates": {"html": "https://wpnews.pro/news/when-your-ai-agent-quietly-gets-a-half-finished-answer", "markdown": "https://wpnews.pro/news/when-your-ai-agent-quietly-gets-a-half-finished-answer.md", "text": "https://wpnews.pro/news/when-your-ai-agent-quietly-gets-a-half-finished-answer.txt", "jsonld": "https://wpnews.pro/news/when-your-ai-agent-quietly-gets-a-half-finished-answer.jsonld"}}