An answer that stops mid-sentence is not a mystery, because the API told you why. The field is in every response and almost every integration ignores it — which is how a truncation ends up being debugged as a prompt problem for two days.
OpenAI-compatible responses carry choices[0].finish_reason
; Anthropic’s carry stop_reason
on the message. Same concept, different vocabulary, and both are populated on every non-streaming response. The single highest-value change most codebases can make in this area is one assertion at the boundary:
OK = {"stop", "end_turn", "tool_calls", "tool_use", "function_call", "stop_sequence"}
def unwrap(response):
reason = (getattr(response.choices[0], "finish_reason", None)
or getattr(response, "stop_reason", None))
if reason is None:
raise IncompleteResponse("no stop reason -- stream ended early")
if reason not in OK:
raise IncompleteResponse(f"finish_reason={reason}")
return response.choices[0].message.content
Without it, a truncated JSON object arrives at your parser, the parser raises a syntax error, and the error you investigate is three layers away from the cause.
Put the same value in your telemetry as a dimension on every request, not only in the error path. The distribution of stop reasons over a week is one of the most informative cheap metrics available: a rising share of length
means your budgets no longer fit the work, a nonzero share of content_filter
means a policy surface is engaging with your traffic and nobody has looked at which prompts, and a share of missing reasons means your transport is dropping streams. All three are invisible if the field is only read when something already threw.
| Value | Description |
|---|---|
| stop / end_turn | Normal completion. The model emitted its end-of-turn token because it had finished. This is the only value that means the answer is whole. |
| length / max_tokens | The output budget ran out. The generation was cut mid-stream and the model had more to say. Never retry blindly — see below, there are three distinct causes. |
| stop_sequence | One of your own stop strings appeared in the output. Common self-inflicted wound: a stop sequence of a double newline against a model that formats with blank lines, or a closing brace against a model emitting nested JSON. |
| content_filter | A safety system cut the response, sometimes after partial output has already streamed. The visible text may look like a normal short answer. Distinct from a model-authored refusal, which completes normally with finish_reason 'stop'. |
| tool_calls / tool_use | Not a truncation at all. The model stopped because it is waiting for you to run a tool and return the result. If your code treats this as an incomplete answer you will drop tool calls silently. |
| refusal | Some APIs surface a declined request as its own stop reason with a structured refusal field, rather than as prose. Route it to your refusal handling, not to your retry logic. |
| _turn | Used for long-running server-side tool loops: the turn is d, not finished, and you are expected to send the response back to continue. Treating it as a completion truncates a multi-step task. |
| null / absent | In a stream, an intermediate chunk legitimately has no finish reason. In a final response it means the stream terminated without a terminal event, which is a transport failure — the most under-handled case in this table. |
Three different causes produce this one value, and they need different fixes.
max_tokens
is genuinely too small.max_tokens
here produces an error rather than a longer answer. The fix is on the input side.And one non-cause that is worth ruling out: a repetition loop that ran until the limit. The stop reason says length, the actual bug is on the repetition loops page, and raising the budget makes the bill larger without making the answer better.
The failure with no error attached. In server-sent-events streaming the response is a sequence of chunks and completion is signalled by a final chunk carrying a finish reason (and, in the OpenAI dialect, a [DONE]
sentinel). If the connection drops, an intermediate proxy times out, a load balancer closes an idle connection, or the client library exits its loop on an exception it swallowed, your stream simply ends. The text you accumulated looks like a short answer. Nothing raises.
Guard it explicitly: track whether a terminal event was observed and treat its absence as a failure. Common causes are worth knowing — proxy and load-balancer idle timeouts shorter than your longest generation, buffering layers that hold chunks until a timeout, and serverless platforms with a hard response-duration cap. If truncation correlates with long answers and disappears when you call the provider directly, the problem is in your own infrastructure and no model parameter will fix it.
The reasoning-model variant of this deserves its own note, because it looks like a hang rather than a truncation. A model that thinks for thirty seconds before emitting a visible token sends nothing during that window, and an idle-connection timeout tuned for a chat stream will close the connection before the first content chunk arrives. From the client’s side the request simply ends with nothing in it. Set the timeout against the time to first visible token, not against a typical streaming interval, and prefer a provider stream that emits keepalive events over one that does not.
One more asymmetry to design around: with streaming, a partial answer has already reached the user by the time you detect the problem. You cannot silently retry and replace it. Either buffer complete responses before rendering — losing the latency benefit that streaming exists for — or render progressively and have a defined way to append a correction or mark the answer as incomplete. Deciding this after shipping usually means the user sees half an answer and no indication that it is half.
Stop-reason vocabularies differ between vendors, and a gateway either normalises them or leaves you with a union type to handle. Whichever you use, check the documented response fields rather than inferring completion from the text, because the text of a truncated answer is indistinguishable from the text of a short one.