llama-server's Responses API marks truncated replies and tool calls as completed

A developer testing llama.cpp build b11753 found that llama-server's OpenAI-compatible /v1/responses endpoint reports truncated replies and tool calls as "completed" with incomplete_details set to null, while the same requests on /v1/chat/completions correctly return finish_reason: "length". In tests with gemma-3-1b-it and Qwen3-0.6B, an 8-token limit returned the text "1 2 3 4" marked completed, and a cut-off write_file tool call returned invalid JSON arguments ({"path": "/) also marked completed, with streaming ending in response.completed and no function_call_arguments.done event. The issue is filed upstream as ggml-org/llama.cpp#30773.

TL;DR : llama-server's OpenAI-compatible /v1/responses endpoint never reports a reply as cut off. On llama.cpp b11753, a request with max output tokens: 8 came back with "status": "completed" , incomplete details: null and the text 1 2 3 4 , stopped mid-count. A tool call cut off by the limit came back as a function call item with arguments {"path": "/ , which is not valid JSON, and both the item and the response said completed . The streaming version ended with response.completed and never sent function call arguments.done . The same requests on /v1/chat/completions said finish reason: "length" . A client that trusts status , which is how the Responses API is meant to be used, will treat half an answer as a whole one, or run a tool with broken arguments. The one signal left is usage.output tokens equal to the limit you sent. Upstream: ggml-org/llama.cpp 30773 https://github.com/ggml-org/llama.cpp/issues/30773 . A disposable LXC container, the b11753 release binary, gemma-3-1b-it-Q4 K M and Qwen3-0.6B-Q8 0 , --jinja -c 4096 . A request that can't finish within its limit: POST /v1/responses {"model": "x", "input": "Count from 1 to 200, separated by spaces.", "max output tokens": 8} {"status": "completed", "incomplete details": null, "output tokens": 8, "text": "1 2 3 4 " } The same question with room to finish used 52 tokens and also said completed , so status is the same whether the reply ended or was cut. On the chat endpoint: POST /v1/chat/completions ... "max tokens": 8 finish reason= length completion tokens= 8 With streaming, the last events of the 8-token request were: response.content part.done, response.output item.done, response.completed status=completed incomplete details=None There is no response.incomplete event. A tool call is where it matters most. One tool, write file path, content , Qwen3-0.6B, temperature: 0 , and /no think in the prompt so the model calls the tool straight away: responses limit=20: status=completed usage.output tokens=20 args='{"path": "/' chat limit=20: finish reason=length completion tokens=20 tool calls= 'write file', '{"path": "/' responses limit=400: status=completed usage.output tokens=89 args='{"path": "/tmp/poem.txt", "content": "Th...' chat limit=400: finish reason=tool calls completion tokens=89 At a 20-token limit the Responses API returned a function call item marked completed whose arguments end after {"path": "/ . At 30 they ended after "content": " . Both endpoints produced the same truncated arguments; only the chat endpoint said so. In the stream there were five response.function call arguments.delta events, then response.output item.done and response.completed . No response.function call arguments.done , and that event was also missing from a call that completed normally. Short limits are rare in interactive use, so most replies really are complete, and completed is the right answer most of the time. The ones that aren't are the long ones: a big file write, a long code edit, a model that thinks for a while first. Those are exactly the replies where an agent is about to do something with the output. The model also isn't wrong. The text is what the model generated, and in the tool-call case the arguments are a correct prefix of a correct call. Nothing errors until the client tries to parse the arguments, and a lenient JSON parser or a retry loop can hide even that. When the call does fail, it looks like the model produced bad JSON, which small models are known for. And the chat endpoint is fine. Anyone who checked finish reason on /v1/chat/completions has seen length work. The Responses API is newer, it's what Codex CLI and other recent agent tools speak, and the only thing llama-server's README says about how it works is that it "works by converting Responses request into Chat Completions request." The conversion is only one way. The request is turned into a chat completion and generated normally, and the server knows the reply stopped at the limit, because the chat endpoint reports length from the same generation. When the result is turned back into a Responses object, in tools/server/server-task.cpp , the response status and the status of every output item message, reasoning, function call are written as the literal string "completed" . At the b11753 tag that is seven {"status", "completed"} entries across to json oaicompat resp and to json oaicompat resp stream , the non-streaming and streaming builders. Neither reads the stop reason, and the string incomplete does not appear anywhere in the file. The Responses specification has status: "incomplete" with incomplete details.reason: "max output tokens" for exactly this case; llama-server never produces it. There's no fix upstream yet; the issue has no replies or PR as of 20 October. On the client side: usage.output tokens == max output tokens as cut off. function call whose arguments don't parse as JSON was cut off, whatever its status says. /v1/chat/completions if your client can. finish reason: "length" is reported correctly there, for text and for tool calls. The toolkit's check-llama-responses-status.sh sends one 8-token request to /v1/responses and one to /v1/chat/completions , and compares: /v1/responses max output tokens=8: status='completed' incomplete details=None output tokens=8 /v1/chat/completions max tokens=8: finish reason='length' http://127.0.0.1:18090: a reply cut off at max output tokens is reported as 'completed'. It exits 1 there, 0 when a cut-off reply is reported as incomplete , and 2 when it can't tell. A status field is only useful if something can make it say no. When you rely on one, find out whether the implementation has a code path that ever produces a value other than success, and test the failure case on purpose: set a limit too small for the answer and look at what comes back. A field that is hard-coded to completed passes every test that only checks successful replies. And when an API is implemented by translating to another one, check that the translation carries the failure signals back, not just the content. Here the information existed one layer down, in the chat endpoint's finish reason , and was dropped on the way out. This is the second place this month where llama-server's compatibility layers lose something on the way through: its Anthropic endpoint ignores which tool you ask for and stops inside the model's reasoning https://homelabpostmortem.com/2026/10/18/llama-server-anthropic-endpoint-ignores-tool-choice-and-stops-inside-reasoning/ . This post's fix is available as a tested, ready-to-run script in the toolkit.