Truncation and constrained generation
An agent asks a model to call a tool. The model starts writing the call as JSON. The token budget expires halfway through. In most engines the caller now holds a broken document: nothing to execute, nothing to parse, sometimes an HTTP 500. Runner finishes the document legally instead, and the agent loop continues.
The practical benefit first
On a local model, a broken tool call is not an error. It is the whole retry budget. #
What a budget is
Every request carries max_tokens, the most tokens the model may generate. Agents set it tight on purpose: local models are slow and context is scarce. A tool call is JSON, and JSON that stops early is not JSON.
What usually happens
The engine returns finish_reason: "length" with an empty call, or leaks the model's tool-call framing into the assistant's prose, or hands back a tool_calls object whose arguments fail to parse. The agent cannot proceed. It retries from scratch, burning the tokens, the time and the context window it was already short of.
What Runner does
Runner is already constraining the call to the tool's JSON schema as it decodes. When the budget expires, it emits the smallest schema-legal ending for the document that has been started, marks finish_reason: "length" so the caller knows the budget was hit, and returns a tool_calls entry whose arguments parse. The agent gets an executable call and a truthful signal, and continues.
"Arguments parse" is a bound the engine holds, not a hope. The streaming validator refuses to nest a document deeper than the parser can read back, so a close can never produce something the engine itself would reject. That was not always true: the fuzzer found a band where the two disagreed, and the fix was to make the validator refuse early rather than to widen the parser, because refusing is the direction that cannot ship a broken document.
Measured
Same box, same schema, same prompt, six engines, seven budgets. #
tool_choice: required, temperature 0, budgets 1, 2, 3, 5, 8, 16 and 64 tokens. The 64-token control proves the failure is truncation, not misconfiguration: every engine completes there. Below it, only Runner returns a parseable call.
How the five failures differ
They rank differently for a caller. An executable call beats a detectable empty or error response, which beats protocol rendered as prose, which beats a tool_calls object whose arguments silently fail to parse. vLLM and llama.cpp leak the framing into content; llama.cpp emits unparseable arguments at 16 tokens; Ollama hides the framing but returns HTTP 500 at 16; TensorRT-LLM and SGLang return an empty, callless message with HTTP 200. None of the five closes the document.
A gate, not a one-off
The property is an engine guarantee, grammar plus closer, not model quality. It holds identically for the random two-layer CI fixture and for granite-4.1-3b, which is why make test-truncation runs on any CPU with no GPU and no competitor, on every release. The agent-torture gate tests the same failure inside multi-turn agent loops. Recipe, raw responses and the substitute-model notes: docs/truncation-benchmark.md.
The technical detail
Forced-truncation recovery is not ordinary constrained decoding. #
Ordinary JSON-Schema constrained decoding restricts which token may come next, so the output is valid if it finishes. It says nothing about what happens when the output cannot finish. Runner's schema compiler produces a streaming validator that knows, at every byte, what the smallest legal completion of the document is: close the string the pattern still accepts, complete the number inside its declared range, supply the required properties, close the object. When the budget expires, that completion is emitted.
The same compiler backs --json, OpenAI response_format, Responses text.format and tool parameter schemas: objects, arrays, enums, const, type unions, numeric bounds, string lengths and anchored patterns, array counts, and the tool-discriminated union agent clients use. Unsupported or ambiguous constraints fail at compile time rather than being silently weakened.
./runner -m model.gguf --serve
curl localhost:8080/v1/chat/completions -d '{
"messages":[{"role":"user","content":"Book a table for two at 19:00."}],
"tools":[{"type":"function","function":{"name":"book_table",
"parameters":{"type":"object","properties":{"people":{"type":"integer"},
"time":{"type":"string"}},"required":["people","time"]}}}],
"tool_choice":"required","max_tokens":8}'
The response carries a tool_calls entry whose arguments parse, and the usual finish_reason: "length" so the caller knows the budget was hit.
Across a full quant ladder, constrained decoding held schema conformance and tool selection at 100% down to Q4_0 on two model families, while argument agreement with the Q8_0 reference decayed to 50%. The closer guarantees the call is well-formed and names the right tool; what goes in the arguments is still the model's. The ladders.
Because the validator knows the legal branches at a decision point, Runner can record choice_logprobs: each schema branch as a legal alternative, a posterior renormalized over them, and the probed probability mass. Routing and calibrated classification need that, and an included calibration tool turns labeled decisions into accuracy, Brier-score and ECE gates.