Your Kimi K3.1 Pro Responses Truncate Silently

A developer documented a silent-truncation failure mode when pointing OpenAI-compatible clients at Kimi K3.1 Pro: streaming loops that only read delta.content miss the finish_reason field on the final chunk, so generations cut off at max_tokens are stored as complete answers. The writeup provides a Python loop that captures finish_reason and raises on "length", a budgeted_max_tokens helper that sizes requests against the documented context limit, and a parity check for comparing the Pro and Highspeed variants, which use separate model strings and are not guaranteed to produce identical output.

You point an existing OpenAI-compatible client at Kimi K3.1 Pro and it works on the first try. Then a long generation comes back cut off mid-sentence, and nothing in your code complained. The request succeeded, the stream closed cleanly, and your user got half an answer. By the end you can detect a truncated stream, size requests against the documented context limit, and choose between the Pro and Highspeed variants with evidence instead of vibes. Every chat completion comes back with a finish reason . It is stop when the model finished naturally and length when it hit your max tokens cap. In a non-streaming call you would see it on the response object. In a streaming call it arrives on the final chunk, and most hand-written stream loops only read delta.content . That is the failure. Your loop consumes text, the connection closes, and finish reason == "length" lands in a chunk you never inspected. The model had more to say and your application stored a truncated answer as if it were complete. The max tokens parameter makes this worse. Leave it unset and some providers apply a conservative default. Set it too high and a long prompt pushes you over the context limit, which fails the whole request instead of truncating it. This loop reads the text and records the termination reason from the last chunk. python from openai import OpenAI client = OpenAI base url="https://api.example.com/v1", api key="sk-...", stream = client.chat.completions.create model="kimi-k3.1-pro", use the exact ID your provider lists messages= {"role": "user", "content": user prompt} , max tokens=2048, stream=True, finish reason = None for chunk in stream: choice = chunk.choices 0 if choice.delta.content: yield choice.delta.content if choice.finish reason: finish reason = choice.finish reason if finish reason == "length": raise RuntimeError "Generation hit the token cap mid-output" The check runs after the loop, not inside it, because the final chunk often carries finish reason with empty content. Raising here turns a silent truncation into a visible failure your retry path can handle. Model IDs differ between providers, so confirm the string against your provider's model list before shipping. A typo returns a 404 that looks like an auth problem. Kimi ships K3.1 Pro in two variants, and they are separate model strings rather than a speed setting on one endpoint. Swapping the string changes latency and output, so treat the choice as a real decision. | Variant | Use it when | Tradeoff | |---|---|---| | K3.1 Pro | Correctness is the priority | Slower per request | | K3.1 Pro Highspeed | Latency and throughput dominate | Not guaranteed output-identical to Pro | Do not assume Highspeed returns the same text faster. Run your own parity check on a sample of real prompts before you migrate anything that feeds a downstream parser. python def parity report prompt, model a, model b : a = client.chat.completions.create model=model a, messages=prompt, temperature=0 .choices 0 .message.content b = client.chat.completions.create model=model b, messages=prompt, temperature=0 .choices 0 .message.content return {"identical": a == b, "a": a, "b": b} Pin the model string in one constant. If a variant change silently alters output, you want the diff to be a deliberate commit, not a config drift nobody noticed. Truncation and request failure have opposite causes. You fail when the prompt plus max tokens exceeds the context limit, and you truncate when max tokens is smaller than the answer needs. Budget explicitly against the limit your provider documents. python def budgeted max tokens prompt tokens, context limit, cap=4096 : remaining = context limit - prompt tokens if remaining <= 0: raise ValueError f"Prompt exceeds the context limit by {-remaining} tokens" return min cap, remaining Count the prompt with the same tokenizer the provider uses, not a character heuristic. A rough estimate is fine for dashboards, but the request that crosses the line is the one you measured badly. A retry loop that re-sends a request containing tool calls can fire a side effect twice. Rate-limit errors are safe to retry; a half-executed tool call is not. python import time def call with retry fn, attempts=4, base=1.0 : for attempt in range attempts : try: return fn except Exception as exc: if attempt == attempts - 1 or "429" not in str exc : raise time.sleep base 2 attempt This retries only on the rate-limit error and re-raises everything else. Give tool calls their own idempotency key so a retry is a duplicate detection problem rather than a double-charge problem. finish reason on the final chunk, so check it after your loop ends. finish reason == "length" means the model hit your cap, not that it finished; treat it as an error. max tokens from the documented context limit instead of copying a constant. Kimi K3.1 Pro and K3.1 Pro Highspeed https://www.kimi.ai/blog/kimi-k3-1-pro announces the two variants. I added the stream-termination checks, token budgeting, and retry handling that the announcement leaves out. These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming. USDT, USDC or USDD · TRC-20 Tron TFTNsfyomKrnUutRjBTGVULp19ByW29KbY