Your Kimi K3.1 Pro Responses Truncate Silently

You point an existing OpenAI-compatible client at Kimi K3.1 Pro and it works on the first try. Then a long generation comes back cut off mid-sentence, and nothing in your code complained. The request succeeded, the stream closed cleanly, and your user got half an answer.

By the end you can detect a truncated stream, size requests against the documented context limit, and choose between the Pro and Highspeed variants with evidence instead of vibes.

Every chat completion comes back with a finish_reason. It is stop when the model finished naturally and length when it hit your max_tokens cap. In a non-streaming call you would see it on the response object. In a streaming call it arrives on the final chunk, and most hand-written stream loops only read delta.content.

That is the failure. Your loop consumes text, the connection closes, and finish_reason == "length" lands in a chunk you never inspected. The model had more to say and your application stored a truncated answer as if it were complete.

The max_tokens parameter makes this worse. Leave it unset and some providers apply a conservative default. Set it too high and a long prompt pushes you over the context limit, which fails the whole request instead of truncating it.

This loop reads the text and records the termination reason from the last chunk.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.example.com/v1",
    api_key="sk-...",
)

stream = client.chat.completions.create(
    model="kimi-k3.1-pro",  # use the exact ID your provider lists
    messages=[{"role": "user", "content": user_prompt}],
    max_tokens=2048,
    stream=True,
)

finish_reason = None
for chunk in stream:
    choice = chunk.choices[0]
    if choice.delta.content:
        yield choice.delta.content
    if choice.finish_reason:
        finish_reason = choice.finish_reason

if finish_reason == "length":
    raise RuntimeError("Generation hit the token cap mid-output")

The check runs after the loop, not inside it, because the final chunk often carries finish_reason with empty content. Raising here turns a silent truncation into a visible failure your retry path can handle.

Model IDs differ between providers, so confirm the string against your provider's model list before shipping. A typo returns a 404 that looks like an auth problem.

Kimi ships K3.1 Pro in two variants, and they are separate model strings rather than a speed setting on one endpoint. Swapping the string changes latency and output, so treat the choice as a real decision.

Do not assume Highspeed returns the same text faster. Run your own parity check on a sample of real prompts before you migrate anything that feeds a downstream parser.

def parity_report(prompt, model_a, model_b):
    a = client.chat.completions.create(
        model=model_a, messages=prompt, temperature=0
    ).choices[0].message.content
    b = client.chat.completions.create(
        model=model_b, messages=prompt, temperature=0
    ).choices[0].message.content
    return {"identical": a == b, "a": a, "b": b}

Pin the model string in one constant. If a variant change silently alters output, you want the diff to be a deliberate commit, not a config drift nobody noticed.

Truncation and request failure have opposite causes. You fail when the prompt plus max_tokens exceeds the context limit, and you truncate when max_tokens is smaller than the answer needs. Budget explicitly against the limit your provider documents.

def budgeted_max_tokens(prompt_tokens, context_limit, cap=4096):
    remaining = context_limit - prompt_tokens
    if remaining <= 0:
        raise ValueError(f"Prompt exceeds the context limit by {-remaining} tokens")
    return min(cap, remaining)

Count the prompt with the same tokenizer the provider uses, not a character heuristic. A rough estimate is fine for dashboards, but the request that crosses the line is the one you measured badly.

A retry loop that re-sends a request containing tool calls can fire a side effect twice. Rate-limit errors are safe to retry; a half-executed tool call is not.

import time

def call_with_retry(fn, attempts=4, base=1.0):
    for attempt in range(attempts):
        try:
            return fn()
        except Exception as exc:
            if attempt == attempts - 1 or "429" not in str(exc):
                raise
        time.sleep(base * 2 ** attempt)

This retries only on the rate-limit error and re-raises everything else. Give tool calls their own idempotency key so a retry is a duplicate detection problem rather than a double-charge problem.

finish_reason on the final chunk, so check it after your loop ends.finish_reason == "length" means the model hit your cap, not that it finished; treat it as an error.max_tokens from the documented context limit instead of copying a constant. Kimi K3.1 Pro and K3.1 Pro Highspeed announces the two variants. I added the stream-termination checks, token budgeting, and retry handling that the announcement leaves out.

These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.

USDT, USDC or USDD · TRC-20 (Tron)

TFTNsfyomKrnUutRjBTGVULp19ByW29KbY