LLM streaming fallback: after the first token, the client owns the failure

A developer's analysis of LLM streaming fallback argues that once a gateway has sent the first token to a client, the client—not the gateway—should own the failure, defaulting to an explicit incomplete state with retries run by the application. The writeup compares three gateways (Worksi, DeltaLLM and Continuum Router) and finds their documented fallback boundaries differ—first byte, first downstream frame, and first successful provider handshake—so 'no fallback after the first token' and 'no fallback after the first byte' describe different systems. It recommends treating the boundary as a written contract and testing the gateway against it, since documentation does not reveal what a client actually receives.

A model gateway can switch providers cleanly until it has sent something to your client. After that point, every option changes an answer someone has already started reading. Gateways document that point in different places, and "first byte", "first frame" and "first token" are not the same moment in a stream. Our position: for agent workloads, the default after any content reaches the client should be an explicit incomplete state, with any retry run by the application. Gateway-side continuation is a reasonable choice for prose and a poor one for tool calls. Whichever you pick, write it down as a contract and test the gateway against it, because documentation does not tell you what your client actually receives. A Hacker News commenter asked: "Do you buffer or stream and then switch? This impacts correctness, especially for partial outputs or tool calls." comment https://news.ycombinator.com/item?id=50268751 . One person's checklist, but the right question. 200 with text/event-stream goes out, the HTTP status is fixed. Later failures travel inside the body or as a dropped connection. In overloaded error can arrive as an event: error that "would normally correspond to an HTTP 529 in a non-streaming context". message start , "a Message object with empty content", and ping events can appear anywhere. text delta , or an input json delta holding a fragment of tool arguments. message delta usage counts are cumulative. So "no fallback after the first token" and "no fallback after the first byte" describe different gateways. Three implementations, as documented on 4 November 2026 not measured for this article : | Implementation | Documented boundary | After the boundary | |---|---|---| | Worksi gateway | Up to two retries with backoff on connection errors, 429 and 5xx, then your fallback models. "Never after the first streamed byte." | The failure reaches the client. | | DeltaLLM https://docs.deltallm.io/en/stable/features/routing/ | Fallback "only before the first downstream response frame". Role and metadata events wait in a bounded pre-commit buffer. | No new provider attempt. A classified stop after partial output completes as content filter or length . Other broken streams close without DONE OpenAI-compatible or get one sanitized event: error frame Anthropic Messages . | | Continuum Router https://docs.continuum.lablup.ai/en/architecture/model-fallback/ | Before the stream begins, an eligible failure hops to the next permitted chain target. On the OpenAI-compatible relay, fallback.mid stream enabled default true buffers the SSE response so it can recover mid-stream. On /anthropic/v1/messages and /v1/responses , the boundary is the first successful provider handshake. | OpenAI-compatible relay: streaming.mid stream fallback.enabled: true continues from accumulated output when enough exists; false restarts on the fallback model. Mid-stream hops skip native-protocol and Unix-socket targets. Messages and Responses: no mid-stream recovery. With mid stream enabled: false , a failure after output begins is a stream error. | DeltaLLM documents holding role events back, so a provider that dies right after one can still be retried there. A gateway whose boundary is the first byte may already have forwarded that event. The "Headers, then nothing" test below checks exactly this. The options are the same at every point: propagate an incomplete or error response, restart at the application level, or continue from what was already generated. What changes is the cost. The table assumes your application executes tools itself, so a failed model call has run no tool. | Failure point | What the client has | Propagate incomplete | Application restart | Continuation | Default we would choose | |---|---|---|---|---|---| | Before headers | Nothing | Only if the chain is exhausted | No delivered text to duplicate | Not applicable | Let the gateway fall back, subject to the checks below | | After headers, before content | A 200 , maybe metadata or pings | Acceptable | No delivered text to duplicate | Not applicable | Retry the failed model call | | After partial text | Text a person may have read | Honest; the user sees a cut-off answer and a retry option | The UI must replace, not append; the new answer may contradict the half already read | One answer from two generations, possibly two models | Propagate incomplete; restart on request | | During tool-call output | A fragment of JSON arguments | Required | Discard the partial block, regenerate the turn | A second generation completing arguments the first one chose | Never execute; regenerate the turn | | After a tool ran | A real side effect | Required for the failed call | Resend the stored tool result, not the user message | Same risks as above | Checkpoint at turn boundaries | "No delivered text to duplicate" is the whole claim in those cells. It does not mean nothing happened upstream. A missing response does not prove a remote operation did not run, which is why Stripe treats a 500 as indeterminate https://docs.stripe.com/error-low-level and asks you to reconcile before retrying with a new key. If your provider runs tools server-side, or earlier steps already wrote somewhere, restart only the failed model call, keep earlier tool results and reconcile anything whose outcome you do not know. Two Continuum settings change what a client receives on the relay: the continuation-or-restart switch above, and same model first , which tries another backend serving the same model before advancing the chain. That narrows the model-mismatch problem. The docs state one limit plainly: "Mid-stream recoveries cannot carry X-Fallback- headers because the SSE response is already committed." Headers will not tell your client a recovery happened. The router's attempt records, linked under one request id, are the operator's evidence. Signaling inside the body is possible: Anthropic's streaming docs describe server-side fallback responses where a content block start and content block stop pair with no deltas marks each model boundary. Check in a recorded transcript whether your gateway's continuation leaves any such mark. If you need to know which model wrote which sentence and cannot see it, keep continuation off. One client failure mode: a connection closes, the accumulated text looks finished, and the code stores it as a complete answer. Treat completeness as an explicit signal. A sketch for Anthropic Messages events, already parsed from SSE: python from dataclasses import dataclass, field @dataclass class Turn: text: list str = field default factory=list tool json: dict int, list str = field default factory=dict stop reason: str | None = None complete: bool = False error: dict | None = None def consume events, turn: Turn - Turn: for ev in events: t = ev "type" if t == "content block delta": d = ev "delta" if d "type" == "text delta": turn.text.append d "text" elif d "type" == "input json delta": turn.tool json.setdefault ev "index" , .append d "partial json" elif t == "message delta": turn.stop reason = ev "delta" .get "stop reason" elif t == "message stop": turn.complete = True elif t == "error": turn.error = ev "error" break ping and unknown event types are skipped on purpose: Anthropic says new event types may be added return turn The rules around it matter more than the code: message stop . Chat Completions streams end with data: DONE . The OpenAI Responses API uses typed events: response.completed for a finished response and error for a failure. A Responses stream that ends without content filter finish or an Anthropic refusal stop all named in DeltaLLM's docs arrive with a normal terminal event. Check the stop reason. The tool-call id is the obvious idempotency key and a risky one. A regenerated turn is a new response, and nothing guarantees its tool call keeps the old id. If the first attempt created a ticket and the regenerated call carries a fresh id, a key built on that id lets both through. Hashing the tool name and arguments does not fix this. The Amazon Builders' Library https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/ explains why: two identical requests can be two intended actions, such as launching two identical instances. A regenerated call with slightly different arguments also hashes differently and slips past. Have your executor create and persist a logical operation id before dispatch, one per intended action: for example run id, step index and the call's ordinal within that step. Store it with the arguments, the original dispatch time and a retry deadline taken from the downstream API's idempotency window. Reuse that id, with the same parameters, on retries inside that window. Two intentional identical actions get two ids. The deadline matters because keys expire. Stripe's API v1 guidance https://docs.stripe.com/error-low-level says it keeps keys for at least 24 hours and can remove them after that, and that reusing a removed key can execute a new request. It limits retries of an unresolved operation to the first 24 hours; after that, it says to retry with neither the old key nor a new one, and to reconcile the outcome instead. Other APIs set their own windows, so read each contract. Past the deadline, stop automatic retries and reconcile before anything repeats. If the parameters must change, Stripe says to first establish that the original operation did not succeed and is not still running. Then checkpoint at turn boundaries. When the model call after a tool fails, resend the stored tool result in a new call; do not restart the agent loop from the user message. If you must regenerate a turn that already acted, match the new calls to logged operations by step and ordinal, and reconcile any mismatch with the downstream system before running anything. This works only if the downstream system honors the key. For a tool whose API takes none, record intent before the call and check it before any retry. A window remains between the call succeeding and your record of it; write that into the runbook. A gateway authorization decision is not an outage. A model allowlist refusal, a budget or quota stop, or a pending approval should end the request, not send it down the chain. A gateway that treats those like a 5xx can route around the restriction: the fallback attempt is not bound by the decision that refused the first one, unless the policy and approval checks stay authoritative for every hop. Never use fallback to get past a refusal. Your own rate limit and a provider's 429 can share a status code and need opposite handling, so test which one your gateway retries. Provider content-policy refusals are a separate case and a deliberate choice. DeltaLLM documents content policy fallbacks for "when a provider rejects the content for policy reasons". Continuum's response-quality fallback is a different mechanism from streaming fallback: it applies only to non-streaming /v1/chat/completions , and its docs exclude streaming responses and the Messages and Responses ingress paths. In enforce mode it can resend a refused request to the next model, which its docs call an operational decision with safety implications; the default shadow mode only judges and logs. If you enable either, the target must still be a model the caller may use, under your mandatory guardrails. Continuum re-checks the key's allowed models for default-chain and quality hops, but explicit chains "are not re-gated per hop", and its docs advise mirroring mandatory guardrail route rules across chained models. Capability mismatches are quieter. Continuum fits output-token budgets to the target and returns a 503 incompatibility error when no target fits. DeltaLLM treats unusable 200 answers, such as non-JSON tool arguments or a tool name the request did not offer, as failures that start retries and fallbacks. Those checks catch malformed output, not a fallback model that makes valid tool calls with worse judgment. That is your evaluation, per model in the chain. Put a fault-injecting proxy or a provider stub between the gateway and the model, record the raw SSE stream at the client, and point tools at a sandbox that counts invocations. The matrix lists checks, not results: run it per endpoint you use and per provider in the chain. | Test | Inject | Client should see your contract | Verify about effects | Evidence to capture | |---|---|---|---|---| | Pre-stream connection failure | Primary refuses the connection or returns 503 before headers | A complete answer from the fallback, or a typed error once the chain is exhausted | Exactly one complete generation reached the client | Response model field, request or trace id, documented fallback headers | | Headers, then nothing | Stub sends 200 and a metadata event, then closes | Recovered answer or an incomplete marker; record which | Nothing stored as final | Raw transcript with event types and timestamps | | Mid-stream disconnect | Close after N text deltas | Incomplete marker | Not cached or persisted as final; a restart replaces the text | Transcript, success signal present or absent, repeated text after recovery | | Partial tool call | Close in the middle of input json delta | Incomplete turn, no tool call | Sandbox counter at 0 | Executor log, fragment length not contents | | Side effect, then failure | Let the tool run, fail the next model call | Recovery from the stored tool result | Sandbox counter at 1 after recovery | Operation id hits, downstream record count | | Gateway policy refusal | Request a disallowed model, exceed a budget or your own rate limit | A typed refusal, never a fallback answer | No request reached any provider | Status and error type, provider-side request log | | Provider content-policy refusal | Stub returns a classified content-policy error or refusal stop | The refusal, or a fallback you configured on purpose | Any fallback target is permitted for the caller and passed your guardrails | Final model, guardrail result, fallback reason | | Fallback capability mismatch | Primary needs tools, JSON output or a long context; fallback lacks one | A loud failure, or a fallback that supports it | No parameters silently dropped | Final model, the request as sent to the fallback, validation result | Each strategy needs different retained data, so decide them separately. Diagnostic transcripts are optional, and a raw SSE transcript is user content. Keep full transcripts in a test environment that runs synthetic prompts, and the metadata above in production. Pattern-based redaction is not a substitute for that split, because it misses names and free text. The Worksi gateway makes up to two retries with backoff on connection errors, 429 and 5xx, then tries the fallback models you chose, and never retries or falls back after the first streamed byte. After that byte, a failure reaches your client. Keep incomplete-state handling, downstream-tool idempotency and checkpoint recovery in your application. Worksi's tool allowlists and approvals decide whether a tool call may run; whether an action that already ran may run again stays with your executor. For the incomplete-state evidence above, the gateway's record and your client's record need to line up. A gateway request without a trace header starts its own trace, so send one per run and keep the same id in your client-side transcript. Gateway refusals come back as typed errors, such as 403 model not allowed or 402 budget exceeded , and are recorded as blocked spans: gateway-side evidence for the policy-refusal test. Metadata-only capture keeps that record without prompt or output text. If you need mid-stream continuation, Worksi's documented behavior does not provide it. Read the gateway's retry and fallback rules https://worksi.ai/product/gateway . Originally published at blog.worksi.ai https://blog.worksi.ai/llm-streaming-fallback-after-first-token . Written by Worksi's AI marketing agents and checked by a separate AI fact-checking agent.