{"slug": "llm-streaming-fallback-after-the-first-token-the-client-owns-the-failure", "title": "LLM streaming fallback: after the first token, the client owns the failure", "summary": "A developer's analysis of LLM streaming fallback argues that once a gateway has sent the first token to a client, the client—not the gateway—should own the failure, defaulting to an explicit incomplete state with retries run by the application. The writeup compares three gateways (Worksi, DeltaLLM and Continuum Router) and finds their documented fallback boundaries differ—first byte, first downstream frame, and first successful provider handshake—so 'no fallback after the first token' and 'no fallback after the first byte' describe different systems. It recommends treating the boundary as a written contract and testing the gateway against it, since documentation does not reveal what a client actually receives.", "body_md": "A model gateway can switch providers cleanly until it has sent something to your client. After that point, every option changes an answer someone has already started reading. Gateways document that point in different places, and \"first byte\", \"first frame\" and \"first token\" are not the same moment in a stream.\n\nOur position: for agent workloads, the default after any content reaches the client should be an explicit incomplete state, with any retry run by the application. Gateway-side continuation is a reasonable choice for prose and a poor one for tool calls. Whichever you pick, write it down as a contract and test the gateway against it, because documentation does not tell you what your client actually receives.\n\nA Hacker News commenter asked: \"Do you buffer or stream and then switch? This impacts correctness, especially for partial outputs or tool calls.\" ([comment](https://news.ycombinator.com/item?id=50268751)). One person's checklist, but the right question.\n\n`200` with `text/event-stream` goes out, the HTTP status is fixed. Later failures travel inside the body or as a dropped connection. In `overloaded_error` can arrive as an `event: error` that \"would normally correspond to an HTTP 529 in a non-streaming context\".`message_start`, \"a Message object with empty content\", and `ping` events can appear anywhere.`text_delta`, or an `input_json_delta` holding a fragment of tool arguments.`message_delta` usage counts are cumulative.\nSo \"no fallback after the first token\" and \"no fallback after the first byte\" describe different gateways. Three implementations, as documented on 4 November 2026 (not measured for this article):\n\n| Implementation | Documented boundary | After the boundary | \n|---|---|---|\n| Worksi gateway | Up to two retries with backoff on connection errors, 429 and 5xx, then your fallback models. \"Never after the first streamed byte.\" | The failure reaches the client. | \n| [DeltaLLM](https://docs.deltallm.io/en/stable/features/routing/) | Fallback \"only before the first downstream response frame\". Role and metadata events wait in a bounded pre-commit buffer. | No new provider attempt. A classified stop after partial output completes as `content_filter` or`length` . Other broken streams close without`[DONE]` (OpenAI-compatible) or get one sanitized`event: error` frame (Anthropic Messages). | \n| [Continuum Router](https://docs.continuum.lablup.ai/en/architecture/model-fallback/) | Before the stream begins, an eligible failure hops to the next permitted chain target. On the OpenAI-compatible relay, `fallback.mid_stream_enabled` (default`true` ) buffers the SSE response so it can recover mid-stream. On`/anthropic/v1/messages` and`/v1/responses` , the boundary is the first successful provider handshake. | OpenAI-compatible relay: `streaming.mid_stream_fallback.enabled: true` continues from accumulated output when enough exists;`false` restarts on the fallback model. Mid-stream hops skip native-protocol and Unix-socket targets. Messages and Responses: no mid-stream recovery. With`mid_stream_enabled: false` , a failure after output begins is a stream error. | \n\nDeltaLLM documents holding role events back, so a provider that dies right after one can still be retried there. A gateway whose boundary is the first byte may already have forwarded that event. The \"Headers, then nothing\" test below checks exactly this.\n\nThe options are the same at every point: propagate an incomplete or error response, restart at the application level, or continue from what was already generated. What changes is the cost. The table assumes your application executes tools itself, so a failed model call has run no tool.\n\n| Failure point | What the client has | Propagate incomplete | Application restart | Continuation | Default we would choose | \n|---|---|---|---|---|---|\n| Before headers | Nothing | Only if the chain is exhausted | No delivered text to duplicate | Not applicable | Let the gateway fall back, subject to the checks below | \n| After headers, before content | A `200` , maybe metadata or pings | Acceptable | No delivered text to duplicate | Not applicable | Retry the failed model call | \n| After partial text | Text a person may have read | Honest; the user sees a cut-off answer and a retry option | The UI must replace, not append; the new answer may contradict the half already read | One answer from two generations, possibly two models | Propagate incomplete; restart on request | \n| During tool-call output | A fragment of JSON arguments | Required | Discard the partial block, regenerate the turn | A second generation completing arguments the first one chose | Never execute; regenerate the turn | \n| After a tool ran | A real side effect | Required for the failed call | Resend the stored tool result, not the user message | Same risks as above | Checkpoint at turn boundaries | \n\n\"No delivered text to duplicate\" is the whole claim in those cells. It does not mean nothing happened upstream. A missing response does not prove a remote operation did not run, which is why [Stripe treats a 500 as indeterminate](https://docs.stripe.com/error-low-level) and asks you to reconcile before retrying with a new key. If your provider runs tools server-side, or earlier steps already wrote somewhere, restart only the failed model call, keep earlier tool results and reconcile anything whose outcome you do not know.\n\nTwo Continuum settings change what a client receives on the relay: the continuation-or-restart switch above, and `same_model_first`, which tries another backend serving the same model before advancing the chain. That narrows the model-mismatch problem. The docs state one limit plainly: \"Mid-stream recoveries cannot carry X-Fallback-* headers because the SSE response is already committed.\" Headers will not tell your client a recovery happened. The router's attempt records, linked under one request id, are the operator's evidence.\n\nSignaling inside the body is possible: Anthropic's streaming docs describe server-side fallback responses where a `content_block_start` and `content_block_stop` pair with no deltas marks each model boundary. Check in a recorded transcript whether your gateway's continuation leaves any such mark. If you need to know which model wrote which sentence and cannot see it, keep continuation off.\n\nOne client failure mode: a connection closes, the accumulated text looks finished, and the code stores it as a complete answer. Treat completeness as an explicit signal. A sketch for Anthropic Messages events, already parsed from SSE:\n\n``` python\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass Turn:\n    text: list[str] = field(default_factory=list)\n    tool_json: dict[int, list[str]] = field(default_factory=dict)\n    stop_reason: str | None = None\n    complete: bool = False\n    error: dict | None = None\n\ndef consume(events, turn: Turn) -> Turn:\n    for ev in events:\n        t = ev[\"type\"]\n        if t == \"content_block_delta\":\n            d = ev[\"delta\"]\n            if d[\"type\"] == \"text_delta\":\n                turn.text.append(d[\"text\"])\n            elif d[\"type\"] == \"input_json_delta\":\n                turn.tool_json.setdefault(ev[\"index\"], []).append(d[\"partial_json\"])\n        elif t == \"message_delta\":\n            turn.stop_reason = ev[\"delta\"].get(\"stop_reason\")\n        elif t == \"message_stop\":\n            turn.complete = True\n        elif t == \"error\":\n            turn.error = ev[\"error\"]\n            break\n        # ping and unknown event types are skipped on purpose:\n        # Anthropic says new event types may be added\n    return turn\n```\n\nThe rules around it matter more than the code:\n\n`message_stop`. Chat Completions streams end with `data: [DONE]`. The OpenAI Responses API uses typed events: `response.completed` for a finished response and `error` for a failure. A Responses stream that ends without `content_filter` finish or an Anthropic `refusal` stop (all named in DeltaLLM's docs) arrive with a normal terminal event. Check the stop reason.\nThe tool-call id is the obvious idempotency key and a risky one. A regenerated turn is a new response, and nothing guarantees its tool call keeps the old id. If the first attempt created a ticket and the regenerated call carries a fresh id, a key built on that id lets both through.\n\nHashing the tool name and arguments does not fix this. The [Amazon Builders' Library](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/) explains why: two identical requests can be two intended actions, such as launching two identical instances. A regenerated call with slightly different arguments also hashes differently and slips past.\n\nHave your executor create and persist a logical operation id before dispatch, one per intended action: for example run id, step index and the call's ordinal within that step. Store it with the arguments, the original dispatch time and a retry deadline taken from the downstream API's idempotency window. Reuse that id, with the same parameters, on retries inside that window. Two intentional identical actions get two ids.\n\nThe deadline matters because keys expire. [Stripe's API v1 guidance](https://docs.stripe.com/error-low-level) says it keeps keys for at least 24 hours and can remove them after that, and that reusing a removed key can execute a new request. It limits retries of an unresolved operation to the first 24 hours; after that, it says to retry with neither the old key nor a new one, and to reconcile the outcome instead. Other APIs set their own windows, so read each contract. Past the deadline, stop automatic retries and reconcile before anything repeats. If the parameters must change, Stripe says to first establish that the original operation did not succeed and is not still running.\n\nThen checkpoint at turn boundaries. When the model call after a tool fails, resend the stored tool result in a new call; do not restart the agent loop from the user message. If you must regenerate a turn that already acted, match the new calls to logged operations by step and ordinal, and reconcile any mismatch with the downstream system before running anything.\n\nThis works only if the downstream system honors the key. For a tool whose API takes none, record intent before the call and check it before any retry. A window remains between the call succeeding and your record of it; write that into the runbook.\n\nA gateway authorization decision is not an outage. A model allowlist refusal, a budget or quota stop, or a pending approval should end the request, not send it down the chain. A gateway that treats those like a 5xx can route around the restriction: the fallback attempt is not bound by the decision that refused the first one, unless the policy and approval checks stay authoritative for every hop. Never use fallback to get past a refusal. Your own rate limit and a provider's 429 can share a status code and need opposite handling, so test which one your gateway retries.\n\nProvider content-policy refusals are a separate case and a deliberate choice. DeltaLLM documents `content_policy_fallbacks` for \"when a provider rejects the content for policy reasons\". Continuum's response-quality fallback is a different mechanism from streaming fallback: it applies only to non-streaming `/v1/chat/completions`, and its docs exclude streaming responses and the Messages and Responses ingress paths. In `enforce` mode it can resend a refused request to the next model, which its docs call an operational decision with safety implications; the default `shadow` mode only judges and logs. If you enable either, the target must still be a model the caller may use, under your mandatory guardrails.\n\nContinuum re-checks the key's `allowed_models` for default-chain and quality hops, but explicit chains \"are not re-gated per hop\", and its docs advise mirroring mandatory guardrail route rules across chained models.\n\nCapability mismatches are quieter. Continuum fits output-token budgets to the target and returns a 503 incompatibility error when no target fits. DeltaLLM treats unusable `200` answers, such as non-JSON tool arguments or a tool name the request did not offer, as failures that start retries and fallbacks. Those checks catch malformed output, not a fallback model that makes valid tool calls with worse judgment. That is your evaluation, per model in the chain.\n\nPut a fault-injecting proxy or a provider stub between the gateway and the model, record the raw SSE stream at the client, and point tools at a sandbox that counts invocations. The matrix lists checks, not results: run it per endpoint you use and per provider in the chain.\n\n| Test | Inject | Client should see (your contract) | Verify about effects | Evidence to capture | \n|---|---|---|---|---|\n| Pre-stream connection failure | Primary refuses the connection or returns 503 before headers | A complete answer from the fallback, or a typed error once the chain is exhausted | Exactly one complete generation reached the client | Response `model` field, request or trace id, documented fallback headers | \n| Headers, then nothing | Stub sends `200` and a metadata event, then closes | Recovered answer or an incomplete marker; record which | Nothing stored as final | Raw transcript with event types and timestamps | \n| Mid-stream disconnect | Close after N text deltas | Incomplete marker | Not cached or persisted as final; a restart replaces the text | Transcript, success signal present or absent, repeated text after recovery | \n| Partial tool call | Close in the middle of `input_json_delta` | Incomplete turn, no tool call | Sandbox counter at 0 | Executor log, fragment length (not contents) | \n| Side effect, then failure | Let the tool run, fail the next model call | Recovery from the stored tool result | Sandbox counter at 1 after recovery | Operation id hits, downstream record count | \n| Gateway policy refusal | Request a disallowed model, exceed a budget or your own rate limit | A typed refusal, never a fallback answer | No request reached any provider | Status and error type, provider-side request log | \n| Provider content-policy refusal | Stub returns a classified content-policy error or refusal stop | The refusal, or a fallback you configured on purpose | Any fallback target is permitted for the caller and passed your guardrails | Final model, guardrail result, fallback reason | \n| Fallback capability mismatch | Primary needs tools, JSON output or a long context; fallback lacks one | A loud failure, or a fallback that supports it | No parameters silently dropped | Final model, the request as sent to the fallback, validation result | \n\nEach strategy needs different retained data, so decide them separately.\n\nDiagnostic transcripts are optional, and a raw SSE transcript is user content. Keep full transcripts in a test environment that runs synthetic prompts, and the metadata above in production. Pattern-based redaction is not a substitute for that split, because it misses names and free text.\n\nThe Worksi gateway makes up to two retries with backoff on connection errors, 429 and 5xx, then tries the fallback models you chose, and never retries or falls back after the first streamed byte. After that byte, a failure reaches your client. Keep incomplete-state handling, downstream-tool idempotency and checkpoint recovery in your application. Worksi's tool allowlists and approvals decide whether a tool call may run; whether an action that already ran may run again stays with your executor.\n\nFor the incomplete-state evidence above, the gateway's record and your client's record need to line up. A gateway request without a trace header starts its own trace, so send one per run and keep the same id in your client-side transcript. Gateway refusals come back as typed errors, such as `403 model_not_allowed` or `402 budget_exceeded`, and are recorded as blocked spans: gateway-side evidence for the policy-refusal test. Metadata-only capture keeps that record without prompt or output text. If you need mid-stream continuation, Worksi's documented behavior does not provide it. [Read the gateway's retry and fallback rules](https://worksi.ai/product/gateway).\n\n*Originally published at [blog.worksi.ai](https://blog.worksi.ai/llm-streaming-fallback-after-first-token). Written by Worksi's AI marketing agents and checked by a separate AI fact-checking agent.*", "url": "https://wpnews.pro/news/llm-streaming-fallback-after-the-first-token-the-client-owns-the-failure", "canonical_source": "https://dev.to/worksi_25025494650/llm-streaming-fallback-after-the-first-token-the-client-owns-the-failure-ici", "published_at": "2026-11-05 14:41:35+00:00", "updated_at": "2026-11-05 14:49:30.754107+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-agents", "developer-tools"], "entities": ["Worksi", "DeltaLLM", "Continuum Router", "Lablup", "Hacker News", "Anthropic"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-streaming-fallback-after-the-first-token-the-client-owns-the-failure", "markdown": "https://wpnews.pro/news/llm-streaming-fallback-after-the-first-token-the-client-owns-the-failure.md", "text": "https://wpnews.pro/news/llm-streaming-fallback-after-the-first-token-the-client-owns-the-failure.txt", "jsonld": "https://wpnews.pro/news/llm-streaming-fallback-after-the-first-token-the-client-owns-the-failure.jsonld"}}