{"slug": "streaming-tool-calls-without-losing-your-mind", "title": "Streaming tool calls without losing your mind", "summary": "A developer from Loop & Retry explains that streaming tool calls from LLM APIs like Anthropic's requires treating partial JSON fragments differently from streaming text, and recommends a tolerant parse for display versus a strict parse for execution. The post details the pitfalls of common approaches—exception-driven parsing, buffering until completion, and string matching—and offers a best-effort partial parser for progress display.", "body_md": "*Originally published on Loop & Retry — field notes on building LLM agents that survive production.*\n\nStreaming exists so the user isn't staring at a blank screen for three seconds. For plain text that's a solved problem: print tokens as they land, and a partial sentence is still readable. For a tool call it isn't solved, because the thing you're streaming is structured data, and `{\"path\": \"/etc/pas`\n\nis not a partial file path — it's invalid JSON that will raise on every parser you own until the closing brace arrives.\n\nMost of the pain I've seen with streaming tool calls comes from treating it like streaming text: assuming the partial payload is usable the moment it looks plausible. It isn't, and the three ways people cope with that all trade off differently.\n\nWith the Anthropic API, a streamed tool call doesn't show up as one JSON blob — it shows up as a `content_block_start`\n\n(type `tool_use`\n\n, with a `name`\n\nand an empty `input`\n\n), followed by a run of `content_block_delta`\n\nevents whose `delta.type`\n\nis `input_json_delta`\n\n, each carrying a fragment of the arguments as raw text in `partial_json`\n\n. You get the *characters* of the JSON object, not the object.\n\n```\nwith client.messages.stream(\n    model=\"claude-sonnet-4-5\",\n    max_tokens=1024,\n    tools=[my_tool_schema],\n    messages=[{\"role\": \"user\", \"content\": \"delete the staging cache\"}],\n) as stream:\n    raw = \"\"\n    for event in stream:\n        if event.type == \"content_block_delta\" and event.delta.type == \"input_json_delta\":\n            raw += event.delta.partial_json  # a fragment, e.g. '{\"targ' then 'et\": \"sta' ...\n        elif event.type == \"content_block_stop\":\n            args = json.loads(raw)  # only NOW is `raw` guaranteed parseable\n```\n\n`raw`\n\nafter three deltas might be `{\"target\": \"sta`\n\n. Feed that to `json.loads`\n\nand you get a `JSONDecodeError`\n\n, every time, until the block actually closes. That's not a bug in your handling — it's the correct behavior of a JSON parser given invalid JSON.\n\n**Parse on every delta and swallow the exception.** The most common first draft: accumulate `raw`\n\n, try `json.loads(raw)`\n\nafter every chunk, catch the exception, move on. It works, in the sense that it doesn't crash — but you're now running exception-driven control flow on the hot path of every tool call, dozens of times per call, and it hides the one exception you actually care about: a genuinely malformed final payload. When every intermediate state also throws, the log line that matters is indistinguishable from noise.\n\n**Buffer everything and parse once at the end.** Wait for `content_block_stop`\n\n, then parse. This is correct and it's what the code above does for execution — but if that's *all* you do, you've quietly opted back out of streaming for tool calls specifically, even while your text responses stream token-by-token. For a tool call with a large argument — a long file body, a multi-paragraph message draft — the user watches nothing happen for the entire generation, then sees the whole result appear at once. You kept the plumbing and lost the point.\n\n**Guess the shape with string matching.** Track open braces, count quotes, assume the value under construction is done when you see a comma at depth 1. This looks fine on the happy path and breaks on the first argument value that contains a brace, an escaped quote, or a comma of its own — which for anything resembling free text (a message body, a code snippet, a path with spaces) is a matter of when, not if.\n\nThe fix is to stop treating \"parse for display\" and \"parse for execution\" as the same operation. They have different tolerance for being wrong.\n\nFor **display**, you want a *tolerant* parse of an incomplete document — good enough to show a progress skeleton, never good enough to act on. A small completer that closes whatever's still open gets you there:\n\n``` python\ndef best_effort_partial(raw: str):\n    \"\"\"Auto-close open strings/brackets so partial JSON parses for DISPLAY ONLY.\n    Never feed this result to anything that executes.\"\"\"\n    fixed = raw\n    if fixed.count('\"') % 2 == 1:\n        fixed += '\"'\n    opens = {\"{\": \"}\", \"[\": \"]\"}\n    stack = [opens[c] for c in fixed if c in opens]\n    for c in reversed(fixed):\n        if c in \"}]\" and stack and stack[-1] == c:\n            stack.pop()\n    fixed += \"\".join(reversed(stack))\n    try:\n        return json.loads(fixed)\n    except json.JSONDecodeError:\n        return None  # still not closeable yet — show nothing this frame\n```\n\nRun that after every delta and you can render `{\"target\": \"staging cache\", \"confirm\": …`\n\nas an incrementally-filling form, the same way a streamed sentence fills in word by word. If it returns `None`\n\nsome frames, that's fine — skip the render, try again on the next delta.\n\nFor **execution**, the rule doesn't bend: only the fully accumulated, natively-parsed JSON from `content_block_stop`\n\nis ever passed to the function that actually deletes the cache or sends the email. `best_effort_partial`\n\nnever touches that path. The two parses can disagree for a few hundred milliseconds — the display guesses `\"confirm\": true`\n\nbefore the model has finished writing `\"confirm\": false`\n\n— and that's an acceptable, purely cosmetic lag, not a correctness bug, because nothing acted on the guess.\n\nSometimes the stream ends and `raw`\n\n*still* isn't valid JSON — not because you parsed too early, but because generation was cut off mid-argument (a `max_tokens`\n\nlimit hit while inside a tool call, or a dropped connection). Check for this explicitly rather than letting the final `json.loads`\n\nthrow a generic error you'll mis-file as a client bug:\n\n```\nif stream.get_final_message().stop_reason == \"max_tokens\" and raw and not is_complete(raw):\n    # Genuinely truncated. Do not attempt to execute a completed-looking guess —\n    # retry with a larger budget or ask the model to continue this specific call.\n    ...\n```\n\nTreat this the same way you'd treat any other incomplete-write case in [tool design generally](https://loopandretry.github.io/posts/designing-tools-an-llm-wont-misuse/?ref=devto): a truncated tool call is an error state to surface, not a partial success to salvage by feeding it through `best_effort_partial`\n\nand hoping.\n\n`json.loads`\n\na growing buffer and treat exceptions as normal.`content_block_stop`\n\n.`stop_reason`\n\nbefore you trust that the block actually closed.", "url": "https://wpnews.pro/news/streaming-tool-calls-without-losing-your-mind", "canonical_source": "https://dev.to/loopandretry/streaming-tool-calls-without-losing-your-mind-14lo", "published_at": "2026-08-19 05:26:45+00:00", "updated_at": "2026-08-19 05:42:49.788905+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-agents"], "entities": ["Anthropic", "Loop & Retry", "Claude Sonnet 4.5"], "alternates": {"html": "https://wpnews.pro/news/streaming-tool-calls-without-losing-your-mind", "markdown": "https://wpnews.pro/news/streaming-tool-calls-without-losing-your-mind.md", "text": "https://wpnews.pro/news/streaming-tool-calls-without-losing-your-mind.txt", "jsonld": "https://wpnews.pro/news/streaming-tool-calls-without-losing-your-mind.jsonld"}}