Streaming tool calls without losing your mind

A developer from Loop & Retry explains that streaming tool calls from LLM APIs like Anthropic's requires treating partial JSON fragments differently from streaming text, and recommends a tolerant parse for display versus a strict parse for execution. The post details the pitfalls of common approaches—exception-driven parsing, buffering until completion, and string matching—and offers a best-effort partial parser for progress display.

Originally published on Loop & Retry — field notes on building LLM agents that survive production. Streaming exists so the user isn't staring at a blank screen for three seconds. For plain text that's a solved problem: print tokens as they land, and a partial sentence is still readable. For a tool call it isn't solved, because the thing you're streaming is structured data, and {"path": "/etc/pas is not a partial file path — it's invalid JSON that will raise on every parser you own until the closing brace arrives. Most of the pain I've seen with streaming tool calls comes from treating it like streaming text: assuming the partial payload is usable the moment it looks plausible. It isn't, and the three ways people cope with that all trade off differently. With the Anthropic API, a streamed tool call doesn't show up as one JSON blob — it shows up as a content block start type tool use , with a name and an empty input , followed by a run of content block delta events whose delta.type is input json delta , each carrying a fragment of the arguments as raw text in partial json . You get the characters of the JSON object, not the object. with client.messages.stream model="claude-sonnet-4-5", max tokens=1024, tools= my tool schema , messages= {"role": "user", "content": "delete the staging cache"} , as stream: raw = "" for event in stream: if event.type == "content block delta" and event.delta.type == "input json delta": raw += event.delta.partial json a fragment, e.g. '{"targ' then 'et": "sta' ... elif event.type == "content block stop": args = json.loads raw only NOW is raw guaranteed parseable raw after three deltas might be {"target": "sta . Feed that to json.loads and you get a JSONDecodeError , every time, until the block actually closes. That's not a bug in your handling — it's the correct behavior of a JSON parser given invalid JSON. Parse on every delta and swallow the exception. The most common first draft: accumulate raw , try json.loads raw after every chunk, catch the exception, move on. It works, in the sense that it doesn't crash — but you're now running exception-driven control flow on the hot path of every tool call, dozens of times per call, and it hides the one exception you actually care about: a genuinely malformed final payload. When every intermediate state also throws, the log line that matters is indistinguishable from noise. Buffer everything and parse once at the end. Wait for content block stop , then parse. This is correct and it's what the code above does for execution — but if that's all you do, you've quietly opted back out of streaming for tool calls specifically, even while your text responses stream token-by-token. For a tool call with a large argument — a long file body, a multi-paragraph message draft — the user watches nothing happen for the entire generation, then sees the whole result appear at once. You kept the plumbing and lost the point. Guess the shape with string matching. Track open braces, count quotes, assume the value under construction is done when you see a comma at depth 1. This looks fine on the happy path and breaks on the first argument value that contains a brace, an escaped quote, or a comma of its own — which for anything resembling free text a message body, a code snippet, a path with spaces is a matter of when, not if. The fix is to stop treating "parse for display" and "parse for execution" as the same operation. They have different tolerance for being wrong. For display , you want a tolerant parse of an incomplete document — good enough to show a progress skeleton, never good enough to act on. A small completer that closes whatever's still open gets you there: python def best effort partial raw: str : """Auto-close open strings/brackets so partial JSON parses for DISPLAY ONLY. Never feed this result to anything that executes.""" fixed = raw if fixed.count '"' % 2 == 1: fixed += '"' opens = {"{": "}", " ": " "} stack = opens c for c in fixed if c in opens for c in reversed fixed : if c in "} " and stack and stack -1 == c: stack.pop fixed += "".join reversed stack try: return json.loads fixed except json.JSONDecodeError: return None still not closeable yet — show nothing this frame Run that after every delta and you can render {"target": "staging cache", "confirm": … as an incrementally-filling form, the same way a streamed sentence fills in word by word. If it returns None some frames, that's fine — skip the render, try again on the next delta. For execution , the rule doesn't bend: only the fully accumulated, natively-parsed JSON from content block stop is ever passed to the function that actually deletes the cache or sends the email. best effort partial never touches that path. The two parses can disagree for a few hundred milliseconds — the display guesses "confirm": true before the model has finished writing "confirm": false — and that's an acceptable, purely cosmetic lag, not a correctness bug, because nothing acted on the guess. Sometimes the stream ends and raw still isn't valid JSON — not because you parsed too early, but because generation was cut off mid-argument a max tokens limit hit while inside a tool call, or a dropped connection . Check for this explicitly rather than letting the final json.loads throw a generic error you'll mis-file as a client bug: if stream.get final message .stop reason == "max tokens" and raw and not is complete raw : Genuinely truncated. Do not attempt to execute a completed-looking guess — retry with a larger budget or ask the model to continue this specific call. ... Treat this the same way you'd treat any other incomplete-write case in tool design generally https://loopandretry.github.io/posts/designing-tools-an-llm-wont-misuse/?ref=devto : a truncated tool call is an error state to surface, not a partial success to salvage by feeding it through best effort partial and hoping. json.loads a growing buffer and treat exceptions as normal. content block stop . stop reason before you trust that the block actually closed.