Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Streaming exists so the user isn't staring at a blank screen for three seconds. For plain text that's a solved problem: print tokens as they land, and a partial sentence is still readable. For a tool call it isn't solved, because the thing you're streaming is structured data, and {"path": "/etc/pas
is not a partial file path — it's invalid JSON that will raise on every parser you own until the closing brace arrives.
Most of the pain I've seen with streaming tool calls comes from treating it like streaming text: assuming the partial payload is usable the moment it looks plausible. It isn't, and the three ways people cope with that all trade off differently.
With the Anthropic API, a streamed tool call doesn't show up as one JSON blob — it shows up as a content_block_start
(type tool_use
, with a name
and an empty input
), followed by a run of content_block_delta
events whose delta.type
is input_json_delta
, each carrying a fragment of the arguments as raw text in partial_json
. You get the characters of the JSON object, not the object.
with client.messages.stream(
model="claude-sonnet-4-5",
max_tokens=1024,
tools=[my_tool_schema],
messages=[{"role": "user", "content": "delete the staging cache"}],
) as stream:
raw = ""
for event in stream:
if event.type == "content_block_delta" and event.delta.type == "input_json_delta":
raw += event.delta.partial_json # a fragment, e.g. '{"targ' then 'et": "sta' ...
elif event.type == "content_block_stop":
args = json.loads(raw) # only NOW is `raw` guaranteed parseable
raw
after three deltas might be {"target": "sta
. Feed that to json.loads
and you get a JSONDecodeError
, every time, until the block actually closes. That's not a bug in your handling — it's the correct behavior of a JSON parser given invalid JSON.
Parse on every delta and swallow the exception. The most common first draft: accumulate raw
, try json.loads(raw)
after every chunk, catch the exception, move on. It works, in the sense that it doesn't crash — but you're now running exception-driven control flow on the hot path of every tool call, dozens of times per call, and it hides the one exception you actually care about: a genuinely malformed final payload. When every intermediate state also throws, the log line that matters is indistinguishable from noise.
Buffer everything and parse once at the end. Wait for content_block_stop
, then parse. This is correct and it's what the code above does for execution — but if that's all you do, you've quietly opted back out of streaming for tool calls specifically, even while your text responses stream token-by-token. For a tool call with a large argument — a long file body, a multi-paragraph message draft — the user watches nothing happen for the entire generation, then sees the whole result appear at once. You kept the plumbing and lost the point.
Guess the shape with string matching. Track open braces, count quotes, assume the value under construction is done when you see a comma at depth 1. This looks fine on the happy path and breaks on the first argument value that contains a brace, an escaped quote, or a comma of its own — which for anything resembling free text (a message body, a code snippet, a path with spaces) is a matter of when, not if.
The fix is to stop treating "parse for display" and "parse for execution" as the same operation. They have different tolerance for being wrong.
For display, you want a tolerant parse of an incomplete document — good enough to show a progress skeleton, never good enough to act on. A small completer that closes whatever's still open gets you there:
def best_effort_partial(raw: str):
"""Auto-close open strings/brackets so partial JSON parses for DISPLAY ONLY.
Never feed this result to anything that executes."""
fixed = raw
if fixed.count('"') % 2 == 1:
fixed += '"'
opens = {"{": "}", "[": "]"}
stack = [opens[c] for c in fixed if c in opens]
for c in reversed(fixed):
if c in "}]" and stack and stack[-1] == c:
stack.pop()
fixed += "".join(reversed(stack))
try:
return json.loads(fixed)
except json.JSONDecodeError:
return None # still not closeable yet — show nothing this frame
Run that after every delta and you can render {"target": "staging cache", "confirm": …
as an incrementally-filling form, the same way a streamed sentence fills in word by word. If it returns None
some frames, that's fine — skip the render, try again on the next delta.
For execution, the rule doesn't bend: only the fully accumulated, natively-parsed JSON from content_block_stop
is ever passed to the function that actually deletes the cache or sends the email. best_effort_partial
never touches that path. The two parses can disagree for a few hundred milliseconds — the display guesses "confirm": true
before the model has finished writing "confirm": false
— and that's an acceptable, purely cosmetic lag, not a correctness bug, because nothing acted on the guess.
Sometimes the stream ends and raw
still isn't valid JSON — not because you parsed too early, but because generation was cut off mid-argument (a max_tokens
limit hit while inside a tool call, or a dropped connection). Check for this explicitly rather than letting the final json.loads
throw a generic error you'll mis-file as a client bug:
if stream.get_final_message().stop_reason == "max_tokens" and raw and not is_complete(raw):
...
Treat this the same way you'd treat any other incomplete-write case in tool design generally: a truncated tool call is an error state to surface, not a partial success to salvage by feeding it through best_effort_partial
and hoping.
json.loads
a growing buffer and treat exceptions as normal.content_block_stop
.stop_reason
before you trust that the block actually closed.