We have watched tokens stream in from an LLM before where they appeared one at a time, like the model was typing. If you used the Anthropic SDK's .stream() method, it just worked and you probably never saw what was on the wire.
This post will majorly focus on how a stream response works and how bugs are handled by SDK behind the hood.
#
- Why Streaming exists
To enable the streaming option we would need to make one change in the post request that is a single field "stream": true
and it will change the response experience.
Here are the pointers we take from the gif.
- The left side shows no streaming as the cursor blinks for 4 seconds then the whole response lands at once.
- The right side shows the streaming where the first word shows up in about 300 milliseconds. Words flow in as the model generates them.
Both the sides have same model, same prompt, same total time it is just the right side started giving response almost 4 seconds earlier. The 4 seconds wait time for a full reply feels broken. A streamed reply that finishes in four seconds feels fast. Streaming doesn't make the model faster it makes the wait disappear.
#
- What's on the wire
When you set stream: true
, the API stops sending a single JSON blob. It opens a persistent HTTP connection and pushes events down the line as the model generates them. The format is Server-Sent Events (SSE) a web standard. Any SSE debugger will read this stream.
Here's what comes through:
A few things to notice:
**The text lives in **delta.text
, nested inside content_block_delta
events. Those are the events we should look after.
stop_reason
moved. In post 1, we saw it right there in the response JSON. Here, it arrives at the very end inside a message_delta
event, just before message_stop
. If the loop bails out as soon as the text stops arriving we will never see it.
Chunks don't line up with tokens or words. You might get "Hello"
in one chunk and " world"
in the next, or both in one. The network decides where the cuts happens and it is not the model, not the API.
That's what the SDK has been hiding from you.
#
- Reading the stream
Streaming sounds complicated until we write the loop. It's just reading bytes, buffering them, splitting on blank lines, and parsing JSON.
Here's the flow:
- The response body is a
ReadableStream
which can be iterated with for await
.
- Each iteration gives us bytes which we can decode to string.
- Buffer the string. A chunk might end mid-message.
- Split the buffer on
\n\n
— that's the SSE message separator.
- Keep the last item in the buffer. It might be incomplete.
- For each complete message, find the
data:
line, strip the prefix, and parse the JSON.
- If the type is
content_block_delta
, print delta.text
.
- If it's
message_delta
, you've got your stop_reason
.
Here is the complete sample code you can use to try out:
The way it is working is that when the chunk ends in the middle of a message split("\n\n")
leaves an incomplete fragment as the last item. pop()
pulls it back into the buffer so the next chunk can finish it. Without this line, every split message crashes the parser.
data.delta.type === "text_delta"
this check matters because content_block_delta can carry other delta types too: input_json_delta
for tool arguments, `thinking_delta`
for extended thinking, `signature_delta`
for verification. For now we only care about text.
*You can find the full implementation *here on GitHub as well.
#
- Three bugs
The code above works on a good day. Here's what breaks it on a bad one.
The ghost stream. The issue is user navigates away with the stream keeps running and tokens keep arriving with nobody to read them. In order to fix this pass an AbortController
signal to fetch
and call abort()
when you're done.
The fix is an AbortController
:
The silent truncation. The API can send an error
event mid stream during overload. If the loop only handles content_block_delta
, the error gets skipped and you end up with a truncated response and no exception. The fix is to handle data.type === "error"
explicitly.
The split packet. A single SSE message can arrive in two TCP packets. Without buffering, JSON.parse
throws on the half. This is what buffer = messages.pop() ?? ""
fixes, it holds the incomplete piece until the next chunk completes it.
stop_reason, in a stream
In post 1, stop_reason
was right there in the response JSON. In a stream, it's the same four values end_turn
, max_tokens
, tool_use
, stop_sequence
but they arrive inside a message_delta
event near the end of the stream.
The same rule from post 1 applies: if you ignore stop_reason
, you'll ship a bug. A max_tokens
cutoff in a streamed response looks exactly like a normal end of stream. You won't know the model was cut off unless you read this event.
Three things to try before the next post
1. Run the streaming code. Then change "stream": true
to false
and run it again. Notice how long you wait before seeing anything. That gap is what your users feel.
2. Add console.error(chunk.length)
inside the for await
loop, before any parsing. Run the code and watch the numbers. You'll see chunks of wildly different sizes it could be 8 bytes here, 400 bytes there. The network decides, not the model. Tokens and chunks are not the same thing.
3. Start a stream, then disconnect your wifi mid response. Watch what happens. The loop hangs, then eventually throws but only if we have added error handling. This sets up the error handling post later in the series.
What's next
TinyAgent can now stream a response. Tokens land as they arrive. stop_reason
shows up at the end. It still has no memory though every call starts blank.
In the upcoming post series we will capture another important details. 😁
Happy Coding! 👩💻