Migration Diary: The Streaming Parser Is What Actually Breaks at Cutover

A migration diary entry argues that streaming server-sent-event parsers, not JSON response bodies, are the layer that breaks during provider cutovers, producing truncated answers, half-assembled tool calls, and a finish_reason that never fires. The author recommends recording raw SSE bytes as fixtures for at least four shapes — a plain text answer, a multi-chunk tool call, a stop-sequence hit, and an error object returned after a 200 status line — and normalizing each provider's chunk shape into one event model that keys tool-call fragments by index rather than arrival order. The cutover should be gated on structural equality of those fixtures rather than a one-time spot check, because two endpoints claiming OpenAI compatibility can still disagree on where usage appears, whether [DONE] is sent, and how tool-call arguments are split across chunks.

Migration Diary: The Streaming Parser Is What Actually Breaks at Cutover Conclusion first: your JSON will diff clean on the first day of the migration, and the server-sent stream underneath it will still hand you truncated answers, half-assembled tool calls, and a finish reason that never fires. Transport is the layer that fails quietly, because nothing raises and nothin g looks wrong until an agent loop calls a function with {} as its arguments. This entry in the migration diary is about that layer: recording real streams as byte fixtures, reducing every provider's chunk shape into one event model, and gating the cutover on structural equality rather than a spot check you ran once in a notebook. A non-streaming request returns a document you can validate, diff, and archive; a streaming request returns a sequence of deltas whose contract lives in details that most API references never fully specify. Two endpoints can both claim OpenAI compatibility and still disagree about where usage appears, whether DONE is sent at all, and how tool-call arguments are split across chunks. Your integration test compares the final message, so it goes green, and your UI still breaks for the second user who tries a tool call. Copying a response out of a terminal gives you a cleaned-up paraphrase with keep-alive comments stripped and line endings normalized, which is exactly the information you need to keep. Record the raw bytes instead, one file per shape you care about. mkdir -p fixtures curl -sN "$OLD HOST/v1/chat/completions" \ -H "Authorization: Bearer $OLD KEY" \ -H "Content-Type: application/json" \ -H "Accept: text/event-stream" \ --data @requests/tool-call.json \ fixtures/old-toolcall.sse Capture at least four shapes before you touch the client: a plain text answer, a multi-chunk tool call, a stop-sequence hit, and an error object returned after a 200 status line. Those four cover the failure modes that survive a happy-path demo, and each one costs you a few kilobytes of disk. The normalizer below treats the stream as a state machine rather than a string split, which is what lets you key tool-call fragments by index instead of by arrival order. It ignores comments and unknown fields on purpose, because a parser that throws on an unrecognized line is a parser that breaks the first time a provider adds one. import json from dataclasses import dataclass, field @dataclass class Event: kind: str text | tool call | usage | end text: str = "" index: int = -1 name: str = "" args: str = "" usage: dict = field default factory=dict def events from lines lines : tool name, tool args = {}, {} for raw in lines: line = raw.decode "utf-8", "replace" .rstrip "\r\n" if not line or line.startswith ":" : continue keep-alive comment if not line.startswith "data:" : continue event:, id:, retry: payload = line 5: .strip if payload == " DONE ": yield Event "end", text=" DONE " return chunk = json.loads payload if chunk.get "usage" : yield Event "usage", usage=chunk "usage" for choice in chunk.get "choices" or : delta = choice.get "delta" or {} if delta.get "content" : yield Event "text", text=delta "content" for call in delta.get "tool calls" or : i = call.get "index", 0 fn = call.get "function" or {} if fn.get "name" : tool name i = fn "name" frag = fn.get "arguments" or "" tool args i = tool args.get i, "" + frag yield Event "tool call", index=i, name=tool name.get i, "" , args=frag if choice.get "finish reason" : yield Event "end", text=choice "finish reason" Two properties matter more than the parsing itself. The reducer is incremental, so a fragment that arrives in a later chunk still lands on the right tool call, and it never blocks on the socket, so a stalled stream shows up as a timeout you control rather than a hung request handler. That is the difference between a cutover you can debug and one you can only restart. Write the fixture test first, because it fails for reasons you can read in a traceback instead of reasons buried in a timeline. The version below splits the argument JSON at an arbitrary offset, which is the failure you are actually guarding against. import json, pytest ARGS = '{"q":"x"}' def sse obj : return "data: " + json.dumps obj + "\n" .encode FIXTURE = b": keep-alive\n", sse {"choices": {"index": 0, "delta": {"role": "assistant"}} } , sse {"choices": {"index": 0, "delta": {"tool calls": {"index": 0, "function": {"name": "search", "arguments": ARGS :4 }} }} } , sse {"choices": {"index": 0, "delta": {"tool calls": {"index": 0, "function": {"arguments": ARGS 4: }} }} } , sse {"choices": {"index": 0, "delta": {}, "finish reason": "tool calls"} } , b"data: DONE \n", def test arguments survive reassembly : log = list events from lines FIXTURE args = "".join e.args for e in log if e.kind == "tool call" assert json.loads args == {"q": "x"} assert log -1 .kind == "end" and log -1 .text == "tool calls" Now point the same reducer at the candidate endpoint and compare shapes rather than text, since the model's wording will legitimately differ while the protocol should not. import urllib.request def http lines url, key, body : req = urllib.request.Request url, data=json.dumps body .encode "utf-8" , headers={"Authorization": "Bearer " + key, "Content-Type": "application/json", "Accept": "text/event-stream"}, with urllib.request.urlopen req, timeout=120 as resp: for line in resp: yield line @pytest.mark.parametrize "fixture", sorted Path "fixtures" .glob " .sse" def test candidate stream shape fixture : recorded = list events from lines fixture.read bytes .splitlines keepends=True live = list events from lines http lines CANDIDATE URL, CANDIDATE KEY, payload for fixture assert e.kind for e in recorded == e.kind for e in live This is where a free endpoint is genuinely useful rather than decorative: you need a second stream to replay against, and you do not want to pay twice for a rehearsal that may run fifty times in an afternoon. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which is enough to stand up that second endpoint for the harness above; treat both as availability claims you verify on the project page rather than as guarantees, and measure the limits yourself before you plan around them. Not every mismatch is a blocker, and treating all of them as fatal is how a migration stalls for a week. Sort each one into cosmetic, negotiable, or fatal, then fix only the last two. Difference Typical symptom Verdict Keep-alive comment lines parser raises on : ping cosmetic, but fix the parser No DONE marker spinner never stops fatal for any UI that waits on it usage missing or late cost ledger reads zero negotiated: count tokens locally Tool arguments split differently {} passed to a function cosmetic if you key by index finish reason null on the last chunk agent loop never exits fatal HTTP 200 then error object silent empty answer fatal, needs its own branch Every migration leaves wiring that still points at the old provider, and the stream layer hides several of them. Delete the retry wrapper keyed on the old error envelope, or it will retry a 400 that the new endpoint means literally. Retire the hardcoded assumption that only one tool call can be in flight, because the second one will arrive later as a duplicate action. Re-point the timeout tuned to the old first-token latency, and remove the dashboard-shaped usage read before you close the account you were reading it from. None of these announce themselves; they surface as odd behavior three weeks after the cutover, when nobody remembers which endpoint is answering. Do not run this migration if you depend on provider-specific features such as logprobs or enforced structured output, because a normalizer cannot invent them. Skip it if you cannot record fixtures, since replaying unrecorded traffic means testing against a moving target. And do not put a free tier underneath a paid SLA without a fallback path you have actually exercised, because availability claims and availability guarantees are different documents. The harness above is deliberately small: one reducer, one fixture per shape, one structural assertion per endpoint. Run it before the cutover rather than after, and keep the fixtures in the repository next to the tests, because they are the only surviving record of what the old stream actually did. Key Takeaways - •This entry in the migration diary is about that layer: recording real streams as byte fixtures, reducing every provider's chunk shape into one event model, and gating the cutover on structural equality rather than a spot check you ran once in a notebook. - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/techpy 768/migration-diary-the-streaming-parser-is-what-actually-breaks-at-cutover-35pd