Conclusion first: your JSON will diff clean on the first day of the migration, and the server-sent stream underneath it will still hand you truncated answers, half-assembled tool calls, and a finish_reason that never fires. Transport is the layer that fails quietly, because nothing raises and nothin
g looks wrong until an agent loop calls a function with {} as its arguments. This entry in the migration diary is about that layer: recording real streams as byte fixtures, reducing every provider's chunk shape into one event model, and gating the cutover on structural equality rather than a spot check you ran once in a notebook. A non-streaming request returns a document you can validate, diff, and archive; a streaming request returns a sequence of deltas whose contract lives in details that most API references never fully specify. Two endpoints can both claim OpenAI compatibility and still disagree about where usage appears, whether [DONE] is sent at all, and how tool-call arguments are split across chunks.
Your integration test compares the final message, so it goes green, and your UI still breaks for the second user who tries a tool call. Copying a response out of a terminal gives you a cleaned-up paraphrase with keep-alive comments stripped and line endings normalized, which is exactly the information you need to keep. Record the raw bytes instead, one file per shape you care about. mkdir -p fixtures curl -sN "$OLD_HOST/v1/chat/completions" \ -H "Authorization: Bearer $OLD_KEY" \ -H "Content-Type: application/json" \ -H "Accept: text/event-stream" \ --data @requests/tool-call.json \ > fixtures/old-toolcall.sse Capture at least four shapes before you touch the client: a plain text answer, a multi-chunk tool call, a stop-sequence hit, and an error object returned after a 200 status line.
Those four cover the failure modes that survive a happy-path demo, and each one costs you a few kilobytes of disk. The normalizer below treats the stream as a state machine rather than a string split, which is what lets you key tool-call fragments by index instead of by arrival order. It ignores comments and unknown fields on purpose, because a parser that throws on an unrecognized line is a parser that breaks the first time a provider adds one. import json from dataclasses import dataclass, field @dataclass class Event: kind: str # text | tool_call | usage | end text: str = "" index: int = -1 name: str = "" args: str = "" usage: dict = field(default_factory=dict) def events_from_lines(lines): tool_name, tool_args = {}, {} for raw in lines: line = raw.decode("utf-8", "replace").rstrip("\r\n") if not line or line.startswith(":"): continue # keep-alive comment if not line.startswith("data:"): continue # event:, id:, retry: payload = line[5:].strip() if payload == "[DONE]": yield Event("end", text="[DONE]") return chunk = json.loads(payload) if chunk.get("usage"): yield Event("usage", usage=chunk["usage"]) for choice in chunk.get("choices") or []: delta = choice.get("delta") or {} if delta.get("content"): yield Event("text", text=delta["content"]) for call in delta.get("tool_calls") or []: i = call.get("index", 0) fn = call.get("function") or {} if fn.get("name"): tool_name[i] = fn["name"] frag = fn.get("arguments") or "" tool_args[i] = tool_args.get(i, "") + frag yield Event("tool_call", index=i, name=tool_name.get(i, ""), args=frag) if choice.get("finish_reason"): yield Event("end", text=choice["finish_reason"]) Two properties matter more than the parsing itself.
The reducer is incremental, so a fragment that arrives in a later chunk still lands on the right tool call, and it never blocks on the socket, so a stalled stream shows up as a timeout you control rather than a hung request handler. That is the difference between a cutover you can debug and one you can only restart. Write the fixture test first, because it fails for reasons you can read in a traceback instead of reasons buried in a timeline. The version below splits the argument JSON at an arbitrary offset, which is the failure you are actually guarding against.
import json, pytest ARGS = '{"q":"x"}' def sse(obj): return ("data: " + json.dumps(obj) + "\n").encode() FIXTURE = [ b": keep-alive\n", sse({"choices": [{"index": 0, "delta": {"role": "assistant"}}]}), sse({"choices": [{"index": 0, "delta": {"tool_calls": [ {"index": 0, "function": {"name": "search", "arguments": ARGS[:4]}}]}}]}), sse({"choices": [{"index": 0, "delta": {"tool_calls": [ {"index": 0, "function": {"arguments": ARGS[4:]}}]}}]}), sse({"choices": [{"index": 0, "delta": {}, "finish_reason": "tool_calls"}]}), b"data: [DONE]\n", ] def test_arguments_survive_reassembly(): log = list(events_from_lines(FIXTURE)) args = "".join(e.args for e in log if e.kind == "tool_call") assert json.loads(args) == {"q": "x"} assert log[-1].kind == "end" and log[-1].text == "tool_calls" Now point the same reducer at the candidate endpoint and compare shapes rather than text, since the model's wording will legitimately differ while the protocol should not. import urllib.request def http_lines(url, key, body): req = urllib.request.Request( url, data=json.dumps(body).encode("utf-8"), headers={"Authorization": "Bearer " + key, "Content-Type": "application/json", "Accept": "text/event-stream"}, ) with urllib.request.urlopen(req, timeout=120) as resp: for line in resp: yield line @pytest.mark.parametrize("fixture", sorted(Path("fixtures").glob("*.sse"))) def test_candidate_stream_shape(fixture): recorded = list(events_from_lines(fixture.read_bytes().splitlines(keepends=True))) live = list(events_from_lines(http_lines(CANDIDATE_URL, CANDIDATE_KEY, payload_for(fixture)))) assert [e.kind for e in recorded] == [e.kind for e in live] This is where a free endpoint is genuinely useful rather than decorative: you need a second stream to replay against, and you do not want to pay twice for a rehearsal that may run fifty times in an afternoon. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which is enough to stand up that second endpoint for the harness above; treat both as availability claims you verify on the project page rather than as guarantees, and measure the limits yourself before you plan around them. Not every mismatch is a blocker, and treating all of them as fatal is how a migration stalls for a week. Sort each one into cosmetic, negotiable, or fatal, then fix only the last two. Difference Typical symptom Verdict Keep-alive comment lines parser raises on : ping cosmetic, but fix the parser No [DONE] marker spinner never stops fatal for any UI that waits on it usage missing or late cost ledger reads zero negotiated: count tokens locally Tool arguments split differently {} passed to a function cosmetic if you key by index finish_reason null on the last chunk agent loop never exits fatal HTTP 200 then error object silent empty answer fatal, needs its own branch Every migration leaves wiring that still points at the old provider, and the stream layer hides several of them. Delete the retry wrapper keyed on the old error envelope, or it will retry a 400 that the new endpoint means literally.
Retire the hardcoded assumption that only one tool call can be in flight, because the second one will arrive later as a duplicate action. Re-point the timeout tuned to the old first-token latency, and remove the dashboard-shaped usage read before you close the account you were reading it from. None of these announce themselves; they surface as odd behavior three weeks after the cutover, when nobody remembers which endpoint is answering. Do not run this migration if you depend on provider-specific features such as logprobs or enforced structured output, because a normalizer cannot invent them.
Skip it if you cannot record fixtures, since replaying unrecorded traffic means testing against a moving target. And do not put a free tier underneath a paid SLA without a fallback path you have actually exercised, because availability claims and availability guarantees are different documents. The harness above is deliberately small: one reducer, one fixture per shape, one structural assertion per endpoint. Run it before the cutover rather than after, and keep the fixtures in the repository next to the tests, because they are the only surviving record of what the old stream actually did.
Key Takeaways #
- •This entry in the migration diary is about that layer: recording real streams as byte fixtures, reducing every provider's chunk shape into one event model, and gating the cutover on structural equality rather than a spot check you ran once in a notebook.
- •This story was reported by Dev.to , covering developments in thedev space.
- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.
📖 Continue reading the full article: