I built the same agent in Strands, LangGraph, and CrewAI — and recorded every LLM call to see how they actually differ

A developer built the same tech-news digest agent in three frameworks — Strands, LangGraph, and CrewAI — and routed all 27 executions through a local recorder proxy to compare LLM calls directly. LangGraph's explicit verify/revise loop cut output variance by 77% (word-count spread 13 to 3) but at 2.5x the tokens and 2.5x the latency, while CrewAI produced byte-identical output across runs with a fixed four-call structure and Strands self-corrected adaptively with variable call counts. Dependency footprints also diverged: Strands 262MB (81 packages), LangGraph 71MB (45), and CrewAI 699MB (142).

Which framework should I actually use? Every comparison article has an opinion. Almost none of them has data. So I built the same agent three times — once in Strands , once in LangGraph , once in CrewAI — ran 27 executions, and routed every single LLM call through a local recorder proxy so the logs are directly comparable. The frameworks write different log formats, different trace shapes, different everything. A one-file proxy in front of all of them fixed that. The headline finding: LangGraph's explicit verify/revise loop cut output variance by 77% word-count spread 13 - 3 — at 2.5x the tokens and 2.5x the latency. Explicit control buys determinism, and you pay for it in tokens and latency. https://github.com/sunnydachs/agent-framework-showdown https://github.com/sunnydachs/agent-framework-showdown A tech-news digest agent, identical across all three frameworks: fetch headlines tool word count tool, revising if out of band The tools are deterministic and local — no network, no LLM inside them — because the thing being measured is the framework's behavior, not the tool's. Same model behind the proxy for all three runs. The philosophy split is real, and it shows up in the implementation: The dependency weight differs too: Strands 262MB 81 packages , LangGraph 71MB 45 , CrewAI 699MB 142 . CrewAI's weight is the flip side of "fastest to prototype." One recorder proxy sits in front of every framework proxy/rec proxy.py : an HTTP server that forwards to any OpenAI-compatible endpoint and writes every request/response pair as JSONL — the full messages the framework sent system prompt, tool schemas, conversation history , the raw response SSE-streamed or JSON , token usage, latency, and status. The API key is stripped before writing. An X-Run-Label header splits traces per run. Strands and LangChain stream SSE, so a small reassembler parse sse.py normalizes streamed responses into the same shape as non-streamed ones content / reasoning / tool calls / usage . Same trace shape across frameworks is what lets you diff messages, tokens, and latency exactly. Without it, "which framework is slower" is vibes. Three scenarios: text - content , testing how each framework copes with schema change Word-count spread across 3 runs max - min, lower = more stable : | Scenario | Strands | LangGraph | CrewAI | |---|---|---|---| | base | 15 | 13 | 0 | | tight | 11 | 3 | 2 | | drift | 12 | 16 | 0 | | | Strands | LangGraph | CrewAI | |---|---|---|---| | LLM calls | 3-5 adaptive | 1 base / 2 tight | 4 fixed | | Total tokens base | 2,370 | 2,007 | 2,532 | | Total latency | 4.2s | 4.7s | 3.8s | LangGraph in base mode made a single LLM call and wrote the draft in one shot. Word-count spread: 13 words. Switch to tight, and the explicit verify/revise loop fires: spread drops to 3 −77% , tokens go 2,007 - 5,262, latency 4.7s - 11.8s. The graph structure guarantees the loop runs — that is the product you're buying, and the price is 2.5x on everything. CrewAI produced byte-identical output across all 3 base and drift runs 96/96/96 words . temperature=0 plus the role prompt dominates. The flip side: the call structure is always 4 calls, fixed. The framework that "just ships it" also just repeats it. Strands self-corrected inside its own loop. In one tight run the model wrote a 77-word draft, called check word count , reasoned "77 is under 95, I need to expand it," revised itself to 103, and verified again — five LLM calls, variable depth per run 5, 3, 4 . The model deciding when to stop is part of the design, and the call count being adaptive is what you get for trusting it. The rename was absorbed 100%: all three frameworks' models called the tool with the new argument name, zero wrong-arg calls, no error recovery triggered. Worth being precise about what this does and doesn't show: a one-argument rename is the gentlest possible schema change. Type changes, removed arguments, or changed return shapes would likely break the model-driven side which trusts its prompt over the wire , while LangGraph would be immune — its tool calls live in code, not in a prompt. python3 runs/run matrix.py 27 runs in ~270s python3 runs/analyze matrix.py - artifacts/matrix report.json The repo README has the full setup three venvs, the recorder proxy, the per-framework run commands . One proxy in front of everyone is the whole trick — when the frameworks write different log formats, a same-shape recorder is the simplest honest answer.