Which framework should I actually use? Every comparison article has an
opinion. Almost none of them has data.
So I built the same agent three times — once in Strands, once in
LangGraph, once in CrewAI — ran 27 executions, and routed every
single LLM call through a local recorder proxy so the logs are directly
comparable. The frameworks write different log formats, different trace
shapes, different everything. A one-file proxy in front of all of them fixed
that.
The headline finding: LangGraph's explicit verify/revise loop cut output
variance by 77% (word-count spread 13 -> 3) — at 2.5x the tokens and 2.5x
the latency. Explicit control buys determinism, and you pay for it in tokens
and latency.
https://github.com/sunnydachs/agent-framework-showdown
A tech-news digest agent, identical across all three frameworks:
fetch_headlines toolword_count tool, revising if out of band
The tools are deterministic and local — no network, no LLM inside them —
because the thing being measured is the framework's behavior, not the tool's.
Same model behind the proxy for all three runs.
The philosophy split is real, and it shows up in the implementation:
The dependency weight differs too: Strands 262MB (81 packages), LangGraph
71MB (45), CrewAI 699MB (142). CrewAI's weight is the flip side of
"fastest to prototype."
One recorder proxy sits in front of every framework
(proxy/rec_proxy.py): an HTTP server that forwards to any
OpenAI-compatible endpoint and writes every request/response pair as JSONL —
the full messages the framework sent (system prompt, tool schemas,
conversation history), the raw response (SSE-streamed or JSON), token usage,
latency, and status. The API key is stripped before writing.
An X-Run-Label header splits traces per run. Strands and LangChain stream
SSE, so a small reassembler (parse_sse.py) normalizes streamed responses
into the same shape as non-streamed ones (content / reasoning / tool_calls /
usage).
Same trace shape across frameworks is what lets you diff messages, tokens,
and latency exactly. Without it, "which framework is slower" is vibes.
Three scenarios:
text -> content),
testing how each framework copes with schema change
Word-count spread across 3 runs (max - min, lower = more stable):
| Scenario | Strands | LangGraph | CrewAI |
|---|---|---|---|
| base | 15 | 13 | 0 |
| tight | 11 | 3 | 2 |
| drift | 12 | 16 | 0 |
| Strands | LangGraph | CrewAI | |
|---|---|---|---|
| LLM calls | 3-5 (adaptive) | 1 (base) / 2 (tight) | 4 fixed |
| Total tokens (base) | 2,370 | 2,007 | 2,532 |
| Total latency | 4.2s | 4.7s | 3.8s |
LangGraph in base mode made a single LLM call and wrote the draft in one
shot. Word-count spread: 13 words. Switch to tight, and the explicit
verify/revise loop fires: spread drops to 3 (−77%), tokens go 2,007 -> 5,262,
latency 4.7s -> 11.8s. The graph structure guarantees the loop runs — that
is the product you're buying, and the price is 2.5x on everything.
CrewAI produced byte-identical output across all 3 base and drift runs
(96/96/96 words). temperature=0 plus the role prompt dominates. The flip
side: the call structure is always 4 calls, fixed. The framework that "just
ships it" also just repeats it.
Strands self-corrected inside its own loop. In one tight run the model
wrote a 77-word draft, called check_word_count, reasoned "77 is under 95,
I need to expand it," revised itself to 103, and verified again — five LLM
calls, variable depth per run ([5, 3, 4]). The model deciding when to stop
is part of the design, and the call count being adaptive is what you get for
trusting it.
The rename was absorbed 100%: all three frameworks' models called the tool
with the new argument name, zero wrong-arg calls, no error recovery
triggered. Worth being precise about what this does and doesn't show: a
one-argument rename is the gentlest possible schema change. Type changes,
removed arguments, or changed return shapes would likely break the
model-driven side (which trusts its prompt over the wire), while LangGraph
would be immune — its tool calls live in code, not in a prompt.
python3 runs/run_matrix.py # 27 runs in ~270s
python3 runs/analyze_matrix.py # -> artifacts/matrix_report.json
The repo README has the full setup (three venvs, the recorder proxy, the
per-framework run commands). One proxy in front of everyone is the whole
trick — when the frameworks write different log formats, a same-shape
recorder is the simplest honest answer.