# I built the same agent in Strands, LangGraph, and CrewAI — and recorded every LLM call to see how they actually differ

> Source: <https://dev.to/sunnydachs/i-built-the-same-agent-in-strands-langgraph-and-crewai-and-recorded-every-llm-call-to-see-how-4ae8>
> Published: 2026-09-21 05:47:42+00:00

Which framework should I actually use? Every comparison article has an

opinion. Almost none of them has data.

So I built the same agent three times — once in **Strands**, once in

**LangGraph**, once in **CrewAI** — ran 27 executions, and routed every

single LLM call through a local recorder proxy so the logs are directly

comparable. The frameworks write different log formats, different trace

shapes, different everything. A one-file proxy in front of all of them fixed

that.

The headline finding: LangGraph's explicit verify/revise loop cut output

variance by **77%** (word-count spread 13 -> 3) — at 2.5x the tokens and 2.5x

the latency. Explicit control buys determinism, and you pay for it in tokens

and latency.

[https://github.com/sunnydachs/agent-framework-showdown](https://github.com/sunnydachs/agent-framework-showdown)

A tech-news digest agent, identical across all three frameworks:

`fetch_headlines` tool`word_count` tool, revising if out of band
The tools are deterministic and local — no network, no LLM inside them —

because the thing being measured is the framework's behavior, not the tool's.

Same model behind the proxy for all three runs.

The philosophy split is real, and it shows up in the implementation:

The dependency weight differs too: Strands 262MB (81 packages), LangGraph

71MB (45), CrewAI 699MB (142). CrewAI's weight is the flip side of

"fastest to prototype."

One recorder proxy sits in front of every framework

(`proxy/rec_proxy.py`): an HTTP server that forwards to any

OpenAI-compatible endpoint and writes every request/response pair as JSONL —

the full messages the framework sent (system prompt, tool schemas,

conversation history), the raw response (SSE-streamed or JSON), token usage,

latency, and status. The API key is stripped before writing.

An `X-Run-Label` header splits traces per run. Strands and LangChain stream

SSE, so a small reassembler (`parse_sse.py`) normalizes streamed responses

into the same shape as non-streamed ones (content / reasoning / tool_calls /

usage).

Same trace shape across frameworks is what lets you diff messages, tokens,

and latency exactly. Without it, "which framework is slower" is vibes.

Three scenarios:

`text` -> `content`),
testing how each framework copes with schema change
Word-count spread across 3 runs (max - min, lower = more stable):

| Scenario | Strands | LangGraph | CrewAI | 
|---|---|---|---|
| base | 15 | **13** | **0** | 
| tight | 11 | **3** | 2 | 
| drift | 12 | 16 | 0 | 

|  | Strands | LangGraph | CrewAI | 
|---|---|---|---|
| LLM calls | 3-5 (adaptive) | 1 (base) / 2 (tight) | 4 fixed | 
| Total tokens (base) | 2,370 | 2,007 | 2,532 | 
| Total latency | 4.2s | 4.7s | 3.8s | 

**LangGraph** in base mode made a single LLM call and wrote the draft in one

shot. Word-count spread: 13 words. Switch to tight, and the explicit

verify/revise loop fires: spread drops to 3 (−77%), tokens go 2,007 -> 5,262,

latency 4.7s -> 11.8s. The graph structure *guarantees* the loop runs — that

is the product you're buying, and the price is 2.5x on everything.

**CrewAI** produced byte-identical output across all 3 base and drift runs

(96/96/96 words). temperature=0 plus the role prompt dominates. The flip

side: the call structure is always 4 calls, fixed. The framework that "just

ships it" also just repeats it.

**Strands** self-corrected inside its own loop. In one tight run the model

wrote a 77-word draft, called `check_word_count`, reasoned "77 is under 95,

I need to expand it," revised itself to 103, and verified again — five LLM

calls, variable depth per run ([5, 3, 4]). The model deciding when to stop

is part of the design, and the call count being adaptive is what you get for

trusting it.

The rename was absorbed 100%: all three frameworks' models called the tool

with the **new** argument name, zero wrong-arg calls, no error recovery

triggered. Worth being precise about what this does and doesn't show: a

one-argument rename is the gentlest possible schema change. Type changes,

removed arguments, or changed return shapes would likely break the

model-driven side (which trusts its prompt over the wire), while LangGraph

would be immune — its tool calls live in code, not in a prompt.

```
python3 runs/run_matrix.py       # 27 runs in ~270s
python3 runs/analyze_matrix.py   # -> artifacts/matrix_report.json
```

The repo README has the full setup (three venvs, the recorder proxy, the

per-framework run commands). One proxy in front of everyone is the whole

trick — when the frameworks write different log formats, a same-shape

recorder is the simplest honest answer.