{"slug": "we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then-we-changed", "title": "We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model.", "summary": "A benchmark of 160 agent tasks found LangGraph and Pydantic AI tied at 100% correctness with gpt-4o, but swapping to gpt-4o-mini dropped overall correctness to 75%, with one date-arithmetic task failing 20 out of 20 times regardless of framework. The results indicate that model choice, not framework, drives correctness on structured tasks.", "body_md": "We ran 160 agent tasks across two frameworks. The frameworks tied. Then we changed the model — and one task failed 20 times in a row.\n\nThe LangChain vs LangGraph question comes up every time someone starts a new agent project. We had a slightly different version of it: once you've chosen LangGraph as your runtime, does picking the library around it actually change production outcomes?\n\nSo we ran a controlled benchmark to find out.\n\nLangGraph 1.2.9 against Pydantic AI 2.13.0. Four tasks, designed to test realistic agent work: order processing with tool calls, shipping quote calculation, refund eligibility with date logic, and inventory reorder decisions. All tasks shared the same tool implementations and the same scorer — the only variable was which library orchestrated the calls.\n\nWe used gpt-4o, temperature 0, no parallel tool calls. The harness is public at [github.com/benchclawio/harness](https://github.com/benchclawio/harness). Every run was logged, nothing was cherry-picked, the raw JSONL is published.\n\n160 task executions total. Here's what happened.\n\nLangGraph: 100% correctness across all four tasks.\n\nPydantic AI: 100% correctness across all four tasks.\n\nNot close — exact tie. Zero correctness gap on any task.\n\nThere was a performance gap: LangGraph was roughly 1.4–1.8 seconds faster per task on median wall time. The cause is Pydantic AI's async-to-sync bridge — when you call it synchronously, it spins an event loop internally, and that overhead is real and consistent. If you're building a latency SLO, that matters. If you're evaluating correctness first, the frameworks are identical with gpt-4o.\n\nThis result is unsatisfying in a useful way. It tells you that the framework abstraction is not where correctness variation lives — at least on these tasks, at this model, at this scale.\n\nSo we ran the same tasks again. Same harness, same task suite. Different model.\n\nWe substituted gpt-4o-mini for gpt-4o. Everything else held constant.\n\nOverall correctness dropped to 75%. The framework split: same.\n\nOne task failed completely, 20 for 20 runs: `refund-policy-minimal-tools`\n\n. The task requires computing whether a return is within the 18-day return window. gpt-4o-mini calculated 19 days inclusive where the answer is 18 days exclusive — a date-arithmetic edge case. It failed this identically on every single run, regardless of which framework was handling the tool calls.\n\nThe other three tasks: 100%. The failure wasn't variance — it was systematic. And it was entirely a model property, not a framework property.\n\nWe had to decide what to compare LangGraph against before we started. Our dependency analysis ([benchclaw.io/langchain-vs-langgraph/](https://benchclaw.io/langchain-vs-langgraph/)) found that LangChain 1.3.14 now declares LangGraph as an unconditional dependency — installing LangChain installs LangGraph. The reverse isn't true. So \"LangChain vs LangGraph\" is less of a choice than it used to be; the more real decision is whether to write in Pydantic AI or in LangGraph primitives once you're in the LangGraph runtime.\n\nEven reframed that way, our benchmark says the framework abstraction doesn't move correctness outcomes on structured tasks at gpt-4o.\n\nWhat did move outcomes was the model. By a lot, on a task that sounds easy.\n\nThe practical implication: if your task suite involves any temporal reasoning — eligibility windows, expiry dates, SLA calculations — test both your intended model and a cheaper fallback explicitly before you ship. \"It works in testing\" is not enough if testing only used gpt-4o and production routes some traffic to gpt-4o-mini or a smaller model.\n\nBefore we ran the model comparison, we expected the framework difference to be small. We expected correctness to hold. What we didn't anticipate was how systematic the failure mode was.\n\ngpt-4o-mini didn't get the date arithmetic right on some runs and wrong on others. It got it wrong 20 out of 20 times, by exactly the same margin. That's not a probabilistic sampling failure — that's a knowledge gap baked into the model at this temperature and this task phrasing.\n\nThe implication is that per-task model validation matters more than framework benchmarking for production agent decisions. You can swap LangGraph for Pydantic AI or back again in an afternoon. You can't patch a model's arithmetic mid-deployment.\n\nFull benchmark with methodology, raw JSONL, and harness: [benchclaw.io/langgraph-vs-pydantic-ai-benchmark/](https://benchclaw.io/langgraph-vs-pydantic-ai-benchmark/)\n\nHarness repo (Apache 2.0, citable): [github.com/benchclawio/harness](https://github.com/benchclawio/harness)\n\nIf you want to reproduce it, the harness runs offline on any machine with Python and an OpenAI key.\n\n*LangGraph 1.2.9, pydantic-ai-slim[openai] 2.13.0, gpt-4o (temperature 0) and gpt-4o-mini (temperature 0). Benchmarked 2026-07-24. All runs logged; nothing omitted.*", "url": "https://wpnews.pro/news/we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then-we-changed", "canonical_source": "https://dev.to/benchclaw/we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then-we-changed-the-model-3hip", "published_at": "2026-08-29 13:27:52+00:00", "updated_at": "2026-08-29 13:49:01.303902+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "large-language-models"], "entities": ["LangGraph", "Pydantic AI", "gpt-4o", "gpt-4o-mini", "LangChain", "benchclaw.io"], "alternates": {"html": "https://wpnews.pro/news/we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then-we-changed", "markdown": "https://wpnews.pro/news/we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then-we-changed.md", "text": "https://wpnews.pro/news/we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then-we-changed.txt", "jsonld": "https://wpnews.pro/news/we-ran-160-agent-tasks-across-two-frameworks-the-frameworks-tied-then-we-changed.jsonld"}}