{"slug": "what-agent-frameworks-cost-on-the-wire-measurements-from-agentic-arena", "title": "What agent frameworks cost on the wire: measurements from agentic-arena", "summary": "A developer built agentic-arena, a benchmark that holds the model, tools, datasets and iteration budget fixed to measure how seven agent frameworks differ on the wire. Six of seven frameworks stayed within a 1.15x band of a hand-rolled stdlib loop on prompt tokens, while smolagents used 3.90x because it restates tool schemas in prose, and five of seven adapters failed to pass a shared request timeout to their client, hanging 20 seconds on a one-second budget.", "body_md": "Most framework comparisons argue from feature lists. I wanted numbers, so [agentic-arena](https://github.com/code-with-rashid/agentic-arena) holds the model, tools, datasets and iteration budget fixed and measures what each framework does differently.\n\n**Read this first:** everything below is measured against a scripted (mock) model, so the turns are byte-identical for every framework. That makes these wire and behaviour measurements. They say nothing about which framework writes better answers.\n\nMean prompt tokens per item on a 15-item tool-use task:\n\n| framework | prompt tokens | vs baseline | \n|---|---|---|\n| vanilla (stdlib loop) | 753.5 | 1.00x | \n| langgraph | 753.5 | 1.00x | \n| pydantic_ai | 794.0 | 1.05x | \n| microsoft_af | 802.0 | 1.06x | \n| google_adk | 836.1 | 1.11x | \n| openai_agents | 856.9 | 1.14x | \n| smolagents | 2935.5 | 3.90x | \n\nSix of seven sit inside a 1.15x band, and none is leaner than the hand-rolled loop. smolagents' 3.90x is a 4,207-character system prompt where the arena asked for 384, including a prose restatement of tools it already sent as a schema. The tools are transmitted, and billed, twice.\n\nThat is a worst case. The same conversation run to 30 turns shrinks the gap from 8.83x on the first request to 1.27x on the 31st, because a fixed overhead decays as the conversation grows.\n\nI scripted 429, 500 and 400 responses. The hand-rolled baseline has no retry at all, so one 429 loses the item. Every framework survives a single 429. Only smolagents survives three in a row, by quietly sleeping roughly two to four minutes on one item. The item passes, so nothing in a scorecard shows it, but in a batch your throughput quietly collapses.\n\nA three-role pipeline doubles the LLM calls and multiplies prompt tokens by 2.5x, whether you build it with a graph library or a or loop. The structure costs that, not the framework.\n\nModel-decided handoffs cost about 10% more prompt than the same pipeline wired structurally. Of that gap, 94% is the ransfer_to_* tool schemas, which ride on every request whether or not anyone delegates. You pay for the options you offer, not the ones you use.\n\nFive of the seven adapters never passed the shared request timeout to their client, so they waited out their library's default through a 20-second hang on a one-second budget. A hanging mock provider exposed it. It is fixed, and gated in CI.\n\nEverything above has a command that regenerates it, and CI regenerates each number on a clean install:\n\n```\ngit clone https://github.com/code-with-rashid/agentic-arena\ncd agentic-arena\npython -m pip install -e .\npython -m arena run --arena tool_use --framework all --mode mock --no-scorecard\n```\n\nFull findings and the reasoning behind each: [https://code-with-rashid.github.io/agentic-arena/findings/](https://code-with-rashid.github.io/agentic-arena/findings/)", "url": "https://wpnews.pro/news/what-agent-frameworks-cost-on-the-wire-measurements-from-agentic-arena", "canonical_source": "https://dev.to/code-with-rashid/what-agent-frameworks-cost-on-the-wire-measurements-from-agentic-arena-13o7", "published_at": "2026-09-30 17:05:46+00:00", "updated_at": "2026-09-30 17:16:55.741248+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "mlops"], "entities": ["agentic-arena", "LangGraph", "Pydantic AI", "Microsoft Agent Framework", "Google ADK", "OpenAI Agents SDK", "smolagents"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/what-agent-frameworks-cost-on-the-wire-measurements-from-agentic-arena", "markdown": "https://wpnews.pro/news/what-agent-frameworks-cost-on-the-wire-measurements-from-agentic-arena.md", "text": "https://wpnews.pro/news/what-agent-frameworks-cost-on-the-wire-measurements-from-agentic-arena.txt", "jsonld": "https://wpnews.pro/news/what-agent-frameworks-cost-on-the-wire-measurements-from-agentic-arena.jsonld"}}