cd /news/ai-agents/i-benchmarked-my-coding-agent-agains… · home › topics › ai-agents › article
[ARTICLE · art-172639] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=· neutral

I benchmarked my coding agent against seven others. It tied, and it was not the cheapest.

A developer benchmarked their open-source coding agent, Turnloom, against seven others on 19 tasks using gpt-5.1, finding that six agents solved between 9 and 11 tasks—a statistical tie given the small sample. Turnloom solved 10 of 19 at $0.160 per task, and the developer attributed its higher output-token cost to using Chat Completions rather than OpenAI's Responses API, an untested hypothesis now on the roadmap. The run also surfaced three bugs in Turnloom, all fixed in version 0.7.1.

recommendation

Benchmark coding agents on your own tasks with metered per-call token counts, and report confidence intervals—19 tasks cannot distinguish agents solving 9 to 11.

more in playbook: Evals and testing for LLM apps
by read2 min views3 publishedNov 19, 2026

I maintain Turnloom, a small open-source coding agent. Before telling anyone to use it, I wanted to know how it does next to the agents people already use. So I ran eight of them on the same tasks, with the same model, and metered every model call the same way.

gpt-5.1 for every agent.

Agent Solved Cost per task Input tokens per task Output tokens per task
OpenCode 11 of 19 $0.091 110,562 6,060
Goose 11 of 19 $0.101 180,595 5,470
Terminus 2 11 of 19 $0.162 51,228 14,056
Turnloom 10 of 19 $0.160 70,192 12,637
mini-SWE-agent 10 of 19 $0.181 134,693 14,796
OpenHands SDK 9 of 19 $0.186 326,500 11,477
Codex CLI 6 of 19 $0.156 275,188 9,483
Aider 1 of 19 $0.061 3,626 5,745

Six agents solved between 9 and 11 tasks. With 19 tasks, that is a tie: the 95% interval for 10 of 19 runs from 32% to 73%. Aider's row is not a fair picture of Aider, because the benchmark runner gives it one message and no way to run commands.

Sending less input did not make Turnloom cheap. Turnloom folds old tool output out of the prompt and keeps it in an archive the model can read back. That worked: among the agents that work through a tool loop, only Terminus 2 sent fewer input tokens. But output tokens cost eight times as much as input, and Turnloom wrote about twice as many as the two cheapest agents. Output was $0.126 of its $0.160 per task.

The likely reason is the API. Leaving Aider aside, the agents with the fewest output tokens call OpenAI's Responses API, which carries the model's reasoning from one call to the next. Turnloom uses Chat Completions, which does not. I have not tested that explanation yet. It is now on the roadmap.

A benchmark is a good bug finder. The run found three problems in Turnloom, including one where an unattended run ended by telling a reader which commands to type. Nobody was reading. All three are fixed in 0.7.1.

It is a small sample, run once, on one model, by the author of one of the agents. The benchmark page lists every limit, the result of every task for every agent, the token counts of every model call, and the scripts. If you run it and get different numbers, I want to know.

pip install turnloom
turnloom connect
turnloom run --read-only "Explain how this project is laid out"

Turnloom is MIT licensed and works with any OpenAI-compatible model, including local ones through Ollama. If you want to help, the announcement lists where to start.

── more in #ai-agents 4 stories · sorted by recency
── more on @turnloom 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/i-benchmarked-my-cod…] indexed:0 read:2min 2026-11-19 · —