{"slug": "i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the", "title": "I benchmarked my coding agent against seven others. It tied, and it was not the cheapest.", "summary": "A developer benchmarked their open-source coding agent, Turnloom, against seven others on 19 tasks using gpt-5.1, finding that six agents solved between 9 and 11 tasks—a statistical tie given the small sample. Turnloom solved 10 of 19 at $0.160 per task, and the developer attributed its higher output-token cost to using Chat Completions rather than OpenAI's Responses API, an untested hypothesis now on the roadmap. The run also surfaced three bugs in Turnloom, all fixed in version 0.7.1.", "body_md": "I maintain [Turnloom](https://github.com/bhaskargurram-ai/turnloom), a small open-source coding agent. Before telling anyone to use it, I wanted to know how it does next to the agents people already use. So I ran eight of them on the same tasks, with the same model, and metered every model call the same way.\n\n`gpt-5.1` for every agent.\n| Agent | Solved | Cost per task | Input tokens per task | Output tokens per task | \n|---|---|---|---|---|\n| OpenCode | 11 of 19 | $0.091 | 110,562 | 6,060 | \n| Goose | 11 of 19 | $0.101 | 180,595 | 5,470 | \n| Terminus 2 | 11 of 19 | $0.162 | 51,228 | 14,056 | \n| **Turnloom** | 10 of 19 | $0.160 | 70,192 | 12,637 | \n| mini-SWE-agent | 10 of 19 | $0.181 | 134,693 | 14,796 | \n| OpenHands SDK | 9 of 19 | $0.186 | 326,500 | 11,477 | \n| Codex CLI | 6 of 19 | $0.156 | 275,188 | 9,483 | \n| Aider | 1 of 19 | $0.061 | 3,626 | 5,745 | \n\nSix agents solved between 9 and 11 tasks. With 19 tasks, that is a tie: the 95% interval for 10 of 19 runs from 32% to 73%. Aider's row is not a fair picture of Aider, because the benchmark runner gives it one message and no way to run commands.\n\n**Sending less input did not make Turnloom cheap.** Turnloom folds old tool output out of the prompt and keeps it in an archive the model can read back. That worked: among the agents that work through a tool loop, only Terminus 2 sent fewer input tokens. But output tokens cost eight times as much as input, and Turnloom wrote about twice as many as the two cheapest agents. Output was $0.126 of its $0.160 per task.\n\n**The likely reason is the API.** Leaving Aider aside, the agents with the fewest output tokens call OpenAI's Responses API, which carries the model's reasoning from one call to the next. Turnloom uses Chat Completions, which does not. I have not tested that explanation yet. It is now on the roadmap.\n\n**A benchmark is a good bug finder.** The run found three problems in Turnloom, including one where an unattended run ended by telling a reader which commands to type. Nobody was reading. All three are fixed in 0.7.1.\n\nIt is a small sample, run once, on one model, by the author of one of the agents. The [benchmark page](https://github.com/bhaskargurram-ai/turnloom/blob/main/benchmarks/README.md) lists every limit, the result of every task for every agent, the token counts of every model call, and the scripts. If you run it and get different numbers, I want to know.\n\n```\npip install turnloom\nturnloom connect\nturnloom run --read-only \"Explain how this project is laid out\"\n```\n\nTurnloom is MIT licensed and works with any OpenAI-compatible model, including local ones through Ollama. If you want to help, the [announcement](https://github.com/bhaskargurram-ai/turnloom/discussions/17) lists where to start.", "url": "https://wpnews.pro/news/i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the", "canonical_source": "https://dev.to/bhaskar_gurram_6bc3033721/i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the-cheapest-5b8i", "published_at": "2026-11-19 16:07:33+00:00", "updated_at": "2026-11-19 16:14:28.701276+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Turnloom", "OpenCode", "Goose", "Terminus 2", "mini-SWE-agent", "OpenHands SDK", "Codex CLI", "Aider"], "also_reported_by": [], "takeaway": "Across eight coding agents on 19 tasks with gpt-5.1, six solved 9-11 tasks (a tie at this sample size), and output tokens cost eight times input, making Turnloom's $0.160 per task driven by output volume rather than input size.", "recommendation": {"advice": "Benchmark coding agents on your own tasks with metered per-call token counts, and report confidence intervals—19 tasks cannot distinguish agents solving 9 to 11.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the", "markdown": "https://wpnews.pro/news/i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the.md", "text": "https://wpnews.pro/news/i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-my-coding-agent-against-seven-others-it-tied-and-it-was-not-the.jsonld"}}