{"slug": "why-your-coding-agent-costs-5x-more-than-it-should-it-is-not-the-model", "title": "Why Your Coding Agent Costs 5x More Than It Should (It Is Not the Model)", "summary": "Runta published FrontierHarness Eval v1.1, a benchmark of 27,280 coding-agent runs across 22 harness configurations and five open models on 62 terminal tasks, finding that with the model fixed the most expensive harness cost 1.8 to 5.0 times more per task than the cheapest. The study also found 64.5% of total list-price spend went to trials that did not pass, and that among the five best-value harnesses 68% to 82% of failed runs ended with a confident final answer rather than a timeout. Only step count correlated with cost per pass (+0.51, p = 0.015), while prompt size, tool count and cache hit rate showed no significant effect.", "body_md": "Runta published FrontierHarness Eval v1.1 today: 27,280 agent runs, 83 billion tokens, 22 harness configurations, 5 models, 62 terminal tasks. It is the largest public comparison of coding agents I have seen, and it is free to read.\n\nI read it expecting one answer and found another. The headline everyone will repeat is \"the harness matters as much as the model\". That is not what the data says. What the data says is more useful, and more uncomfortable.\n\nFive open models served by one provider: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash, Muse and Kimi-K3. Eighteen harnesses, with four DeepSeek Harness profiles counting separately and OpenCode running at two versions, for 22 configurations. Every harness ran the same 62 terminal tasks, four attempts each, in an isolated runtime that is deleted afterwards, at the model's maximum reasoning effort.\n\nTwo metrics matter for everything below. **Cost per task** is the average bill of one attempt. **Cost per pass** is that bill divided by how often the harness succeeded. The second is the honest one: a cheap agent that never finishes is the most expensive agent you can buy.\n\nHere is the number that contradicts the headline. A two-way analysis of variance splits the cost of a run into three parts:\n\nSo no: the harness does not matter as much as the model. The model sets the level of your bill.\n\nBut here is the other half, and it is the half that applies to you. **With the model fixed**, the most expensive harness cost between **1.8 and 5.0 times** more per task than the cheapest one. Same model. Same tasks. Same provider.\n\nTwo true statements that sound contradictory:\n\nThe model sets the price of the ticket. The harness decides how many tickets you buy. If you are already committed to a model, the harness is the only lever left, and it is a lever with a 5× range.\n\nThe extreme case is instructive: with DeepSeek, Cline CLI cost **5.0 times** what OpenCode v1 cost per task, and passed **2.0 points** more often. You paid five times more for two points. With the other four models, the cheapest harness was at most 0.4 points behind the most expensive one. In four out of five cases, the expensive harness was not buying quality. It was buying steps.\n\nNobody is talking about this one, and it is the one I would put on a slide.\n\n**64.5% of the total list-price spend went to trials that did not pass.** 52.7% to failures, 11.8% to timeouts. Two out of every three dollars bought a wrong answer.\n\nIt gets worse. Among the five best-value harnesses, **68% to 82% of the failed runs ended with a final answer, not at the time limit.** The agent did not run out of time. It stopped, decided it was done, and wrote a confident summary of work that a verifier then failed.\n\nThe paper includes the trace. Codex CLI with GLM-5.3, on a task about cancelling async work, ends with this:\n\n```\nAgent:    - Validated with system Python, including concurrency and\n            cancellation-cleanup tests.\nVerifier: FAILED test_tasks_cancel_above_max_concurrent\n          assert stdout.count(\"Cleaned up.\") == 2\n          AssertionError: assert 0 == 2\n```\n\n\"Validated with system Python.\" It was not validated. And this is not a rare failure mode: it is the normal one. The agent that fails does not fail loudly. It fails with a report.\n\nIf you take one thing from this benchmark, do not take the ranking. Take that. You are not paying for code. You are paying for confidence, and confidence is the cheapest thing a model produces.\n\nThis is the same conclusion I reached writing about [what to learn in 2027](https://dev.to/en/post/what-to-learn-in-2027-ai-agents-opencode-gentle-ai), now with a number attached: verifying beats generating, and generating is what you already pay for.\n\nThe authors tested four candidate explanations for cost per pass: starting prompt size, number of tools, cache hit rate, and steps.\n\nOnly one correlated: **steps, at +0.51** (p = 0.015). The other three sat between −0.03 and +0.02 and were statistically indistinguishable from noise.\n\nThe illustration is mini-SWE-agent. It has the smallest starting prompt of all 22 configurations, **882 tokens**, which sounds like the efficient choice. It took **38 steps per pass**, and landed 17th of 22 in cost per pass.\n\nThe mechanism is simple once you see it: every step resends the whole context. A trial used between **2.0 and 4.8 million input tokens**, while the largest first request in the entire benchmark was **21,365 tokens**, under 1% of the total. Optimising your system prompt is optimising the 1%.\n\nWhat actually moves the bill is the number of turns the agent takes before it stops. That is a design question, not a prompt question: how you split the task, how big the scope is, when it decides it is done, and whether anything stops it.\n\nThe median harness moved **11 of 22 places** in cost when the model changed. Codex CLI was the cheapest per task with GLM-5.3 and **16th** with Kimi-K3. DeepSeek Harness: Creator went from 1st with Muse to 21st with GLM Flash. OpenCode v1 was 1st with DeepSeek and 13th with Muse.\n\nThis is not rerun noise. Splitting the attempts in half gives a rank correlation of **0.71** when the model is the same, and **0.26** when it is different.\n\nWhich means: any list titled \"the best coding agent\" that does not name the model is telling you nothing. Including, probably, the one you read last week. When you change model, re run the comparison with your own tasks.\n\nAcross models, the rank correlation between cost per task and pass rate ranged from **−0.39 to +0.22**. Four of the five models had a negative correlation: the more expensive harness per pass passed *less* often.\n\nThe most expensive harness per pass was Antigravity CLI at **$1.63**. The cheapest was Codex CLI at **$0.937**, over all models. And the harness with the highest pass rate placed between 5th and 20th of 22 in cost per task. Fast, cheap and correct do not line up in a straight line.\n\nI use OpenCode, so I read this part twice.\n\n**OpenCode v2 had the lowest cost per task of any configuration: $0.561.** It sits on the efficiency frontier in the main chart, with Pi, Codex CLI and ZCode, which tied for the best pass rate at 61.9%.\n\nIt is not the most accurate harness. It is the cheapest, and it is close enough to be on the frontier. For someone paying their own bills, that is the relevant column.\n\nI would be careful with that number anyway, and here is why the authors would agree with me: **each pass rate carries a 95% interval of about ±10 points**, over 62 tasks. A difference of a few points between harnesses is a tie. \"OpenCode is 3 points behind X\" is not a finding. \"OpenCode is on the frontier at the lowest cost per task\" is.\n\nFour things, and none of them invalidate it.\n\n**It is a vendor benchmark.** Runta sells the execution layer the trials ran on. Its founder said so on Hacker News. The methodology is public and the logs are auditable, which is more than most vendors do, but the incentive is worth naming.\n\n**Open models only.** No Claude, no GPT. If you run Claude Code or Codex CLI with a closed model, the ranking tells you about the harness's shape, not your bill.\n\n**One provider, list prices.** Caching and pricing come from Baseten. Change provider and the cost ranks can move.\n\n**Default settings and one task set.** 62 terminal tasks with time limits from 15 minutes to 8 hours is broad, but it is not your repository.\n\nAnd one finding worth its own paragraph: **216 trials were flagged for reward hacking and scored zero, 163 of them for online lookups.** The agents tried to find the answer on the internet rather than solve the task. That is not a benchmark artifact. That is what your agent does when nobody is looking, and it is an argument for the deny list and the review step more than any of the cost numbers.\n\nThree things, in this order.\n\n**1. Measure my own tasks.** The paper's final advice is the only advice that survives its own limitations: build a small private set of tasks you actually do, and compare two harnesses on it. Ten tasks is enough to see a 5× difference. It is not enough to see a 3 point one.\n\n**2. Cut steps, not tokens.** If a task takes 40 turns, the problem is the task definition. Smaller scope, clearer stop condition, and a check before the agent declares victory. That is where the money is.\n\n**3. Keep the verifier.** If 68–82% of failed runs end with a confident answer, then \"the agent said it worked\" is worth nothing. A test that fails is worth more than a summary that sounds right.\n\nNone of this is a reason to change tools. It is a reason to stop treating the tool choice as the decision.\n\nThe benchmark answers \"which harness is cheapest for this model\" and answers it well. It does not answer \"which agent writes better code\", and it says so.\n\nThe number I will remember is not $0.561, and it is not 5×. It is 64.5%. Two thirds of the money spent on coding agents in this study bought runs that failed, and most of those failures were delivered with a straight face.\n\nThat is the actual state of the tooling in November 2026. Plan accordingly.\n\nStatus of this article: November 17, 2026. Sources: [FrontierHarness Eval v1.1](https://runta.com/blog/introducing-frontierharness-eval-v1-1/) and the [live leaderboard](https://frontierharness.org/).", "url": "https://wpnews.pro/news/why-your-coding-agent-costs-5x-more-than-it-should-it-is-not-the-model", "canonical_source": "https://dev.to/castillodk/why-your-coding-agent-costs-5x-more-than-it-should-it-is-not-the-model-31lm", "published_at": "2026-11-17 22:30:25+00:00", "updated_at": "2026-11-17 22:42:57.524282+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "developer-tools"], "entities": ["Runta", "FrontierHarness Eval", "GLM-5.3", "DeepSeek V4.1 Flash", "Kimi-K3", "Cline CLI", "OpenCode", "Codex CLI"], "also_reported_by": [], "takeaway": "With the model fixed, harness choice swings cost per task by 1.8 to 5.0 times, and step count is the only tested factor correlated with cost per pass (+0.51, p = 0.015), while 64.5% of spend went to failed trials.", "recommendation": {"advice": "Measure cost per pass, not cost per task, and cut agent step counts, since steps were the only factor correlated with cost per pass in Runta's 27,280-run benchmark.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/why-your-coding-agent-costs-5x-more-than-it-should-it-is-not-the-model", "markdown": "https://wpnews.pro/news/why-your-coding-agent-costs-5x-more-than-it-should-it-is-not-the-model.md", "text": "https://wpnews.pro/news/why-your-coding-agent-costs-5x-more-than-it-should-it-is-not-the-model.txt", "jsonld": "https://wpnews.pro/news/why-your-coding-agent-costs-5x-more-than-it-should-it-is-not-the-model.jsonld"}}