{"slug": "things-i-believed-about-coding-agents", "title": "Things I believed about coding agents", "summary": "An analysis of 51,000 coding-agent sessions from 190 engineers found that top-quartile users get 7.8 minutes of agent work per minute of their own time versus about half a minute for the bottom quartile — a 15x return on identical tools and models — despite spending the same five to six hands-on minutes and sending the same median of three prompts per session. The ConductorScore operator behind the study reports that prompt effort correlates zero to slightly negative with score, and that the difference lies in what users hand over: the top quartile assigns tasks rather than asking questions. The author also retracts an earlier token-cost-per-commit metric after finding it counted only agent-run commits, not those typed in a terminal.", "body_md": "Engineers in the top quartile of coding-agent users spend five hands-on minutes per session. Engineers in the bottom quartile spend six. The top quartile engineer gets 7.8 minutes of agent work per minute of its own. The bottom quartile engineer gets half a minute. Same tools, same models, same month, **fifteen times the return** on the same effort.\n\nI run [ConductorScore](https://conductorscore.com), which scores engineers on how well they work with coding agents by reading their local Claude Code, Codex and Cursor transcripts. That gives me something most people arguing about agents don't have: the logs. 321 engineers submitted scores between May and October 2026. After dropping anyone who tripped an integrity check, 190 remain, with 51,000 sessions and about 640 billion tokens between them. They are self-selected, heavy, curious users, and I'll come back to what that means.\n\nI went in with seven beliefs about what separates the engineers who get a lot from these tools from the engineers who get a little. Most didn't survive. Two of them I had already written up, with charts, before I checked what my own numbers counted. Here are the seven, in the order I lost them.\n\n## 1. I thought the best agent users were better prompters. In fact, they ask for more and type less\n\nThe folklore says the engineers who get the most out of an agent write better prompts: more context, more care, more thinking between turns. I believed it. It is flattering to the people it describes, and I am one of them.\n\nThe transcripts don't show it. Prompt text never leaves the engineer's machine, so I can't measure prompt length. I can measure effort around the prompt, and there is none to find.\n\nEngineers in every score quartile send a median of three substantive prompts per session. They spend a median of five to six hands-on minutes per session, top and bottom alike. Minutes of human time per prompt: 4.4 at the top, 5.6 at the bottom. Pool each engineer's whole history and the gap widens the wrong way: nine minutes per prompt at the top, eighteen at the bottom. The correlation between time-per-prompt and score is zero to slightly negative.\n\nWhat differs is what comes back.\n\n*Figure 1. Median engineer in the bottom vs. top score quartile, 41 and 48 engineers. The first three measures are not part of the score. The fourth is its largest component, so read that panel as the definition of \"top quartile\" rather than a finding.*\n\nFor the same prompt, the top quartile gets 116 agent messages; the bottom gets 75. For the same hands-on minute, 7.8 agent minutes against half a minute. The difference is not in the asking. It is in what is asked for. The bottom quartile hands the agent a question. The top quartile hands it a task and walks away. Call it ambition: the top quartile expects more from the model, and the model delivers.\n\nAndrej Karpathy made the same point from the other direction last December, when he [wrote](https://x.com/karpathy/status/2004607146781278521) that he had never felt this much behind as a programmer, that he could be 10X more powerful if he just properly strung together what had become available over the past year, and that the gap was a skill issue on his side. He wasn't complaining about the model. He was complaining about how little he was asking of it.\n\nSo the lever is not the prompt. It is what you hand over with it. The rest of this piece is about that.\n\n## 2. I thought the gap was skill. It's trust, and it shows up in who presses commit\n\nTwo weeks ago I drafted a post around one number. The top quarter of engineers pays about $3 of tokens per commit; the bottom quarter pays about $21. Same tools, same models, same month, seven times the cost for the same unit of output. It is a good number. I was proud of it.\n\nThen I went back to check what it counts. It counts commits the *agent* ran, through a shell tool. A `git commit` typed by the engineer in their own terminal never appears in a transcript. The metric was never measuring efficiency. It was measuring who presses commit, and once you know that, the gap reads completely differently.\n\n*Figure 2. Median engineer in the bottom vs. top score quartile, 25 and 37 engineers with ten or more hands-on sessions. \"Agent commits\" are git commits the agent ran in a shell tool; the engineer's own commits never appear in a transcript, so 5,556 is agent-edited lines per rare agent commit, not commit size.*\n\nThe bottom quartile does not ship less code. Its engineers take the wheel back before the commit. The agent edits thousands of lines, and the human commits them by hand, outside the transcript, where no metric sees them. The 5,556 figure is not the size of their commits. It is the number of agent-edited lines per time they let the agent commit, and they rarely do.\n\nI checked the alternative story, that trusted agents commit in smaller chunks. Within a session, lines per commit *rise* with the number of commits, from 39 in one-commit sessions to 115 in sessions with nine or more. The agent isn't chunking. The human is gating.\n\nStrip the commit habit out and what survives of the efficiency claim? Less than I'd like. The top quartile pays *more* per thousand lines edited, $5.91 against $3.46, and 2.7 times more per hands-on hour, $33 against $12. It gets 1.8 times the lines per hands-on hour in return. And it gets about ninety times the agent commits per hands-on hour, which is the point: the money buys a finished loop, not a faster typist.\n\nCost per commit turns out to be a trust gauge. I'd still put it on a dashboard, but labelled honestly: it tells you how far your engineers let the agent go, not how efficient they are.\n\n## 3. I thought planning was overhead that paid off on big tasks. It's what lets the agent run at all\n\nEvery major coding agent has a plan mode. Most engineers skip it; only 19% of sessions in this dataset start with a plan. I used to think that was rational. Planning felt like a tax you paid on large tasks and skipped on small ones, and \"planning works\" studies mostly compare people who plan with people who don't, which tells you that organised people are organised.\n\nSo I did the within-person version. ConductorScore marks a session as planned when plan mode, a planning skill or a written task list appears before the first edit. 147 engineers plan sometimes but not always, with at least thirty hands-on sessions each. Same engineer, same tools, same codebase, planned days versus unplanned days.\n\nThe result that made me sit up is not the productivity ratio. It is this: the median unplanned session has **zero** unattended agent minutes. The agent answers and stops. The median planned session has 29. Sessions with any unattended time at all: 37% unplanned, 62% planned. Per-engineer leverage, agent minutes per hands-on minute: 0.0 unplanned, 1.4 planned, and higher for 84% of the 147.\n\nPlanning isn't one good habit among several. It is the mechanism behind \"let it run.\" Without a plan, the agent mostly never runs.\n\nThe productivity numbers follow. Planned sessions cost 2.7 times the human time, 12 minutes against 4.5, and return 36 times the lines, 894 against 25. Per hands-on minute, the median engineer gets 5.3 times more. 133 of 140 engineers edit more per minute when they plan first. For 41 of them, the typical unplanned session edits nothing at all.\n\n*Figure 3. Same engineers, planned vs. unplanned sessions. Top row: one engineer, 241 sessions, 27% of them planned. Bottom row: all 147 engineers who plan sometimes, median of each engineer's per-session medians.*\n\nThe top row is one engineer, 241 sessions, planning 27% of the time. Unplanned: nine hands-on minutes buy eleven agent minutes and 230 lines, and three sessions in four edit anything. Planned: eleven and a half hands-on minutes buy 55 agent minutes and 762 lines, nine sessions in ten edit something, and the longest unattended run goes from eight minutes to 36. Two and a half extra minutes of their time, five times the agent time.\n\nThe obvious objection stands. Planned sessions run 7.6 times longer on the wall clock, which means engineers plan when the task is big. The data supports \"planning helps.\" It cannot support \"planning causes.\" I'd still bet on it, for one reason: in the whole scoring model, \"coding without a plan\" is the only anti-pattern with a sizeable negative correlation to score. Reverts, retries, repeated prompts, rage quits: barely register. No plan: registers.\n\nAnd the practice is rare even at the top. Top-quartile engineers plan 31% of their sessions. Bottom-quartile, 9%. Nobody is near 100%.\n\n## 4. I thought you needed the whole system. It's a staircase, and the first step is the big one\n\nBy this point I had a theory, and I liked it. The engineers who get the most from agents run a *system*: plan first, dispatch subagents in parallel, let the agent commit small and often, write tests first, use an orchestrator like superpowers that enforces the sequence, and grant permissions broadly enough that the agent never waits for a click. Do all of it and productivity jumps. Do one or two and nothing much happens.\n\nIt is a satisfying theory because it explains why so many people try an agent and shrug. I tested what I could.\n\nFirst, the practices do not travel together. I flagged each engineer for six measurable habits and checked how often they co-occur. The strongest pairing outside a definitional one is 0.27. Planning and subagent use: 0.03. Planning and orchestrator use: 0.05. People adopt these piecemeal, one at a time, in no particular order. There is no bundle in the wild.\n\nSecond, the payoff is a staircase, not a cliff.\n\n*Figure 4. 143 engineers with ten or more hands-on sessions, grouped by how many of six measurable practices they use: planning in at least 25% of sessions, a subagent in at least 10%, agent commits at 0.5 per session or more, 500 or fewer lines per agent commit, below-median approval friction, any orchestrator skill. Medians per tier.*\n\nLeverage climbs about half a turn per practice. Lines per hour climb through the first two practices and then flatten. Agent commits per hour climb sixty-fold from the bottom tier to the top. Nothing in the table says \"one or two do nothing.\" The first step pays the most.\n\nWhich steps? Holding the number of other practices fixed, planning multiplies leverage by 2.4 and lines per hour by 1.7. Letting the agent commit multiplies leverage by 2.0. Using an orchestrator, 1.4. Dispatching subagents, 1.1.\n\n*Figure 5. For each practice, the median outcome of engineers who use it divided by the median of those who don't, matched on how many of the other practices each group uses. Grey rows are proxies I don't trust: \"small commits\" measured as lines per commit penalises engineers who edit a lot, and the approval-wait proxy is contaminated by how long commands take to run. The last two rows are on my list and not in the data.*\n\nTwo of the six come out below one once you hold the others fixed, and I don't believe either. \"Small commits,\" measured as lines per agent commit, mostly flags engineers who edit less code in the first place. \"Low approval friction\" is the permission-mode proxy I'll come back to, and it points in opposite directions on the two outcomes, which is what a broken proxy looks like.\n\nOrchestrators deserve a closer look because the result is odd. The 31 engineers who use one, superpowers and its relatives, get higher leverage (2.2 against 1.7), commit through the agent three times as often per session, and have the agent edit *fewer* lines per hands-on hour, 2,944 against 3,729. The transcripts can't see hand edits, so I can't tell you what those engineers do in the gap. I can tell you what I do. Since I started working inside superpowers I edit far less by hand, and the engineers I've watched who keep hand-editing alongside the agent see the smallest gains. The data is consistent with that reading. It cannot confirm it.\n\nTwo items on my list I could not test at all. Test-driven development: fifteen engineers invoked a TDD or testing skill, and they look excellent, 2.6 against 1.7 on leverage, but they are self-selected power users, skill use feeds the score, and shell output is not uploaded, so I cannot see whether anyone actually ran tests. Permission mode: my best proxy, how long the agent waits after a tool call, is contaminated by how long the tool itself takes, and the clean version of it correlates with nothing. If you have transcripts that record permission mode directly, I would like to see them.\n\nSo the theory survives, bent. I expected a bundle and found a staircase. The bundle may describe the ceiling; the top tier does commit through the agent sixty times more often than the bottom. But nobody has to climb it at once, and the first step, a plan, is the biggest one.\n\n## 5. I thought cache reads were the cheap part\n\nFirst, a correction to something I wrote earlier. I had the dataset at 90% cache reads and 7% fresh input. Codex reports cached input inside its input total, so I was double-counting 47 billion tokens. The real split is about 97% cache reads, 2% cache writes, under 1% fresh input, and 0.3% output. Of 640 billion tokens, the model wrote two billion.\n\nI had filed cache reads under \"cheap.\" Anthropic bills them at five to ten cents on the dollar against fresh input, and most cost discussions treat them as a rounding error. Then I priced the dataset model by model at list rates.\n\n*Figure 6. Claude sessions only, 586 billion tokens, priced per model at Anthropic list rates with cache reads at 5 to 10% of base input. Share of tokens above, share of cost below.*\n\nCache reads are 64% of Claude spend. Output, the thing everyone budgets for, is 13%. Cache writes, the first load of each new context, are most of the rest at 23%. The cheapest token is the biggest line item, because there are three hundred of them for every token the model writes.\n\nThat ratio, roughly 300 to 1, is why every blended-price calculator I can find is wrong for agents. The standard formula averages input and output at a 3:1 ratio: (3 × input price + output price) ÷ 4. Some use 1:1. Both were built for chatbots. Apply the 3:1 blend for Opus 5.5 to this dataset's tokens and you get $5.1 million. Bill every input token at the base rate, which is what a dashboard does when it does not know about caching, and you get $2.4 million. The actual list-price figure is about $300,000. The formulas miss by eight to sixteen times, and they miss in the one place that matters.\n\nFor what it's worth, the numbers hold up against outside sources. Requesty measured a 92% cache-hit rate in Claude Code. Anthropic says engineers should expect $150 to $250 a month; the median engineer here runs $118 per active month at list, the 75th percentile $385.\n\nThree things follow. The effective price of an agent is cache-read rate × context size × tool calls per prompt, and nothing else matters much. An engineer who keeps a 150,000-token context alive across fifty tool calls has just read 7.5 million tokens from one prompt. Short CLAUDE.md files, fresh sessions and small contexts are cost controls whether the engineer knows it or not. And the most consequential price change of the year for agent users was not any output price: it was Opus 5.5 cutting cache reads from ten to five cents on the dollar, which almost nobody noticed because almost nobody's cost model has a line for it.\n\n## 6. I thought the best engineers used the cheap model when it suited\n\nThe standard advice is \"use the cheaper model, it's good enough for most things.\" I believed a version of it: that the best engineers route by task, big model for hard problems, small model for the rest.\n\nNot according to the output tokens. I looked at which model family wrote the output, month by month, because output is the honest measure of work done. Opus wrote 59% to 78% of it in every month but August. Sonnet sat between 15% and 26%. OpenAI models, mostly through Codex, took 5% to 18%. August is the exception, Opus at 41% and Sonnet at 38%, and I wouldn't read much into a month with a smaller, lumpier set of heavy users.\n\nThe October surprise is Haiku. It went from under 1% every month to 14% of output in October, which lines up with the Haiku 5.5 release. It is the first time a small model has taken real share in this dataset.\n\n*Figure 7. Share of output tokens by model family, per month. Output tokens are what the model writes, not context re-read. Families under 0.5% omitted.*\n\n*Figure 8. Of 321 scored engineers, the number with sessions from each agent. One engineer can appear under several.*\n\nAnd they don't pick one agent either. 28% of engineers, 91 of 321, have sessions from two or more coding agents: 290 use Claude Code, 100 use Codex, 29 use Cursor, with heavy overlap. The \"pick your agent\" framing that vendors and procurement teams both love does not describe what engineers do.\n\nHere is where I still disagree with my own data. I think model differentiation is coming, and I think the October Haiku number is the first sign of it. As teams get cost-sensitive and models get specialised, the choice of model will stop being an engineer's habit and become a decision made by software in front of the call. There is a class of model built for exactly that job. Jev, from TypeSafe AI, released into early access in September on the back of a $40 million seed round, is a discriminative model: it takes a context and typed questions and returns probabilities, at a fraction of a cent per thousand tokens and an order of magnitude faster than a frontier model. Its makers describe it as a classification, routing and scoring engine, not a generator. Nobody has shown me a Jev-class model routing between coding models yet. I expect to see it within a year.\n\nUntil then, the engineers who measure their own output have made their choice with their own quota, and the choice is Opus.\n\n## 7. I thought leverage was climbing because mine was\n\nMy own ratio of agent minutes to hands-on minutes went from 2.0 in May to 2.5 in June to 6.0 in September and 4.8 in October. I assumed everyone's was rising. The tools were getting better; the habits were spreading. Of course it was rising.\n\nThe pooled number agreed with me. Add up every engineer's agent minutes and divide by every engineer's hands-on minutes and you get 3.4, with a nice bump through the summer. That is where I would have stopped if I had wanted a slide.\n\n*Figure 9. Agent minutes per hands-on minute, monthly: all engineers pooled, the median engineer, and me. The median line counts engineers with 30 or more hands-on minutes that month; mine is drawn for the months where I had that many. Time accounting changed on October 24; the decline holds with later uploads removed.*\n\nTake each engineer's own ratio each month, then the median across engineers, and the line goes the other way. In May, June and July it was about 2.0. Since August it has been 1.6. The pooled number is dominated by a handful of very heavy users whose agents run for hours. I am one of them, sitting between the 75th and 90th percentiles. The median engineer is not that person, and the median engineer is getting slightly less autonomy from their agent than six months ago.\n\nI checked whether the time-accounting change we shipped on October 24 caused it. Remove every upload after that date and the decline is the same.\n\nI leaned toward a composition story: the user base nearly tripled from July to September, and newer cohorts are less practised. That is about half true. Engineers who joined by July ran 2.0, 2.0, 2.1 through the spring, then 2.0, 1.6, 1.8 from August. Engineers who joined from August on run 1.3 to 1.6. The newcomers drag the median down, yes. But the veterans dipped in September too, and have only partly recovered. The other half of the story may be that as agents got better, people started using them for quick questions they used to google, which adds sub-minute sessions with no leverage at all. A quarter of all sessions are over in under a minute.\n\nThe lesson is embarrassing and simple. I generalised from my own dashboard. Don't.\n\n## What I'd do with this\n\n**If you're the engineer.** Start every real session with a plan, even two lines; it is the single practice with the largest effect, and it is the one that makes the agent run on its own. Let the agent commit, and read the diff after rather than before; the commit is where trust shows, and the engineers who withhold it get the least back. Add one practice at a time; each step pays, and you don't have to climb the whole staircase in a day. Pick the big model and don't apologise; the people producing the most output already have.\n\n**If you're the CTO at a Series A or B.** Measure the share of sessions in which the agent committed. It is cheap, it is non-invasive, and it is the most honest trust gauge in the logs. Report cost and lines per hands-on hour together, never cost per commit alone. Report the median engineer, split by cohort, and expect the median to fall when a new cohort joins; if it hasn't recovered in a quarter, the tool is not the problem. Price agents by cache reads × context size, and throw away any calculator that asks for an input-to-output ratio. Don't standardise on one agent yet; a quarter of your best people will run two anyway, so buy observability across agents instead of exclusivity with one.\n\n## Where I could be wrong\n\nThese are engineers who chose to score themselves on agent proficiency. They skew heavy and curious; the population-wide numbers are surely worse on every measure here. Unattended time and skill use feed the score, so parts of sections 1 and 4 are partly definitional; read them as correlations. Planning is confounded by task size and I have not separated the two. Agent-edited lines are not merged lines, and nothing here measures code quality. Six months of data on tools that change monthly is a short baseline. And two of my seven claims were already wrong once.\n\n## Re-read your canonical numbers\n\nI started with seven beliefs and kept two and a half. The two I had already written up were the ones I was most confident about, because I had numbers for them. The numbers were fine. I had misread what they counted.\n\nEvery engineering team now has a set of rules about coding agents. Use the cheap model. Watch for reverts. Plan on big tasks only. Budget in tokens. Some of them are right. The transcripts to check them are already on every engineer's disk.\n\n**Which rule about coding agents does your team follow that nobody has checked against the logs?**\n\n*Method: 321 engineers scored on ConductorScore between May and October 2026; 190 after integrity and data-quality checks, used for all quartile comparisons. 51,000 sessions, about 640 billion tokens. The author's own account is excluded from every figure except section 7, where it is labelled. Token costs are at Anthropic list prices by model as of October 30, 2026; nearly all of this usage was on subscriptions and was not billed at list. \"Hands-on minutes\" is time the engineer was actively reading, typing or approving; \"unattended\" is time the agent ran with no human input. Within-person planning comparisons use engineers with at least 30 hands-on sessions and a planning rate between 5% and 95%. The practices analysis uses 143 engineers with at least 10 hands-on sessions. Quartile tables in section 2 use the same 143 where the agent-commit rows are concerned; the cost rows use all 190.*\n\n*Sources: Anthropic, [Pricing](https://platform.claude.com/docs/en/about-claude/pricing) and [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) documentation. Modelgrep, [Blended cost](https://modelgrep.com/glossary/blended-cost). Requesty, [The coding agent economy](https://www.requesty.ai/coding-agent-economy) (May 2026). Andrej Karpathy, [post on X](https://x.com/karpathy/status/2004607146781278521), December 26, 2025. Wikipedia, [Jev (AI model)](<https://en.wikipedia.org/wiki/Jev_(AI_model)>). ConductorScore, [conductorscore.com](https://conductorscore.com).*", "url": "https://wpnews.pro/news/things-i-believed-about-coding-agents", "canonical_source": "https://alonbochman.substack.com/p/seven-things-i-believed-about-coding", "published_at": "2026-11-03 23:38:35+00:00", "updated_at": "2026-11-03 23:50:10.248759+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["ConductorScore", "Claude Code", "Codex", "Cursor", "Andrej Karpathy"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/things-i-believed-about-coding-agents", "markdown": "https://wpnews.pro/news/things-i-believed-about-coding-agents.md", "text": "https://wpnews.pro/news/things-i-believed-about-coding-agents.txt", "jsonld": "https://wpnews.pro/news/things-i-believed-about-coding-agents.jsonld"}}