Three pieces of evidence landed this week, and together they say something the individual headlines miss: almost nobody is measuring agent-assisted engineering at the point where it would count.
1. Thirty Percent More Code Went In. Nothing More Came Out the Other End. #
A Harvard working paper built on engineering telemetry from 718 companies and 725,938 workers tracked firms adopting coding agents on top of the assistants they already had. Lines added per engineer-month rose about 30 percent, commits 20 percent, pull requests 23 percent. Then the measure that matters — issues resolved, meaning deployed to production — moved 0.12 on a 3.67 baseline, standard error 0.17. Statistically that is zero, and the interval rules out any gain above roughly 12 percent.
The authors checked whether engineers had simply taken on bigger work: no shift in issue size, no growth in pull request size. What they found was a queue. Time from pull request to merge rose 3.45 days on a seven-day baseline, a 49 percent increase. Formal change requests nearly doubled, from 13 percent of pull requests to 25, and review comments per pull request rose 35 percent. The share of engineers doing any review rose 14 percent while the share writing code didn't move: teams reallocated humans into review and still didn't clear it.
One detail is worth a slide: by March 2026 nearly 80 percent of these firms had adopted an AI review tool, and those tools produced 23 percent of review comments while touching only 10.8 percent of pull requests. The side of the pipeline we automated first is the side that was never the constraint. It is a working paper, not peer-reviewed, but it is the only published measurement of the whole pipeline.
Why it matters:
- For ICs: Your throughput metric is now a vanity metric. The defensible skill is clearing review: small diffs, honest descriptions, the judgment to approve.
- For leaders: If you bought agents and tracked commits, you bought a number. Instrument merge latency, change-request rate and deploys per engineer before renewal.
- For founders: Review capacity is the unserved market. Everyone is selling drafting into a bottleneck that drafting created.
- Automate a stage and the constraint moves to the next human checkpoint — which is where to measure.
2. The Benchmark Number on the Model Card Is Not Your Number #
A leaderboard published this week paired 22 agent harnesses with five models across 62 terminal tasks: 27,280 runs, 83 billion tokens. With the model held fixed, the most expensive harness cost 1.8 to 5 times as much per task as the cheapest — and the ranking reshuffles when you change models. One widely used CLI was cheapest per task on one model and 16th of 22 on another. Paying more bought nothing: cost per task and pass rate correlated between minus 0.39 and plus 0.22.
Two results go straight into practice. Cost tracks steps, not prompt bloat: prompt size, tool count and cache hit rate showed no relationship to cost per pass, while steps per pass correlated at plus 0.51, since every step resends the whole context. And 64.5 percent of spend went to failed runs, 68 to 82 percent of which ended with a confident final answer rather than at the time limit. The agent said it had validated the work; the verifier disagreed.
Hold that next to Microsoft's new in-house coding models, pitched as 98 percent of one frontier model's performance at 40 percent of the cost, averaged over four benchmarks. Probably true on those benchmarks, in whatever harness produced them — and not a prediction about your repo. Pass rates here carried a 95 percent interval of about 10 points either way, so a few points' gap is a tie. The only number describing your situation comes from running the candidates on your own tasks.
Why it matters:
- For ICs: Switching models means re-evaluating the harness. The scaffold you tuned around one model may be the expensive option on the next.
- For leaders: Two-thirds of agent spend is failed runs ending in confident summaries. A stop-and-check before the agent declares victory is the highest-return line item you have.
- For founders: Harness and model are a pair now, not a stack. Selling one while implying the other is interchangeable is a support burden on delay.
3. Ninety Thousand Dollars of Evals, and the Honest Answer Is Still "We Can't Tell" #
The most expensive piece of honest reporting I read this week re-ran a test-technique eval on a new frontier model and spent about $90,000 of tokens doing it. The design: implement a real spec from its RFC — Zstd, then IMAP, then Pandoc — under conditions that each mandate a verification approach, from test-driven development to property testing, fuzzing and proof assistants, scored against held-out tests. Last generation's run found that naming a technique mostly made agents worse, and that specifying nothing was among the best conditions. The rerun found the reverse. It also found the Zstd task saturated — better than 99 percent correct at a dollar a run, against 63 percent at $3.50 a generation earlier — so the benchmark behind the original finding has stopped measuring anything.
The conclusion is the part to keep. With 25 runs per condition on the one task hard enough to discriminate, you can see that conditions differ and still not say which is better, or why. Settling it would take a hundred times the spend. That is the real state of the evidence under nearly every agent-coding best practice in circulation. One gets dispatched in passing: the claim that Rust suits agents because its compiler errors guide them doesn't survive contact with the evals. A reader's case study attached to the piece is sharper than any benchmark: a statechart rewrite, 30,000 lines of production code, 300,000 lines of generated-testing scaffolding, and generated tests caught 20 of 230 defects. Agent reviewers found the other 210.
Why it matters:
- For ICs: Treat agent workflow advice as folklore until someone shows the runs. Asking for the work and verifying it yourself keeps beating elaborate rituals.
- For leaders: Techniques that worked on last quarter's model may be noise on this one, and a saturated internal benchmark is just a pass-rate display. Budget re-measurement as a standing cost, not a project.
- For founders: Nobody can afford meaningful evals of their own setup. Cheap, repo-specific measurement is a product, and the people who need it most already pay for the uncertainty.
The Verdict: Real or Hype? #
The drafting speedup → Real. More code, more commits, more pull requests across 718 firms — that part isn't in dispute. Published benchmark and vendor numbers as a buying signal → Hype. Harness choice alone swings cost fivefold on a fixed model, which makes any number detached from your setup decorative. Review throughput as the binding constraint → Real but unevenly distributed. Teams that moved people and tooling into review are shipping the gains; teams that bought only drafting are funding a longer queue.