Anything You Can Count, an Agent Can Climb

Anthropic made claude.ai and its desktop app roughly 3x faster in a two-week sprint, cutting 75th-percentile time to a typeable page from 3.1 seconds to 0.55 seconds across more than 3,000 merged changes with no rollback, by having agents optimize against CI-ratcheted benchmarks. Separately, Trail of Bits spent six months before a Miden VM audit having AI agents build an LSP server, decompiler, static analysis engine and Lean executor model; the tooling surfaced over 400 missing type-validation sites, a high-severity prover-supplied remainder flaw exploitable to forge Falcon signatures, and two arithmetic bugs missed by unit tests.

Three teams shipped wildly different things this month — a faster web app, a cryptographic audit, and a new graph algorithm — and what they share is not the model they pointed at the problem. 1. Measurement Used to Be Step Zero. It Is Step One Now. Anthropic made claude.ai and its desktop app about 3x faster https://claude.dev/blog/how-we-made-claude-ai-faster/ in a two-week sprint. At the 75th percentile, time to a typeable page on a fresh load went from 3.1 seconds to 0.55, starting a Claude Code session from 0.8 to 0.3, loading a cloud session from 2.6 to 0.73. More than three thousand changes merged, no customer-facing incident, no rollback. The whole thing ran out of one Slack channel. The interesting part is not the speedup, it is the scaffolding. They hit twelve of thirteen targets by day three, then went looking for more things to count. Every new benchmark had to do two jobs: give the agent a number it could move in the lab, and become a CI ratchet that could only tick downward. Benchmarks that were flaky, or that did not actually correlate with user-visible latency, got thrown out rather than allowed to become a wrong hill. When they proved that deterministic instruction counts tracked wall-clock time, an hour of work cut instructions on two hot paths by 48% and 31% — and wall-clock by 78% and 44%. Their own summary is the line worth stealing: the highest-leverage thing available was finding more things to measure. What the humans supplied was not code. It was ambition pushing threads past targets they had already hit , taste should a table fill in cell by cell, or wait for whole rows? , and direction — keeping each of a hundred and fifty concurrent threads pointed at exactly one benchmark. One 900-line PR was killed with a single line: two milliseconds per send is not worth maintaining a build plugin. Why it matters: - For ICs: if you cannot state your task as a number that moves, you are going to hand-hold the agent through every step. Spend the first hour building the measurement, not the fix. - For leaders: your observability budget is now an engineering-throughput budget. Untelemetered surfaces are the ones agents cannot help you with. - For founders: a metric that only ratchets down is a cheap, durable moat against the regression tax that usually eats velocity gains. 2. The Audit Started Six Months Before Anyone Read the Code Trail of Bits was hired to review the Miden VM, a zero-knowledge machine with its own assembly language and almost no developer tooling. So before the review began, they spent six months having agents build the tooling https://blog.trailofbits.com/2026/09/18/auditing-in-the-age-of-good-enough-ai/ : an LSP server, a decompiler, a static analysis engine over a custom intermediate representation, and a model of the VM executor in Lean. The instruments found the bugs. Static analysis flagged over 400 locations where type validation was missing from publicly reachable procedures, plus one high-severity issue: an unvalidated prover-supplied remainder in a modular reduction routine, exploitable to forge Falcon signatures and drain accounts. Separately, 95 machine-checked correctness proofs covering the library's binary arithmetic surfaced two more bugs the unit tests had missed — a 64-bit rotation that broke on large inputs when the shift was a multiple of 32, and a 256-bit multiply that dropped caller-owned values off the stack. The economics underneath is the part to circle. They say outright that they could not have justified this two years ago: exploratory tooling with unpredictable payoff is impossible to sell a client in advance. Today a failed side project only costs tokens. The decompiler took over a hundred AI-generated commits across months, and most of its eventual value was its internal representation rather than the decompilation it was built for — exactly the kind of bet nobody funds when the input is senior engineer months. The client kept the static analysis engine. Why it matters: - For ICs: the internal tool you have wanted for years — the linter, the visualizer, the domain-specific checker — just moved inside a week's budget. Build it before the project that needs it. - For leaders: speculative tooling now has a payoff profile closer to a lottery ticket than a headcount request. Fund several, kill them fast. - For founders: in high-assurance markets, "we built custom analysis for your stack before we started" is a differentiator competitors cannot fake with a bigger model. 3. Ten Agents Proved a Faster Algorithm. Nobody Has Measured It Running. Ten agents, one shared message board, fifteen hours and 733 messages produced C-HD https://www.vals.ai/blogs/faster-shortest-path-algorithm , an exact shortest-path algorithm for directed graphs with non-negative real weights, carrying a complete Lean proof of correctness and of a runtime bound that beats Dijkstra's within a certified density range. The gate was mechanical: a reproducible Lean build, a comparator confirming the proof establishes the stated theorem using only permitted axioms and passing the kernel, and two peer reviews before anyone could claim success. Failed approaches were logged so other agents would not retry them. Then comes the honest paragraph. The constants in the formal construction are enormous. The implementation was never benchmarked on large graphs. The improvement is polylogarithmic. It is a genuine result and it is not a speedup. That caveat is the whole theme in one artifact. An oracle gives you precisely what it checks and nothing adjacent. Lean was asked to certify an asymptotic bound, so an asymptotic bound is what arrived — the same failure mode Anthropic's team guarded against by discarding benchmarks that did not track wall-clock time. Choosing the metric is now the act with the most leverage and the most risk in the loop, because the agent will optimize it to exhaustion whether or not it was the right one. Why it matters: - For ICs: write down what your benchmark does not capture, before you start climbing it. That note is the only thing standing between you and a beautifully optimized irrelevance. - For leaders: review proposed metrics the way you review architecture. A bad one now gets pursued with unlimited patience. - For founders: verifiers are becoming infrastructure. Proof checkers, differential testers and correlation harnesses are where the defensibility is moving. - The pattern across all three: the model is no longer the constraint, and neither is the code. The constraint is whether you can state success as something a machine can score. The Verdict: Real or Hype? Benchmark-ratcheted agent loops → Real. Three thousand merged changes and zero rollbacks is not a demo, it is an operating model. Agent-built custom tooling ahead of the work → Real. When a failed side project costs tokens instead of engineer-months, the calculus on speculative instruments flips permanently. Formal proof as a substitute for measurement → Real but early. A certified bound with enormous constants and no benchmark is real mathematics and zero milliseconds saved.