The Measuring Sticks Keep Breaking

Anthropic announced that Claude autonomously formalized a complete proof of Fermat's Last Theorem in Lean 4, generating roughly 13.4 million lines of code and 29,500 intermediate theorems in 11 days, completing Freek Wiedijk's list of 100 formalization challenges. Separate research tracking 17,000 real coding agent sessions found Claude Code, Codex, and Cursor choose the same tool in only 42% of cases, and that agents favor grep over semantic tools like LSP on simple code-location tasks.

Three things happened this week that each look like isolated data points until you notice they're pointing in the same direction: what we thought we understood about AI capability keeps turning out to be a lag indicator. 1. A 350-Year-Old Proof Was Formalized in 11 Days. That's Not the Headline. On September 4th, Anthropic announced that Claude had autonomously formalized a complete proof of Fermat's Last Theorem https://www.anthropic.com/research/formalizing-fermats-last-theorem in Lean 4 — producing approximately 13.4 million lines of code, proving 29,500 intermediate theorems along the way, and completing the task in 11 days. The proof follows the Wiles-Taylor-Wiles argument from 1995, the same proof that took Andrew Wiles years to produce and months more for the mathematical community to verify. This completes Freek Wiedijk's list of 100 formalization challenges — a benchmark spanning more than two decades of human mathematical effort. Kevin Buzzard, who had been leading a years-long EPSRC-funded project toward the same goal, published a reaction https://xenaproject.wordpress.com/2026/09/04/flt-anthropic-has-beaten-me-to-it/ that was honest to the point of being remarkable: acknowledging being beaten to it, while noting that the human work — building number theory foundations in Lean's math library — remains relevant for different reasons. The actual headline isn't the speed, though 11 days is striking. It's what the task required: sustained formal reasoning across a massive, interconnected proof structure where every step has to be machine-verifiable and one logical slip breaks everything downstream. This isn't autocomplete. It's something closer to what a research mathematician actually does, executed at a scale no individual could attempt. The open question is whether this was the result of a specific capability investment or a general reasoning leap that will appear across other hard formal domains. The honest answer is we don't fully know yet — and that uncertainty itself is worth sitting with. Why it matters: For ICs: If AI can formalize a 350-year-old proof, it can likely formalize correctness proofs for the critical systems logic you've been hoping someone would eventually write. That use case just became realistic to scope. For leaders: The boundary between "AI-assisted" and "AI-led" research work is no longer clearly drawn. Plan for what happens when an AI produces something your team can't independently verify at the same speed. For founders: Formal verification has always been too expensive to be practical for most teams. That cost curve is shifting, and organizations that treat it as a real option earliest will have a durable quality advantage. Formal mathematics was considered a ceiling for AI capability — a domain requiring creativity and sustained symbolic reasoning. That ceiling just moved. 2. Coding Agents Prefer Grep. We Would Have Predicted LSP. New research tracking 17,000 real coding agent sessions https://armature.tech/blog/which-tools-coding-agents-install across 75 repositories finds that the three major coding agents — Claude Code, Codex, and Cursor — choose the same tool in only 42% of cases. The choices are consistent within each agent but diverge sharply across them: Cursor uses web search in roughly two-thirds of sessions, Codex in 94%, while Claude Code reaches for it only about 30% of the time. Repository language shapes outcomes more than most teams expect — the winning payment SDK in TypeScript projects isn't necessarily the winner in Python ones. A parallel study on grep versus Language Server Protocol https://www.agentconnect.md/blog/grep-beat-lsp-harness/ found something that runs against conventional devtools intuition: on simple code-location tasks, semantic tools like LSP were chosen only 0–6% of the time. On reference-completeness tasks, semantic navigation rose to 45–57% — but the key predictor wasn't whether the codebase used a strongly typed language. It was lexical noise. On messy repos, LSP improved F1 scores by up to 0.246. On clean codebases, it offered no measurable advantage over grep. There's a practical engineering insight buried here. When LSP results included inline source context instead of just file locations, agent pass rates improved from 67% to 83% and follow-up file reads dropped significantly. The model wasn't the limiting factor — what the model received was. Harness design is half the job, and most teams aren't treating it that way. Why it matters: For ICs: When your coding agent makes a poor choice, look at the context it received before blaming the model. The interface shapes the output more than you'd expect. For leaders: Agent evaluations run on clean, idealized codebases don't predict production behavior. Test your setup on the actual state of your repo. For founders: "More tools" is the obvious product pitch. "Better context delivery" is the less obvious one — and probably the harder one to copy. 3. The Benchmarks Couldn't Even Keep Up for a Year Artificial Analysis released Intelligence Index v4.2 https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-2 this week as an interim update ahead of their planned v5 release. The reason: GPQA Diamond, a scientific reasoning benchmark considered rigorous when launched, has been saturated — models have maxed it out. It's been retired from the index. v4.2 replaces it with AA-Briefcase, an agentic knowledge work evaluation built around multi-week projects with thousands of source files. The update also doubles the proportion of private, held-out test sets to 40% of overall weighting — specifically to reduce the ability for labs to optimize against published evaluations. Their framing is measured, but the underlying situation is less so: a major benchmark organization felt compelled to ship an out-of-cycle update because their index was at risk of becoming irrelevant. This is the measurement problem playing out in real time. The frontier is moving fast enough that Claude Fable 5.1 tops the overall index, GPT-6 Astra leads on token efficiency, and the organizations responsible for tracking who's winning are visibly working to keep the scorecard from going stale before the ink dries. It's not that we lack good evaluation infrastructure — it's that the infrastructure is running behind. Why it matters: For ICs: Model selection decisions made six months ago may be based on benchmarks that have since been saturated. Revisit comparisons using current, unsaturated evaluations before committing to a stack. For leaders: "We evaluated the models and made a decision" is a snapshot, not a strategy. Build in a regular cadence to recheck model choices as the evaluation landscape shifts. For founders: If your product positioning depends on leading a specific benchmark, assume that benchmark has a shorter shelf life than your roadmap does. The Verdict: Real or Hype? AI in formal mathematical verification → Real. Formalizing Fermat's Last Theorem is not a demo — it's a working system completing a 20-year benchmark, with concrete implications for rigorous software verification. Coding agent tool selection being counterintuitive → Real. The research is reproducible across 17,000 sessions, and the practical lesson is actionable: clean context beats rich tooling. Benchmark saturation as a chronic problem → Real but early. The pace is genuinely unsettling, and the organizations tasked with measuring AI capability are visibly working to keep up — but the measurement infrastructure is evolving alongside the models, just slightly behind.