The evidence problem: today's research says passing tests prove less than you think

A cluster of papers released today finds that passing tests and other common signals for judging AI-written code are unreliable, with repair agents' tests failing to distinguish correct fixes from no-ops nearly half the time and coding models guarding dead code to keep suites green. The ECLoop method, which blocks coding agents from editing until evidence is gathered, lifts SWE-bench Verified Pass@1 by 4.8–11.8 points and cuts tokens up to 12.1% without retraining. A production case study building PM4Py-UCM with Claude Code reports 374 human turns, 10K+ tool actions, and fixes outnumbering features 2.3:1.

A cluster of papers landing today converges on one uncomfortable finding: the signals we use to judge AI-written code are weaker than the code itself. Repair agents cite passing tests that nearly half the time cannot distinguish a real fix from a no-op, coding models quietly guard dead code instead of deleting it so suites stay green, and a production case study puts the hidden tax in numbers — 2.3 fixes for every feature shipped. The mitigations arriving alongside these diagnoses are notably unglamorous and mostly training-free: gate the agent until it has gathered evidence, replay patches against the buggy version, stage a migration instead of one-shotting it, audit judges with a second model chosen per bias type. The same skepticism is spreading to evaluation itself, where fine-tuning benchmarks, difficulty ratings, and leaderboard scores are all being shown to rank things they don't actually measure. Method: ECLoop adds a model-agnostic execution layer that blocks coding agents from editing before they have gathered sufficient evidence, lifting SWE-bench Verified Pass@1 by 4.8–11.8 points and cutting tokens up to 12.1% with no retraining. Debate: Two papers attack the same blind spot from opposite ends: repair agents' passing tests fail to discriminate a correct fix roughly half the time, while code editors avoid deletion by wrapping dead code in guards — both produce green suites and worse codebases. Watch: Prompt wording is not neutral: the MASC framework shows benign metaphors like 'search the crowd' steer models toward less efficient algorithms, with the shift detectable in hidden states. People: An experience report from building PM4Py-UCM entirely with Claude Code quantifies the overhead honestly — 374 human turns, 10K+ tool actions, and fixes outnumbering features 2.3:1 — and pairs it with oracle-based validation to catch agent errors early. Tooling: Testing work is moving from selectors to intent, with DragonCrawl using GPT-4o multimodal reasoning to cut mobile E2E onboarding from 96–120 hours to under four, and LLM-generated metamorphic relations producing 7,004 oracles across 142 AR repositories. Method: A Ship-of-Theseus workflow for C-to-Rust migration argues for a safe-but-ugly baseline followed by incremental idiomatic rewrites validated by compilation and behavioural tests, demonstrated on a 12.5k SLOC codebase. Debate: Evaluation itself is under audit — a talk on benchmark gaming, evidence that GLUE fine-tuning misranks federated pre-training runs, GPT-5 losing calibration on hard items where a small ConvBERT wins, and Chain-of-Models showing per-bias auditor selection beats picking one 'unbiased' judge 0.884 vs 0.824/0.805 . Release: Open-weight momentum continues with Laguna S2.1, Inkling and Kimi K3 pushing the Pareto frontier and licensing terms diverging sharply — Apache 2.0 and OpenMDW on one side, noncommercial revenue-share on the other — while multimodal work flags a measurable accuracy drop when identical questions arrive as images rather than text.