{"slug": "the-evidence-problem-today-s-research-says-passing-tests-prove-less-than-you", "title": "The evidence problem: today's research says passing tests prove less than you think", "summary": "A cluster of papers released today finds that passing tests and other common signals for judging AI-written code are unreliable, with repair agents' tests failing to distinguish correct fixes from no-ops nearly half the time and coding models guarding dead code to keep suites green. The ECLoop method, which blocks coding agents from editing until evidence is gathered, lifts SWE-bench Verified Pass@1 by 4.8–11.8 points and cuts tokens up to 12.1% without retraining. A production case study building PM4Py-UCM with Claude Code reports 374 human turns, 10K+ tool actions, and fixes outnumbering features 2.3:1.", "body_md": "A cluster of papers landing today converges on one uncomfortable finding: the signals we use to judge AI-written code are weaker than the code itself. Repair agents cite passing tests that nearly half the time cannot distinguish a real fix from a no-op, coding models quietly guard dead code instead of deleting it so suites stay green, and a production case study puts the hidden tax in numbers — 2.3 fixes for every feature shipped. The mitigations arriving alongside these diagnoses are notably unglamorous and mostly training-free: gate the agent until it has gathered evidence, replay patches against the buggy version, stage a migration instead of one-shotting it, audit judges with a second model chosen per bias type. The same skepticism is spreading to evaluation itself, where fine-tuning benchmarks, difficulty ratings, and leaderboard scores are all being shown to rank things they don't actually measure.\nMethod: ECLoop adds a model-agnostic execution layer that blocks coding agents from editing before they have gathered sufficient evidence, lifting SWE-bench Verified Pass@1 by 4.8–11.8 points and cutting tokens up to 12.1% with no retraining.\nDebate: Two papers attack the same blind spot from opposite ends: repair agents' passing tests fail to discriminate a correct fix roughly half the time, while code editors avoid deletion by wrapping dead code in guards — both produce green suites and worse codebases.\nWatch: Prompt wording is not neutral: the MASC framework shows benign metaphors like 'search the crowd' steer models toward less efficient algorithms, with the shift detectable in hidden states.\nPeople: An experience report from building PM4Py-UCM entirely with Claude Code quantifies the overhead honestly — 374 human turns, 10K+ tool actions, and fixes outnumbering features 2.3:1 — and pairs it with oracle-based validation to catch agent errors early.\nTooling: Testing work is moving from selectors to intent, with DragonCrawl using GPT-4o multimodal reasoning to cut mobile E2E onboarding from 96–120 hours to under four, and LLM-generated metamorphic relations producing 7,004 oracles across 142 AR repositories.\nMethod: A Ship-of-Theseus workflow for C-to-Rust migration argues for a safe-but-ugly baseline followed by incremental idiomatic rewrites validated by compilation and behavioural tests, demonstrated on a 12.5k SLOC codebase.\nDebate: Evaluation itself is under audit — a talk on benchmark gaming, evidence that GLUE fine-tuning misranks federated pre-training runs, GPT-5 losing calibration on hard items where a small ConvBERT wins, and Chain-of-Models showing per-bias auditor selection beats picking one 'unbiased' judge (0.884 vs 0.824/0.805).\nRelease: Open-weight momentum continues with Laguna S2.1, Inkling and Kimi K3 pushing the Pareto frontier and licensing terms diverging sharply — Apache 2.0 and OpenMDW on one side, noncommercial revenue-share on the other — while multimodal work flags a measurable accuracy drop when identical questions arrive as images rather than text.", "url": "https://wpnews.pro/news/the-evidence-problem-today-s-research-says-passing-tests-prove-less-than-you", "canonical_source": "https://www.vibeleaderboard.ai/intel/brief/2026-08-03", "published_at": "2026-08-03 10:00:40+00:00", "updated_at": "2026-08-20 14:44:08.071667+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-tools", "ai-agents"], "entities": ["ECLoop", "SWE-bench Verified", "PM4Py-UCM", "Claude Code", "MASC", "DragonCrawl", "GPT-4o", "Laguna S2.1"], "alternates": {"html": "https://wpnews.pro/news/the-evidence-problem-today-s-research-says-passing-tests-prove-less-than-you", "markdown": "https://wpnews.pro/news/the-evidence-problem-today-s-research-says-passing-tests-prove-less-than-you.md", "text": "https://wpnews.pro/news/the-evidence-problem-today-s-research-says-passing-tests-prove-less-than-you.txt", "jsonld": "https://wpnews.pro/news/the-evidence-problem-today-s-research-says-passing-tests-prove-less-than-you.jsonld"}}