{"slug": "the-best-coding-agent-still-gets-6-in-10-changes-wrong-that-is-your-review-load", "title": "The best coding agent still gets ~6 in 10 changes wrong. That is your review load.", "summary": "Specific Labs' Real-SWE benchmark, published in September 2026, found that the best frontier coding agent resolves only 38.8% of tasks on private enterprise codebases, meaning roughly six in ten agent-generated changes are wrong. The benchmark runs agents on realistic production changes across billing, tax, and multi-service migrations, with reference solutions touching a median of 11 files. The findings suggest review teams should treat agent-generated pull requests as drafts and concentrate review depth on codebase areas with low pass rates.", "body_md": "While tech review articles argue about what PR length is safe to let an agent produce, the review problem is set by a simpler number: how often the agent is wrong.\n\nSpecific Labs' Real-SWE benchmark, published September 2026, runs frontier agents on private, licensed enterprise codebases. The best model-and-harness combination resolves 38.8% of tasks, pass@1 averaged over eight runs per task. Everything else scores lower, down to 16.2%.\n\nRead that the other way. The strongest agent setup, on realistic production changes across billing, tax, and multi-service migrations, is wrong on roughly six in ten tasks. Reference solutions touch a median of 11 files, against six in FrontierCode and DeepSWE.\n\nThe point for review teams is not the ranking. It is that the acceptance decision is the part that did not get faster. An agent can land a change in minutes. Deciding whether that change is correct, whether it preserves behavior across the other ten files it touched, still needs a human who understands the system. That is why review time climbs even when PRs land: every agent change carries a ~60% chance it needs real correction, and the correction is not free.\n\nTwo workflow implications.\n\nFirst, treat agent-generated PRs as drafts by default, not as candidates. The benchmark says over half are wrong. Reviewing a draft and reviewing a \"finished\" change are different mindsets, and the second one skips the assumptions the agent should not have made.\n\nSecond, measure the acceptance rate per agent, per area of the codebase, and adjust the review depth from that, per path. If a codebase area trends low pass rates, that is where the attention budget should concentrate, not spread evenly across every PR. The codebase matters more than the raw volume, and the failure rate tells you which paths will burn the most reviewer time.\n\nThe expensive work moved downstream to the people who understand the system. Benchmarks like Real-SWE give you the number that says how much of it there is.", "url": "https://wpnews.pro/news/the-best-coding-agent-still-gets-6-in-10-changes-wrong-that-is-your-review-load", "canonical_source": "https://dev.to/tessainsley/the-best-coding-agent-still-gets-6-in-10-changes-wrong-that-is-your-review-load-3h5n", "published_at": "2026-09-13 04:15:01+00:00", "updated_at": "2026-09-13 04:56:41.689900+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "developer-tools", "ai-tools"], "entities": ["Specific Labs", "Real-SWE", "FrontierCode", "DeepSWE"], "alternates": {"html": "https://wpnews.pro/news/the-best-coding-agent-still-gets-6-in-10-changes-wrong-that-is-your-review-load", "markdown": "https://wpnews.pro/news/the-best-coding-agent-still-gets-6-in-10-changes-wrong-that-is-your-review-load.md", "text": "https://wpnews.pro/news/the-best-coding-agent-still-gets-6-in-10-changes-wrong-that-is-your-review-load.txt", "jsonld": "https://wpnews.pro/news/the-best-coding-agent-still-gets-6-in-10-changes-wrong-that-is-your-review-load.jsonld"}}