Dev ToolsArticle The pass rate jumped, but the volume and lint metrics show long-horizon code quality is still unsolved.
Lenn Voss Model launches follow a ritual by now: vendor charts, a week of vibes, and a scramble of third-party benchmark runs. Most of those runs tell you little, because the standard coding benchmarks are saturated enough that frontier models cluster within noise of each other. But one community run of Claude Opus 5 deserves a closer look — not because of the headline number, but because of the benchmark underneath it, and what the secondary metrics reveal about how the number was earned.
Dex Horthy of HumanLayer ran Opus 5, Opus 4.8, and Sonnet 5 against a slice of SlopCodeBench, a benchmark out of UW–Madison that flips the usual setup. Instead of handing the agent a complete problem statement, it reveals requirements progressively across checkpoints. The agent extends its own prior solution at each stage, and a strict pass means the new checkpoint's tests and every inherited regression test go green. That's much closer to actual software work than any one-shot eval: you don't get the full spec up front, and your biggest liability is the code you wrote last sprint.
Why this benchmark exists at all #
The name is the thesis. The SlopCodeBench paper (March 2026, revised in May) found that across 36 problems and 196 checkpoints, agent-written code was 2.3x more verbose and 2.0x more structurally eroded than comparable human repositories, with erosion increasing in 77% of trajectories. The best frontier agent at publication cleared just 14.8% of checkpoints strictly, and no agent solved a single problem end-to-end. While SWE-bench leaderboards compress toward the top, this one has enormous headroom — which makes it one of the few places a new model can actually show separation.
That's the context for Horthy's run, which covered three of the benchmark's problems — a circuit simulator (easy, 8 checkpoints), a database migration tool (medium, 5), and a config-service REST API (hard, 4) — with fresh context per checkpoint, identical prompts across models, and black-box grading.
The headline and the asterisk #
Opus 5 passed 24% of the 17 checkpoints strictly. Opus 4.8 and Sonnet 5 managed 6% each. A quadrupling looks dramatic until you convert to raw counts: four checkpoints versus one, concentrated in the easy circuit problem's opening stages. At this sample size, that's a promising signal, not a verdict. And the ceiling finding held across the board — no model reached the end of any of the three problems without defects.
The more interesting data is what Opus 5 did to get there. It produced roughly 29,000 source lines against about 9,000 for each of the other two models. Just over half of that output was tests, versus 11% for Opus 4.8. It wrote five times as many functions, each individually simpler — which reads either as good decomposition or as fragmentation, depending on your priors. And the quality linters were brutal to everyone: between 89% and 98% of generated lines triggered rule violations across all three models.
So the ranking moved. The pathology didn't.
Volume is the real story #
The agentic-coding shift of the last two years moved the bottleneck from writing code to reviewing it. Read Opus 5's numbers through that lens and the win gets more expensive: three times the code is three times the review surface, and a wall of agent-written tests is not the safety net it appears to be. Tests written by the same model that wrote the implementation tend to encode the implementation's assumptions — including its bugs. Worse, in an iterative setting like this one, a sprawling early test suite ossifies architecture. Refactoring means rewriting hundreds of tests, so the agent doesn't refactor, and structural debt compounds into exactly the later-checkpoint failures SlopCodeBench is designed to expose.
That's presumably why Hacker News, usually eager to celebrate a 4x, shrugged — the thread drew a modest 88 points, and the prevailing sentiment was "nice improvement, not revolutionary," with several commenters saying they still reach for other models for day-to-day coding.
What to actually do with this #
If you're deciding whether to move your agent workflows to Opus 5, this run says: probably yes for iterative work, but don't touch your guardrails. Concretely:
Budget for review load, not just pass rate. If Opus 5's volume behavior holds in your stack, your diffs get bigger. Cap patch size in your harness and reject oversized changes rather than skimming them.Gate on structure, not just tests. Every model in this run shipped lint-violating code at 89%+ rates. Put complexity and duplication checks (radon
,lizard
, ESLint'scomplexity
rule) in the agent's inner loop, where it can react, not just in CI where a human absorbs the failure.Review agent tests adversarially. Treat a 51%-tests diff as a claim to verify, not evidence of rigor. Spot-check that tests assert behavior from the spec, not behavior from the implementation.Own the context strategy. The methodology here — fresh context per checkpoint, spec restated each time — mirrors the intentional-compaction practices that the best agent teams already use. Long-horizon quality is at least as much a harness problem as a model problem, and this data backs that up: model choice changed the pass rate, but no model choice fixed erosion.
The verdict #
Opus 5's gain looks real, and SlopCodeBench is exactly the kind of unsaturated, long-horizon eval the field should be watching instead of another decimal on SWE-bench. But a 3-problem subset with a four-versus-one checkpoint spread is a preview, not a result — and the durable finding cuts across every model tested: agents still can't inhabit a codebase over time without degrading it. Until a model moves that number, "which model writes the least slop per passing test" is the benchmark that matters, and the answer to code erosion still lives in your harness, your linters, and your review process — not in the model dropdown.
Sources & further reading #
[Benchmarking Opus 5 on SlopCodeBench](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md)— github.com -
[SlopCodeBench: Benchmarking How Coding Agents Degrade Over Iterative Development](https://arxiv.org/abs/2603.24755)— arxiv.org -
[Benchmarking Opus 5 on SlopCodeBench - Hacker News discussion](https://news.ycombinator.com/item?id=49076391)— news.ycombinator.com
[Lenn Voss](https://sourcefeed.dev/u/lennart_voss)· Cloud & Infrastructure Writer
Lenn writes about cloud platforms, Kubernetes internals, and the infrastructure decisions that quietly make or break engineering organizations. Based in Berlin's vibrant tech scene, they have a talent for turning dense platform-engineering topics into prose that people actually finish reading.
Discussion 0 #
No comments yet
Be the first to weigh in.