{"slug": "slopcodebench-10-2026-opus-5-5-astra-wins-overall-xhigh-considered-harmful", "title": "SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhigh considered harmful", "summary": "SlopCodeBench's October 2026 run found gpt-6 Astra posting the highest strict pass rate at 16.3%, up from gpt-5.5's 14.8% in May, across 36 problems and 196 checkpoints per setting. Author Dex Horthy reported that medium reasoning effort outperformed xhigh for some newer models, with xhigh costing 2.11-4.22x as much as medium in the three complete pairs, and Opus 5.5 scoring better on medium than on xhigh. Strict pass rates for the tested frontier models — Opus 5.5, Fable 5.1, SOL 6 and 6.1, Astra, and GLM 5.3 — ranged from 12.2% to 16.3%, while core pass rates landed between 65.8% and 74.0%.", "body_md": "# SlopCodeBench update 10/2026 - Opus 5.5 update, Astra improves by 1.5% over gpt-5.5, xhigh considered harmful\n\nOctober 15, 2026 - [Dex Horthy](https://x.com/dexhorthy) w/ contributions from [Gabe Orlanski](https://x.com/gorlanski)\n\n**NOTE:** This is Part I of our October 2026 writeup: the Q4 state of the slop. Gabe and the [SlopCodeBench team](https://arxiv.org/abs/2603.24755) will share their analysis and a more qualitative deep dive into slop patterns in Part II.\n\nFor my previous writeups on SlopcodeBench, see [benchmarking-opus-5-on-slop-code-bench.md](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md) and [benchmarking-sol-fable-kimi-on-slop-code-bench.md](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-sol-fable-kimi-on-slop-code-bench.md). Both include deeper dives on the core structure of SCB and how each of its challenges is broken into checkpoints.\n\nIn May, the last time SCB was published, gpt-5.5 got a **14.8% strict pass rate on slopcodebench**. This month, I ran the frontier models through the same campaigns, and the highest scorer was gpt-6 astra, hitting **16.3%**.\n\nWhile the benchmark reports several different pass rates, the one we focus on is **strict pass rate** - in how many checkpoints did the model pass every test case while still passing all regression tests the frontier models we tested here had strict pass rates in the **12.2-16.3%** range:\n\n- Opus 5.5\n- Fable 5.1\n- SOL 6 and 6.1\n- Astra\n- GLM 5.3\n\nFor comparison, the benchmark also reports a \"core pass rate\" which is less strict, which lands in the **65.8-74.0%** pass rate for these models.\n\nOne interesting thing here that I'll dig into - medium effort performed better than XHigh for some newer models. XHigh costs **2.11-4.22x** as much as Medium in the three complete pairs, and most notably, **opus 5.5 did *BETTER* on medium effort than on xhigh**\n\nOver the past few months, I ran subsets of the bench. I [started with three problems](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-opus-5-on-slop-code-bench.md), then [tried six](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-sol-fable-kimi-on-slop-code-bench.md). This time: **36 problems and 196 checkpoints per setting**. Since we used a subset before, we can't use the earlier subsets to measure improvement or decline. The full benchmark lets us compare these results with the official research findings. It also means while the 30% fable 5 score from the latter of those two is interesting, its not comparable here, because it only covers 6 out of 36 problems.\n\n[SlopCodeBench](https://arxiv.org/abs/2603.24755) gives a coding agent a problem in stages. It writes a program, gets new requirements, and changes the existing code. Later checkpoints test both new behavior and old behavior.\n\nThe idea here is that this maps more onto how software is really built - with unclear and evolving requirements during the course of the build.\n\nFor folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock `just-solve` prompt, seed 42, the `any-case` runner pass policy, and Docker with Python 3.12 and uv.\n\nI ran each model and effort setting once. Each checkpoint builds on the code from earlier checkpoints. Some problems have more checkpoints than others. Each checkpoint counts equally toward the pass rate, so problems with more checkpoints have more weight.\n\n- **Strict:** every test category passes, including Regression.\n- **Core:** all tests in a nonempty Core category pass. For example, if a model fails checkpoint 2, it can still get a core pass for a later checkpoint, even if the ck2 tests remain failing.\n\n*Strict passes: 24-32 of 196. Core passes, including strict: 129-145. The tinted portion isn't extra strict success.*\n\n| Complete campaign | Strict passes | Strict rate | Core rate | Estimated cost | \n|---|---|---|---|---|\n| GPT-6 Astra XHigh | 32/196 | 16.33% | 69.39% | $758.62 | \n| GPT-6 SOL Medium | 30/196 | 15.31% | 67.86% | $245.69 | \n| GPT-6 Astra Medium | 29/196 | 14.80% | 66.33% | $261.05 | \n| GPT-6.1 SOL Medium | 29/196 | 14.80% | 65.82% | $178.51 | \n| Opus 5.5 Medium | 28/196 | 14.29% | 73.98% | $193.47 | \n| GPT-5.6 SOL XHigh | 28/196 | 14.29% | 70.92% | $753.30 | \n| Fable 5.1 XHigh | 26/196 | 13.27% | 72.45% | $1,550.07 | \n| GPT-6 SOL XHigh | 26/196 | 13.27% | 68.37% | $517.32 | \n| Opus 5.5 XHigh | 25/196 | 12.76% | 71.94% | $817.04 | \n| GLM 5.3 XHigh | 24/196 | 12.24% | 66.33% | $365.13 | \n\nAstra XHigh has the most strict passes in this snapshot. But it's lead over SOL 6.1 Medium is only **two checkpoints**.\n\nOpus Medium leads Core success at **145/196**, but passes only **28/196** strictly.\n\n*XHigh costs more in all three pairs; strict success rises only for Astra. These single runs don't isolate effort as the cause.*\n\n| Model | Strict: Medium → XHigh | Core: Medium → XHigh | Estimated cost multiplier | \n|---|---|---|---|\n| Opus 5.5 | 28 → 25 | 145 → 141 | 4.22x | \n| GPT-6 SOL | 30 → 26 | 133 → 134 | 2.11x | \n| GPT-6 Astra | 29 → 32 | 130 → 136 | 2.91x | \n\nThis doesn't prove that more reasoning hurts coding. It says **higher effort was not a reliable way to buy more strict passes in these runs**. One campaign per setting can't tell me how much is run-to-run variation.\n\n*Recorded estimates span $178.51-$1,550.07 within a 12.24-16.33% strict-score range.*\n\nGPT-6.1 Medium has the lowest recorded cost at $178.51, with the same strict count as Astra Medium. Fable's recorded estimate is about **8.68x** as much, with three fewer strict passes.\n\nCheckpoint success also isn't the same as ending with a working program.\n\n*Only each problem's last checkpoint counts here. Failed test cases aren't distinct bugs; skips stay separate.*\n\n| Complete campaign | Strict final checkpoints, out of 36 | Final failed test cases | Of those, Regression | \n|---|---|---|---|\n| GPT-6 Astra XHigh | 3 | 1,435 | 1,178 | \n| GPT-6 Astra Medium | 1 | 1,747 | 1,516 | \n| GPT-6 SOL XHigh | 1 | 1,651 | 1,366 | \n| GPT-6 SOL Medium | 1 | 2,831 | 2,479 | \n| GPT-6.1 SOL Medium | 3 | 2,757 | 2,404 | \n| GPT-5.6 SOL XHigh | 2 | 2,727 | 2,376 | \n| Fable 5.1 XHigh | 1 | 2,463 | 2,126 | \n| Opus 5.5 Medium | 0 | 2,508 | 2,206 | \n| Opus 5.5 XHigh | 1 | 2,446 | 2,123 | \n| GLM 5.3 XHigh | 1 | 2,748 | 2,311 | \n\nFirst - the models left a ton of junk and scratch code while working. My first reaction was to remove the random temp files from the quality scores. But I chatted with [gabe](https://x.com/gorlanski) and he made a good point: **the benchmark should measure how much garbage models leave in the codebase. If we're not reading the code, we need to measure it.** Tests, scratch code, and generated copies probably shouldn't get a free pass just because they aren't the main program.\n\nI did use a fixed list of folder names to exclude installed packages, virtual environments, and caches from both analyses.\n\nThe main results analyze all leftover code. For comparison, we also calculate the same metrics after removing a fixed list of temporary and generated files. That lets us see how much the cleanup changes the numbers. The rules and the actual difference are below.\n\nThese figures cover **the same 28 problems for all ten completed runs**, not all 36 problems. The scanner couldn't fully measure some files in eight problems (more on that in gabe's update later), so we left those problems out for every model.\n\nSome high level details on how we compute \"what % of this code triggers at least one slop rule\":\n\n- The scanner flags slop patterns and repeated code within each file.\n- \"Coverage\" is the share of source lines flagged by those checks.\n- We count each line only once, even if several checks flag it.\n- A check can flag a whole function without every line being a problem.\n\nWe total the flagged lines and source lines across all 28 problems, then divide to get the percentage. Larger programs therefore have more weight. **In the earlier posts, we averaged each problem's percentage instead.**\n\nOpus XHigh has the lowest slop percentage at **77.75%**, versus **94.25%** for Opus Medium. Interestingly, once again, the xhigh version has fewer strict passes: 25 versus 28.\n\n### while Opus 5.5 Medium had more strict passes, Opus 5.5. XHigh wrote code that was close to ~5x cleaner.\n\nIt is possible that this is because newer models come with new / different slop patterns, and the paper's original battery of 200+ linters is focused on a set of hand-curated patterns derived from agent-authored PRs in the months preceding the original publication in march 2026 and the v1.1 in may 2026. These new models may produce new maintainability anti-patterns that we haven't built linters for yet (more on that in Gabe's follow up post).\n\n*Opus Medium leaves the most code, even after filtering. We don't remove all tests.*\n\nFor the filtered score, we leave out folders named `_work`, `tmp`, `temp`, `scratch`, `generated`, `build`, or `dist`, plus `*_pb2.py` and `*_pb2_grpc.py` files. We use the same rules for every run. We don't guess whether code looks generated or exclude tests as a group. In these runs, the filter only removed `_work` files: **45 files and 41,265 source lines**, all from Opus Medium's final `dynamic_buffer` code. Its total drops from **216,881 to 175,616 source lines**, and slop-presence drops to **92.90%**. It still ranks most-frequent on slop presence. The filter changes nothing for the other models.\n\n*Fable has the lowest mean complexity, 3.57; GPT-6 SOL Medium has the highest, 11.70. Functions and methods each get one vote.*\n\nAs you might expect, splitting code into more helper functions can lower average complexity without making the code easier to change. Less code isn't always better either.\n\nThere's a ton more metrics and I'm excited to get Gabe's take on the more qualitative side of these outputs.\n\nThe full benchmark is hard in a way that matters to me: the agents get the main behavior right much more often than they keep **all** tested behavior working as requirements change.\n\nWe're getting better at maintaing a codebase over time (14.8% to 16.3% during may->september), but not as fast as anyone would have you believe. I think we'll be keeping the lights on for a while to come.\n\n**NOTE:** This is Part I of our October 2026 writeup: the Q4 state of the slop. Gabe and the [SlopCodeBench team](https://arxiv.org/abs/2603.24755) will share their analysis and a more qualitative deep dive into slop patterns in Part II.", "url": "https://wpnews.pro/news/slopcodebench-10-2026-opus-5-5-astra-wins-overall-xhigh-considered-harmful", "canonical_source": "https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-new-models-on-slop-code-bench.md", "published_at": "2026-10-15 19:09:34+00:00", "updated_at": "2026-10-15 19:28:54.786214+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "developer-tools"], "entities": ["SlopCodeBench", "Dex Horthy", "Gabe Orlanski", "gpt-6 Astra", "gpt-5.5", "Opus 5.5", "Fable 5.1", "GLM 5.3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/slopcodebench-10-2026-opus-5-5-astra-wins-overall-xhigh-considered-harmful", "markdown": "https://wpnews.pro/news/slopcodebench-10-2026-opus-5-5-astra-wins-overall-xhigh-considered-harmful.md", "text": "https://wpnews.pro/news/slopcodebench-10-2026-opus-5-5-astra-wins-overall-xhigh-considered-harmful.txt", "jsonld": "https://wpnews.pro/news/slopcodebench-10-2026-opus-5-5-astra-wins-overall-xhigh-considered-harmful.jsonld"}}