{"slug": "is-xhigh-reasoning-effort-worth-it-for-a-coding-agent-what-slopcodebench-10-2026", "title": "Is xhigh reasoning effort worth it for a coding agent? What SlopCodeBench 10/2026 and an 85-task Auto run measured", "summary": "HumanLayer's SlopCodeBench 10/2026 run, published 15 October 2026, found that xhigh reasoning effort did not reliably improve coding-agent results: Claude Opus 5.5 passed 28 of 196 strict checkpoints at medium effort and 25 at xhigh while costing $817.04 versus $193.47, a 4.22x multiplier. Across the three models run at both levels, xhigh cost 2.11 to 4.22 times medium, and strict passes rose for only one model (GPT-6 Astra, 29 to 32) while falling for Claude Opus 5.5 and GPT-6 Sol (30 to 26). HumanLayer cautioned that each setting ran once, so the results are single measurements, and that the data does not prove more reasoning hurts coding, only that higher effort was not a reliable way to buy more strict passes.", "body_md": "# Is xhigh reasoning effort worth it for a coding agent? What SlopCodeBench 10/2026 and an 85-task Auto run measured\n\nFor most coding work, the public numbers say xhigh reasoning effort does not pay for itself on current models. In HumanLayer's [SlopCodeBench](https://stackness.dev/tools/slopcodebench) [run](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-new-models-on-slop-code-bench.md), published on 15 October 2026, [Claude Opus](https://stackness.dev/tools/claude-opus) 5.5 passed 28 of 196 checkpoints at medium effort and 25 at xhigh. The xhigh run cost 4.22 times as much: $817.04 against $193.47. Across the three models run at both levels, xhigh cost 2.11 to 4.22 times medium. Strict passes rose for one model and fell for two. Each setting ran once, so these are single measurements, not settled differences.\n\n## Xhigh cost 2.1 to 4.2 times medium and bought no reliable gain\n\nSlopCodeBench 10/2026 ran three models at both medium and xhigh effort on the full benchmark: 36 problems and 196 checkpoints, where each checkpoint extends code the agent wrote at the previous one. A strict pass means every test category passes, regression tests included. The table is HumanLayer's, with cost per strict pass derived by me from its estimated cost.\n\n| Model | Strict passes, medium → xhigh | Cost, medium → xhigh | Multiplier | $ per strict pass, medium → xhigh | \n|---|---|---|---|---|\n| Claude Opus 5.5 | 28 → 25 | $193.47 → $817.04 | 4.22x | $6.91 → $32.68 | \n| [GPT](https://stackness.dev/tools/gpt) -6 Sol | 30 → 26 | $245.69 → $517.32 | 2.11x | $8.19 → $19.90 | \n| GPT-6 Astra | 29 → 32 | $261.05 → $758.62 | 2.91x | $9.00 → $23.71 | \n\nThe \"xhigh considered harmful\" headline appears only in the post's title. The body is more careful: \"This doesn't prove that more reasoning hurts coding. It says higher effort was not a reliable way to buy more strict passes in these runs.\" The costs are recorded inference cost, not invoices, and HumanLayer published no token counts or run times. GPT-6 Astra, OpenAI's top model, led the full table at 32 strict passes.\n\n## What does the reasoning effort setting change in a coding agent?\n\nReasoning effort tells the model how many output tokens to spend on a task: thinking, tool calls and the text it writes. Anthropic's [effort docs](https://platform.claude.com/docs/en/build-with-claude/effort) call it \"a behavioral signal, not a strict token budget\". At higher effort, Claude may make more tool calls, explain its plan and write longer summaries. Claude offers `low`, `medium`, `high`, `xhigh` and `max`. Opus 5.5 defaults to `medium` in the API and in [Claude Code](https://stackness.dev/tools/claude-code).\n\nOpenAI's [reasoning docs](https://developers.openai.com/api/docs/guides/reasoning) list levels from `none` to `max`, depending on the model, and say of `xhigh`: \"Only use when your evals show a clear benefit that justifies the extra latency and cost.\" [Codex](https://stackness.dev/tools/codex-cli) reads the setting from `model_reasoning_effort`. Anthropic's [Opus 5.5 guide](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5-5) says the same thing: \"Reserve `xhigh` and `max` for work where you've measured a quality gain.\" Thinking tokens are billed as output tokens: $20 per million on Opus 5.5 and $50 on GPT-6 Astra for prompts up to 272,000 tokens.\n\n## Opus 5.5 passed fewer checkpoints at xhigh than at medium\n\nOpus 5.5 at xhigh passed three fewer strict checkpoints than at medium, 25 against 28, and spent $623.57 more to do it. Per checkpoint, it cost $0.99 at medium and $4.17 at xhigh. Its core pass rate, which counts only the main tests, also fell, from 73.98% to 71.94%.\n\nXhigh did produce cleaner code by one measure. On 28 matched problems, 77.75% of the xhigh run's lines were flagged by at least one of the benchmark's slop checks, against 94.25% for medium. The post calls that \"close to ~5x cleaner\". Neither the ratio of flagged lines (1.21) nor the ratio of unflagged lines (3.9) gives 5, and the post shows no metric that does. Commenters on [Hacker News](https://news.ycombinator.com/item?id=50108717) raised the noise: \"When scores are ~15%, it's unclear how much is just in the noise.\"\n\n## A strict pass cost $6 to $9 at medium and $15 to $60 at xhigh\n\nAcross all ten campaigns in the SlopCodeBench table, every medium run cost between $6.16 and $9.00 per strict pass, and every xhigh run cost between $15.21 and $59.62. The cheapest was GPT-6.1 Sol at medium, with 29 passes for $178.51. The most expensive was Anthropic's [Claude Fable](https://stackness.dev/tools/claude-fable) 5.1 at xhigh, with 26 passes for $1,550.07; the first version of the post noted that Fable's cost metering had gaps.\n\nGPT-6 Astra was the only model where xhigh bought more passes, and each extra pass cost $165.86. For the three models run at both levels, the price of a pass varied 1.3 to 1.6 times between models and 2.4 to 4.7 times between effort levels.\n\n## Caveman's Auto saved 54% mostly by running Opus at medium\n\n[Caveman](https://stackness.dev/tools/caveman)'s README claims Auto is \"54% cheaper with Auto, same score\" across \"85 coding tasks vs Opus 5.5 at xhigh\". The [benchmark doc](https://github.com/JuliusBrussee/caveman/blob/main/docs/AUTO-BENCHMARK.md) behind it is an interim run dated 3 October 2026, paused at 85 of 169 paired tasks from SWE-Interact and SWE-Together, one try per task in Claude Code. Auto resolved 43.7% against 44.3% for Opus 5.5 at xhigh, at $3.59 against $7.83 per attempt.\n\nThe doc explains the saving itself. Auto ran Opus 5.5 at medium on 87% of requests, Opus at high on 8% and [Claude Sonnet](https://stackness.dev/tools/claude-sonnet) 5.5 on 5%. In its words: \"Most of the saving comes from lower effort (Opus xhigh to Opus medium or high), not from cheaper models.\" The score gap of 0.5 points has a 95% interval of −6.8 to +5.7 points, and median time fell from 35.2 to 15.9 minutes. Caveman sells the cloud service Auto runs on, ran the test itself and has not published the raw runs. The baseline it undercut on cost, xhigh on every request, is two levels above Opus 5.5's default.\n\n## Higher effort pays on hidden edge cases and first-try runs\n\nHigher effort helped in two measured cases. In an [observational study](https://arxiv.org/abs/2607.02436) of 90 runs building one app, Opus 4.7 in Claude Code produced a perfect result on the first try in 5 of 18 runs at high and 16 of 18 at xhigh, for a median cost premium of 9% to 29%. The pooled mean score barely moved, from 41.0 to 41.8 of 42 points. The gain was in needing fewer corrections.\n\nAnthropic's [own analysis](https://claude.dev/blog/spending-your-effort/) of Terminal-Bench 3.0, a public community-sourced benchmark, found that \"higher effort is best for tasks with lots of hidden edge cases.\" Fable 5.1 passed 140 of 370 attempts at low and 214 at max. That compares the lowest effort level with the highest, not medium with xhigh. The docs point the same way: Claude Code suggests `max` for \"finding security vulnerabilities\", OpenAI suggests xhigh for \"security and code review\".\n\n## CursorBench and a real-repo replay flatten out above high\n\nOther public measurements of Opus 5.5 show little gain from high to xhigh. [CursorBench](https://cursor.com/cursorbench) reports 52.5% at medium for $2.91 per task, 56.0% at high for $3.97, 56.0% at xhigh for $6.98 and 57.8% at max for $13.43. The page gives no task count or run date (CursorBench 4.0 dates from 10 September 2026) and warns that small differences \"may not be statistically meaningful\". Anthropic's [launch post](https://www.anthropic.com/news/claude-opus-5-5) puts Opus 5.5 at 54.6% on FrontierCode at medium, against 54.4% at max.\n\n[paddo.dev](https://paddo.dev/blog/default-was-right/) replayed six real merged changes, 20 runs per level in total: Opus 5.5 got 13 clean runs at medium for $44.10 and 12 at xhigh for $139.79. The [original SlopCodeBench paper](https://arxiv.org/abs/2603.24755) ran every model but one at high and compared no effort levels, so the October numbers are the first published effort comparison on this benchmark that I could find.\n\n## Measure effort on your own repository\n\nThe public numbers show a pattern, but the decision depends on your own tasks. Replay 10 to 20 merged changes from their parent commits at two levels, and record pass, broken existing tests and cost for each run. In Claude Code, unset `CLAUDE_CODE_EFFORT_LEVEL` first, because it overrides the `--effort` flag:\n\n```\nunset CLAUDE_CODE_EFFORT_LEVEL\nclaude -p \"$(cat task.md)\" --effort medium --output-format json > medium-1.json\nclaude -p \"$(cat task.md)\" --effort xhigh --output-format json > xhigh-1.json\njq '.total_cost_usd' medium-1.json xhigh-1.json\n```\n\n`total_cost_usd` is a client-side estimate, and `modelUsage` breaks it down per model, subagents included. Codex reports tokens but no dollars:\n\n```\ncodex exec -c model_reasoning_effort='\"xhigh\"' --json \"$(cat task.md)\" > xhigh-1.jsonl\n```\n\nFrom each `turn.completed` event, price `input_tokens` minus `cached_input_tokens` at the input rate, `cached_input_tokens` at the cached rate and `output_tokens` at the output rate. Check one run to see whether `reasoning_output_tokens` is already inside `output_tokens`. Pin one model per comparison. Yong Yu [estimated](https://dev.to/yong_yu_f98e15562e9b120a0/medium-beat-xhigh-for-opus-55-on-slopcodebench-cost-per-pass-the-noise-floor-and-an-effort-ab-2i2b) that telling 14.29% from 12.76% apart would take about 40 full SlopCodeBench campaigns per setting. A paired test on your own tasks, counting only the tasks where the two levels disagree, can show a large difference with a few dozen tasks; a gap of one or two points still needs hundreds.\n\n## Method and sample\n\nEvery SlopCodeBench figure here comes from HumanLayer's published tables, with one run per model and effort level, on harness versions and dates that differ between runs. The 196 checkpoints are not independent trials: each one builds on the code from the one before. The cost-per-pass figures are my arithmetic on HumanLayer's estimated cost. No tokens or run times were published, so no per-token cost can be derived. The Caveman figures are a vendor's interim result.\n\nOn Stackness, as of 27 October 2026, Claude Code is on 8 public stacks, Claude Opus on 4 and Codex on 2, bots removed and mine included ([how I count](https://stackness.dev/about/data-sources)). Stacks do not record effort settings. The [AI tools developers list](https://stackness.dev/categories/ai-tools) shows which agents people pair with which models.\n\n## Key numbers\n\n- **4.22x** : cost of Opus 5.5 at xhigh against medium on SlopCodeBench,**15 October 2026** ([HumanLayer](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/benchmarking-new-models-on-slop-code-bench.md) ).\n- **28 vs 25** : Opus 5.5 strict passes of 196 checkpoints, medium vs xhigh,**15 October 2026** (HumanLayer).\n- **2.11x to 4.22x** : xhigh cost against medium across three models,**15 October 2026** (HumanLayer).\n- **$6.16 to $9.00** : cost per strict pass for medium runs, against**$15.21 to $59.62** for xhigh runs,**15 October 2026** (derived from HumanLayer).\n- **0.46x** : Caveman Auto's cost against Opus 5.5 at xhigh, 43.7% vs 44.3% resolved on 85 tasks,**3 October 2026** ([Caveman](https://github.com/JuliusBrussee/caveman/blob/main/docs/AUTO-BENCHMARK.md) ).\n- **5 of 18 vs 16 of 18** : first-try perfect runs, Opus 4.7 at high vs xhigh,**5 September 2026** ([arXiv 2607.02436](https://arxiv.org/abs/2607.02436) ).\n\n## New posts by email\n\nAbout one email a week with what we published. No account needed.\n\n## Which of these do you use?\n\nNobody on Stackness runs SlopCodeBench yet. Tick what you use and be the first.\n\n[Play Lonny Eagle Amateur Skater 4A pixel art skateboarding game. Signing up opens it straight away.Sign up to play](https://stackness.dev/games/lonny-eagle)", "url": "https://wpnews.pro/news/is-xhigh-reasoning-effort-worth-it-for-a-coding-agent-what-slopcodebench-10-2026", "canonical_source": "https://stackness.dev/blog/is-xhigh-reasoning-effort-worth-it-coding-agent-slopcodebench-numbers", "published_at": "2026-10-27 10:27:24+00:00", "updated_at": "2026-10-27 11:14:38.131322+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "large-language-models"], "entities": ["HumanLayer", "SlopCodeBench", "Claude Opus 5.5", "GPT-6 Sol", "GPT-6 Astra", "Anthropic", "OpenAI", "Claude Code"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/is-xhigh-reasoning-effort-worth-it-for-a-coding-agent-what-slopcodebench-10-2026", "markdown": "https://wpnews.pro/news/is-xhigh-reasoning-effort-worth-it-for-a-coding-agent-what-slopcodebench-10-2026.md", "text": "https://wpnews.pro/news/is-xhigh-reasoning-effort-worth-it-for-a-coding-agent-what-slopcodebench-10-2026.txt", "jsonld": "https://wpnews.pro/news/is-xhigh-reasoning-effort-worth-it-for-a-coding-agent-what-slopcodebench-10-2026.jsonld"}}