{"slug": "what-opus-5-s-novel-level-reasoning-benchmark-actually-means", "title": "What Opus 5's Novel-Level Reasoning Benchmark Actually Means", "summary": "The ARC Prize Foundation reported that Anthropic's Opus 5 scored 30.2% on the ARC-AGI-3 benchmark, measuring action efficiency against a median human player, not problems solved. The independent benchmark, built in March 2026, showed Opus 5 newly beat five Public Demo environments that no model had previously solved, with four of those at or above median-human efficiency. The result is real, but the benchmark's name and scoring rule have been omitted from most recaps, making the claim unverifiable for readers.", "body_md": "July 25, 2026\n\nOpus 5 novel reasoning has exactly one number behind it, 30.2% on ARC-AGI-3, and that benchmark belongs to the ARC Prize Foundation, which built it in March 2026 and scored the run itself. The figure measures action efficiency against a median human player, not problems solved. The one change that touches your day is that Anthropic's recommended starting effort for coding moved down, from xhigh on Opus 4.8 to high on Opus 5, so the config you carried over is running above the recommended default.\n\nEvery number below comes from a page that is not trying to sell you the model. That is the whole method, and it is the reason the figures here do not match the recaps you scrolled past.\n\nI worked from the ARC Prize Foundation's own results page for Opus 5, the ARC Prize blog post that defines the scoring rule, and the Claude platform and Claude Code documentation. I did not use the launch announcement, because a vendor post about a vendor's model is a claim to check rather than evidence to cite.\n\nSo the evidence admitted here is narrow on purpose.\n\nTwo things are missing on purpose. I could not open the System Card PDF, so the **evaluation-awareness** and OSS-Fuzz findings other writeups quote are absent here rather than paraphrased from someone else's paraphrase. The practitioner threads are hours old with thin comment counts, so nothing in this post speaks for the community.\n\nEvery recap I opened uses the phrase novel problem solving and then moves straight on. None of them names the benchmark, which leaves a reader who wants to verify the claim with nowhere to go.\n\nThe benchmark is **ARC-AGI-3**. The ARC Prize Foundation built and published it in March 2026, roughly four months before Opus 5 existed, and ARC Prize ran and published the Opus 5 verification on [its own results page](https://arcprize.org/results/anthropic-claude-opus-5).\n\nAnthropic did not build this test and did not score this run. That works in Anthropic's favour. An **independent scorer** with a public results page is the strongest form a benchmark claim can take.\n\nIt also tells you where the floor sits. ARC Prize reported that humans solve 100% of the environments while frontier AI systems, as of March 2026, scored below 1%, which is the baseline the ARC-AGI-3 score you saw quoted is standing on.\n\nThe result is real. What is missing is the benchmark's name, and a number shipped without it cannot be checked by the person reading it. Every recap shipped it that way.\n\nStart with the part that earns its headline. Opus 5 newly beat five Public Demo environments that no model had beaten before, and secondary coverage puts four of those five at or above **median-human efficiency**. Learning an unfamiliar interactive environment from scratch and then playing it in fewer moves than the median human is a real result.\n\nIt helps to know what an environment is. ARC-AGI-3 drops an agent into an interactive game whose rules are never explained, and the agent has to infer the mechanics by acting and watching what changes on screen. People solve all of them. Machines, four months ago, mostly could not get started.\n\nNow read what the number counts. The Opus 5 novel reasoning headline is not thirty percent of the problems solved, and it is not coverage of the benchmark either.\n\nEach level is scored by how many actions the model took against the median human player rather than by whether it finished. A score of 100% would mean beating every level of every environment at or above that median human efficiency, and the per-level cap runs to 115%, so moving faster than the median human earns more than a perfect mark. That rule lives on the [ARC Prize human-dataset post](https://arcprize.org/blog/arc-agi-3-human-dataset) rather than on the results page, which is exactly why no recap quotes it.\n\nScope matters as much as the rule. The set is the 25 Public Demo environments, so five newly beaten environments is a **coverage** figure and 30.2% is an **efficiency** figure, and those are two different quantities that recaps blur into one.\n\nThe results table also renders the number as 30.16% and rounds it to 30.2% in prose. The headline is a rounded figure.\n\nZoom out to the board Opus 5 tops and it is thinner than the headline implies.\n\n| Category | ARC-AGI-3 score | |\n|---|---|---|\n| Claude Opus 5 | 30.2% | |\n| GPT-5.6 Sol | 7.8% | |\n| Claude Opus 4.8 | 1.5% | |\n| GPT-5.6 Terra | 0.8% | |\n| GPT-5.5 | 0.4% | |\n| Gemini 3.1 Pro | 0.4% | |\n| Grok 4.5 | 0.3% | |\n| GPT-5.4 | 0.2% | |\n| GPT-5.6 Luna | 0.2% | |\n| Claude Opus 4.7 (Adaptive) | 0.2% | |\n| Grok 4.20 | 0.1% |\n\nTen of the eleven models on the published ARC-AGI-3 leaderboard sit under 8%. GPT-5.6 Sol is second at 7.8% and Opus 4.8 third at 1.5%, both as of 2026-07-25 on [the published board](https://benchlm.ai/benchmarks/arcAgi3), and Fable 5 has no entry on it at all.\n\nThis is the finding that changes a file on your machine. Claude Opus 5 effort levels run five rungs deep, and the docs tell you to [start at high](https://platform.claude.com/docs/en/build-with-claude/effort), the default, then adjust from your own evals. The same page told Opus 4.8 users to start at\n\n`xhigh`\n\nfor coding and agentic work.The recommendation moved down a rung. Not up.\n\nNothing holds your old value for you. A level you previously set simply carries over, and the Claude Code docs tell migrators to run a fresh effort sweep on their own evals rather than reuse levels tuned on an earlier model. So the developer who copied a config across is running above the recommended start, paying more per request, and filing the whole thing under upgrade.\n\n```\n- { \"model\": \"claude-opus-4-8\", \"effort\": \"xhigh\" }\n+ { \"model\": \"claude-opus-5\" }\n```\n\nDeleting the parameter beats setting it to high. Both land on the same rung, and only one of them documents that you are sitting on the recommended default rather than carrying a value forward from a model that no longer runs your work.\n\nMax is not the escape hatch either. Opus 5 scores slightly worse at max than at xhigh on two published benchmarks while costing more, per The Decoder, and the Claude Code docs describe the top rung as prone to overthinking. The reported mechanism is the model refactoring code nobody asked it to touch.\n\nOpus 5 generated about 100 million tokens across one independent benchmark suite against Opus 4.8's 120 million, roughly a sixth fewer, and that run still cost slightly more to complete, per Implicator.ai's write-up of the numbers. Fewer tokens does not automatically mean a smaller invoice.\n\nThe docs are a recommendation tuned for the median workload, not a measurement that xhigh stopped paying. I have no published high-against-xhigh comparison on Opus 5, and neither does anyone else I could find. What the change tells you is that Anthropic no longer thinks xhigh is the right place to start, which makes the burden of proof yours if you keep it, not theirs.\n\nDivide 30.16 by 7.8 and you get **roughly 4x**, which is where the three-times-the-next-model line comes from. Stated as a lead over the next model on the published leaderboard, it is fair. Stated as a lead over the field, it is not, because that board carries no Fable 5 row and the absence is what keeps the multiple tidy.\n\nNow the Opus 5 coding benchmarks. On DeepSWE v1.1 agentic coding, Opus 5 lands third at 68.8%, behind GPT-5.6 Sol at 72.7% and [Fable 5](/blog/guides/ways-to-make-the-most-out-of-claude-fable-5) at 69.7%, per The Decoder's reading of the launch table as of 2026-07-25.\n\nThat is 3.9 points off the leader and 0.9 off Fable 5. Two of the five benchmarks where Opus 5 is not first are effectively ties, 0.1 points on FrontierCode v1.1 and 0.2 on Humanity's Last Exam without tools.\n\nOne is a 1.6-point loss on the Legal Agent Benchmark and one is a 6.2-point loss on HealthBench Professional. The honest one-line summary of that table is parity on the coding rows, not across the table.\n\nThese are vendor-origin numbers relayed by a third party, and only the DeepSWE row is independently confirmed, so weight them accordingly.\n\nNo source anywhere connects an ARC-AGI-3 gain to a measured coding improvement. The absence is itself the finding, and I am not going to fill it with a transfer story nobody has evidence for.\n\nWhat would change my mind is narrow and obvious. Run one coding eval against Opus 4.8 and Opus 5 at a held-constant effort level and publish both halves. Until somebody does that, a puzzle-benchmark jump predicts nothing about your repository.\n\nA bare model-ID swap is usually fine, which is precisely why the failures that do happen are so confusing. Three of them are worth knowing before you ship the change. The fourth is the effort value you carried over, which I argued above rather than argue twice.\n\nTwo more things developers hit on day one, both from first-day threads rather than documentation, so read them as individual reports and not as confirmed behaviour.\n\nTwo things developers hit on day one\n\nDevelopers on r/ClaudeCode report the desktop client turns thinking off for you, and Opus 5's docs state it accepts disabled thinking only at effort high or below. The two together are rejected on every request, so the failure reads as a problem with xhigh when it is really the combination. Pick high in the desktop app, or make the call yourself with thinking left on, and xhigh runs again.\n\nDevelopers on r/ClaudeCode report seeing exactly that split, and no Anthropic doc states it, so treat it as one report rather than a limit you can plan around. If a long session behaves differently from what you expected, check which surface you are on before you rewrite the prompt or blame the model.\n\nThe claude-opus-5 400 error is not triggered by the swap on its own. It fires when a single request both disables thinking and sets effort above high, a rule that [the whats-new page for Opus 5](https://platform.claude.com/docs/en/about-claude/models/whats-new-opus-5) calls a breaking change from Opus 4.8 and enforces on every request.\n\n```\n{\n  \"model\": \"claude-opus-5\",\n  \"effort\": \"xhigh\",\n  \"thinking\": { \"type\": \"disabled\" }\n}\n```\n\nThere are two fixes and they are not equivalent. Drop the effort to high if what you actually wanted was a shorter wait. Leave thinking switched on if what you actually wanted was the higher rung.\n\nRun `claude update`\n\nand land on v2.1.219 or later before you touch anything else. Below that version Opus 5 does not run, and neither does the category-based fallback that decides what happens when a classifier reroutes your request, per the [Claude Code model configuration docs](https://code.claude.com/docs/en/model-config).\n\nA request flagged as cybersecurity content re-runs on Opus 4.8 rather than failing outright, so the response you get back may not be from the model you asked for. The docs do not describe a signal that tells you it happened.\n\nThe migration checklist\n\nRun claude update and land on v2.1.219 or later. Below that version Opus 5 does not run at all, and neither does the category-based fallback that decides what happens when a classifier reroutes your request.\n\nChange xhigh to high, or delete the parameter, because high is the default on Opus 5 and the docs tell migrators to run a fresh effort sweep on their own evals rather than reuse the levels they tuned on Opus 4.8. Deleting the line is the stronger move, since it documents that you are on the recommended start instead of a number you inherited.\n\nSend one request that both disables thinking and sets effort above high, and it comes back rejected every time, because the rule is enforced per request. The two fixes are not equivalent. Drop the effort to high if what you wanted was the shorter wait, and stop disabling thinking if what you wanted was the higher rung.\n\nThe four verification patterns, a final verification step for non-trivial tasks, use a subagent to verify, double-check your answer, and re-verify before responding. Opus 5 verifies its own work without being told to, and the docs say instructions like these cause over-verification, so removing them cuts wasted tokens with no loss in quality.\n\nBecause effort shapes the rendered prompt, so changing it between requests does not preserve the cached prefixes from earlier turns. Pick one level at the start of a long session and keep it there, even when a single hard task tempts you to reach for a higher rung.\n\nRevisit it if you tuned it for an Opus 4.8 workload with thinking off, since max_tokens caps thinking plus response text together and the docs suggest starting at 64,000 for the top two rungs. Do not expect the effort dial to control length, because lowering effort can reduce thinking volume without reliably shortening the visible response. Ask for the length you want in the prompt instead.\n\nOne line item deserves pulling out of that list. Anthropic's prompting Claude Opus 5 guide says the model verifies its own work without being told to, and that explicit verification instructions tip it into over-verification, so the four verification patterns sitting in your CLAUDE.md now [cost more than they return](/blog/guides/reduce-ai-coding-tool-token-usage) on Opus 5.\n\nCheck what else reads that file first, because a subagent on an older model still needs them, and the cyber-flagged re-run on Opus 4.8 above is one path where you get the older model without asking.\n\nHere is where my reading of Opus 5 novel reasoning could be wrong, stated before somebody else states it for me.\n\nThe ARC-AGI-3 run used **High effort only**, because of a short testing window. A Max-effort figure could land later and move the headline up or down. The same results page already reports ARC-AGI-2 twice at two settings, 90.4% at Max against 88.3% at High, which is the cleanest available proof that effort is a free variable inside a published chart.\n\nEffort is not held constant across the launch claims generally. ARC-AGI-3 ran at High, ARC-AGI-1 and ARC-AGI-2 at Max, Terminal-Bench at Max. Reading rows across that table compares configurations as much as it compares models.\n\nThree more things I will not stand behind.\n\nEvery leaderboard figure in this post is stamped 2026-07-25 and was one day old when I wrote it. Boards move, and this one will.\n\nMove the workload, unless your traffic is security-adjacent. A cyber-flagged request re-runs on Opus 4.8, so on that traffic you are paying Opus 5 rates for last-generation output with no signal that it happened. Opus 5 is not the leap the headline sells, and on agentic coding it is third rather than first, but the gap is small enough that the lower recommended effort and the ARC-AGI-3 result still justify the move.\n\nThe change list is four lines long. Run claude update to v2.1.219 or later, delete the effort parameter from the config you carried over instead of translating it, delete the four verification patterns from CLAUDE.md, and revisit max_tokens if you tuned it with thinking disabled, because it caps thinking and response text together and the docs suggest starting at 64,000 for the top two rungs.\n\nThen settle the rest yourself. Sweep effort on your own repository with your own evals, and hold one level for the whole session, because effort shapes the rendered prompt and changing it mid-session throws away the cached prefixes from every earlier turn. Let a number you measured outrank a number somebody else published.\n\nThat is the actual takeaway. The Opus 5 novel reasoning number belongs to someone else's benchmark. The setting in your config belongs to you.", "url": "https://wpnews.pro/news/what-opus-5-s-novel-level-reasoning-benchmark-actually-means", "canonical_source": "https://rizz.dev/blog/meta-analysis/opus-5-novel-reasoning-benchmark", "published_at": "2026-07-25 00:00:00+00:00", "updated_at": "2026-07-25 15:06:10.575781+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research"], "entities": ["Opus 5", "ARC Prize Foundation", "ARC-AGI-3", "Anthropic", "Claude"], "alternates": {"html": "https://wpnews.pro/news/what-opus-5-s-novel-level-reasoning-benchmark-actually-means", "markdown": "https://wpnews.pro/news/what-opus-5-s-novel-level-reasoning-benchmark-actually-means.md", "text": "https://wpnews.pro/news/what-opus-5-s-novel-level-reasoning-benchmark-actually-means.txt", "jsonld": "https://wpnews.pro/news/what-opus-5-s-novel-level-reasoning-benchmark-actually-means.jsonld"}}