# What Claude Code's multi-agent code review costs and finds > Source: > Published: 2026-10-21 15:10:53+00:00 ## TL;DR **Claude Code’s multi-agent `/code-review max` caught 23 of 27 serious bugs on hard pull requests, against 21 for a single agent, but cost about $21 a review instead of 11 cents and took 56 minutes instead of 40 seconds.** - **The trade-off.** Two more bugs is close to this benchmark’s noise, and the multi-agent review also made twice as many claims you have to read. Still, one serious bug in production can cost far more than $21, so it comes down to what a missed bug costs your team. - **How I measured it.** 20 hard pull requests from a public benchmark, with Sonnet 5.5 doing the work in both settings to keep the bill manageable. - **What I’d do.** Start with`/code-review high` on Opus 5.5: across all 50 benchmark PRs it caught 51 of 66 serious bugs for about 36 cents each. If you run`max` , read its transcript to see which subagents actually ran, because some of mine skipped verification to stay under budget. Anthropic’s Claude Code team has [written down how it reviews code](https://claude.dev/blog/solving-code-review-at-anthropic/): several agents look for bugs at once, “then a separate group votes on each finding, and it only gets posted if enough of them agree it’s real.” Anthropic’s Deputy CISO, Jason Clinton, [calls narrow review agents](https://claude.com/resources/articles/how-anthropic-secures-its-ai-native-software-development-lifecycle) “much more effective than one mega-prompt or super security agent.” Those posts are about Anthropic’s managed service and its internal process, and neither gives a count of the bugs the agents find or miss per pull request, how much else they say, how long a review takes or what it costs. The local `/code-review` command that ships with Claude Code, which in the team’s words “runs a lighter version locally”, has the same shape at its `max` setting and is the one anyone can run, so I measured it. I ran Claude Code’s local `/code-review` command, not Anthropic’s managed Code Review service[1](#n1), with one agent and with its multi-agent fan-out, scored both on 20 pull requests from a public benchmark[5](#n5), and put them against Quickchat Review, the reviewer I built[4](#n4). The benchmark’s own score put the three leading reviews within 0.8 points of each other, so I also had two AI judges compare the reviews head to head[10](#n10). **With its agents on Sonnet 5.5, the fan-out does catch bugs. It does not catch all of them, it takes most of an hour, it adds twice as many claims that the answer key does not list, and it cost $20.95 per finished review.**[8](#n8) Quickchat Review works on the same idea, many narrow AI reviewers and a check before anything is posted. On Opus 5.5 the judges’ ratings do not separate it from the fan-out, and it got there for $4.61 and 8.2 minutes per pull request, under a quarter of the fan-out’s recorded price per finished review and about a seventh of its wall-clock time under our concurrency and rate limits. ## What I measured **The product.** Claude Code 2.1.303[1](#n1) and its local [`/code-review` command](https://code.claude.com/docs/en/code-review-local), a skill bundled with the CLI that works on any plan and bills to your usage or API key; Anthropic’s managed [Code Review service](https://code.claude.com/docs/en/code-review) is a separate product I did not test. **How it ran.** Headless, in a locked-down Docker sandbox with no GitHub token and no GitHub writes[2](#n2): ``` claude -p '/code-review high ...' --output-format json ``` In Claude Code’s own transcripts, `high` and `max` select two different review recipes, and the documentation describes neither[3](#n3). At `high` it starts one agent: `high effort → 8 inline angles → dedup (no verify) → ≤10 findings`. At `max` it fans out: `max effort → 5+5 angles × 8 candidates → 1-vote verify → sweep → ≤15 findings`, about ten finder agents, a verifier agent per candidate, then a sweep. To make `max` affordable on twenty PRs I forced every subagent onto Sonnet 5.5, capped each review at $30 and gave it 90 minutes[4](#n4); none of that is a shipped default, and neither are the rest of my settings[2](#n2). The two recipes differ in more than their number of agents, so this compares the two settings Claude Code ships, not parallelism alone[3](#n3). *The three review shapes compared in this post, each as its tool intends it.* **The benchmark.** Martian’s [Code Review Bench](https://github.com/withmartian/code-review-benchmark): 20 of its 50 PRs from Sentry, Grafana, Cal.com, Discourse and Keycloak, scored by Martian’s own pipeline with one LLM judge, GPT-6.1 Sol, for every reviewer[7](#n7). Its pipeline splits each review into claims and matches them against the key; a claim that matches nothing is an unmatched claim[7](#n7). Twenty and not fifty because the full set of `max` reviews would have cost about $1,000 and run for about 16 hours[5](#n5). The 20 are the PRs where Quickchat Review on two other models had done worst, a selection expected to favour every other reviewer; Claude Code `high` on Opus 5.5 and Quickchat Review on Sonnet 5.5 also ran on all 50[5](#n5). **Costs and times.** Claude Code’s dollar figures are the estimates it records for each session, apart from one recount labelled where it appears, and Quickchat Review’s are computed from token usage, both at list prices; every time is wall-clock with several reviews running at once[8](#n8)[9](#n9). `max` has three cost figures, each labelled where it appears: $20.95 per finished review, $19.90 over all 20 PRs in the charts, and about $440 actually spent[8](#n8). ## The fan-out matched two more serious issues than one agent Both runs reviewed the same 20 PRs with Sonnet 5.5 doing the work[4](#n4). The fan-out matched two more of the 27 serious answer-key issues and the same number of issues overall; two issues is about the scoring noise on this benchmark, so I cannot say whether that difference would repeat[7](#n7). Match counts are totals over the 20 PRs and the other rows are means; the `max` column’s time and cost rest on its 19 finished reviews, and its match and claim counts cover all 20, with the timed-out review counted as zero and five reviews scored from reports recovered from their transcripts[9](#n9). | Metric | `/code-review high` (1 agent) | `/code-review max` (16.5 agents a review) | |---|---|---| | Serious answer-key issues matched, 20 PRs together (27 in the key) | 21 | 23 | | All answer-key issues matched, 20 PRs together (65 in the key) | 47 | 47 | | Unmatched claims, mean per PR | 11.9 | 23.9 | | Wall-clock time, mean per finished review | 40 s | 56 min | | Recorded cost, mean per finished review | $0.11 (20 of 20 finished) | $20.95 (19 of 20 finished) | The fan-out still missed 4 of the 27 serious issues. It produced twice as many unmatched claims, and its recorded subtotal was 188 times one agent’s, $398.03 against $2.12[8](#n8). *On the same 20 hard PRs with Sonnet 5.5, Claude Code `max` matched 2 more serious answer-key issues than `high` and the same number overall. The chart’s $19.90 a review spreads the recorded subtotal over all 20 PRs; per finished review it is $20.95.* ## What happens inside a max review **The unforced configuration, on one small PR.** On browser-use #857, a 97-line change with five defects in our answer key[6](#n6), `/code-review high` on Opus 5.5 found all five in 70 seconds for 32 cents. `max` with Opus 5.5 on every agent[4](#n4) started 9 finder agents and 14 verifiers, made 608 API calls and had spent **$40.64**, by our recount of those calls[8](#n8), when our 30-minute limit stopped it at 31 minutes. It printed nothing, because Claude Code prints the review only when it finishes. *Cumulative cost of the all-Opus 5.5 `max` run on browser-use #857 until the 30-minute limit stopped it with nothing printed, with the `high` run on Opus 5.5 as the dot near the origin.* **The model running the recipe decides how much of it runs.** Same `max` prompt, same PR, a different model forced onto the review agent and every child: Opus 5.5 launched 9 finders and 14 verifiers, Sonnet 5.5 launched 5 finders and no verifiers, and Haiku 5.5 launched no finders and 5 verifiers[6](#n6). The Sonnet run cost $13.31 and took 32 minutes, the Haiku run $0.53 and 13.5 minutes, and both found all five defects. Each is a single run, and the Sonnet and Haiku runs had a $15 budget cap the agents could see, while the Opus run had none[6](#n6). *Opus 5.5 launched nearly all of the `max` recipe and printed nothing before the limit, while Sonnet 5.5 and Haiku 5.5 each skipped part of it and still found all five defects.* **Agents cut work when they can see the budget.** Five of the 19 finished `max` reviews launched no verifier at all. The agents can see a running budget line in their context, and four of the five named it as the reason, one in these words: “Per-candidate verifier agents were skipped because of budget.”[3](#n3) No review reached the $30 cap, so an uncapped `max` might run more verifiers and cost more[3](#n3). **Cost did not track PR size.** Finished reviews cost $15.17 to $26.09 each, with only a weak association with changed lines (r = 0.18)[8](#n8). *Claude Code `max` with Sonnet 5.5 agents spent $15 to $26 on every finished hard PR, close to the $30 cap it could see, while `high` on Sonnet 5.5 stayed between six and sixteen cents.* **The verifier checks that a bug is real, not that the PR introduced it.** In the all-Opus `max` run on browser-use #857 the verifiers assessed 18 candidate bugs, marked 16 confirmed and 2 plausible, and refuted none; two of the confirmed bugs existed before the PR[6](#n6). **Several reviews exited successfully without a usable report.** Five of the twenty `max` reviews returned a closing note instead of the review, for example: “The last background agents have finished. The report above stands, with no findings to add or remove.” Those five PRs delivered 2 of their 63 findings. One review ran into the 90-minute limit without a report, and 25 of 339 agents died on rate-limit errors from our API key, 23 of them inside 11 reviews that still reported success, so a check of the exit status alone would have missed all of this[9](#n9). ## The answer key could not separate the leaders The answer key did rank the reviews: `max` scored 30.0 on the benchmark’s F2, well below the leaders. It did not separate the leaders: Quickchat Review on Sonnet 5.5 at 43.9, Claude Code `high` on Opus 5.5 at 43.2 and Claude Code `high` on Sonnet 5.5 at 43.1, three very different setups within 0.8 points. Martian’s key holds about 3.7 entries per PR, the reviews made 8 to 26 claims per PR, and the key cannot tell a true claim it lacks from a wrong one[7](#n7). So I added a second instrument: a judge that reads two reviews of the same PR side by side, with the diff, the touched files and the answer key, and says which one a maintainer would rather receive, with a verdict each for recall, precision, finding quality and writing[10](#n10). Two judges from different providers, GPT-6.1 Sol and Claude Opus 5.5, rated 298 matchups each with identical prompts[10](#n10), and the results are on the Elo scale, over the reviews that exist[11](#n11). ## What the judges found The judges rated the fan-out highest, though not clearly above Quickchat Review on Opus 5.5. The five main configurations on the same 20 PRs: | Configuration | Serious issues matched (of 27) | All issues matched (of 65) | Unmatched claims per PR | Wall-clock time per review | Recorded cost per finished review | Answer-key F2 | Judges’ Elo | |---|---|---|---|---|---|---|---| | Claude Code `max` , Sonnet 5.5 agents | 23 | 47 | 23.9 | 56 min | $20.95 | 30.0 | **1666** | | **Quickchat Review, Opus 5.5** | 22 | 44 | 16.2 | 8.2 min | $4.61 | 35.0 | **1628** | | Claude Code `high` , Opus 5.5 (1 agent) | 22 | 43 | 9.8 | 78 s | $0.33 | 43.2 | 1518 | | **Quickchat Review, Sonnet 5.5** | 20 | 37 | **6.2** | 3.2 min | $1.17 | 43.9 | 1489 | | Claude Code `high` , Sonnet 5.5 (1 agent) | 21 | 47 | 11.9 | 40 s | $0.11 | 43.1 | 1398 | Match counts are totals over the 20 PRs, unmatched claims are means over all 20, and time and cost are means, for `max` over its 19 finished reviews[8](#n8). Rows on the same model compare complete configurations, which differ in more than the harness; Quickchat Review also ran six quality passes on GPT-6 Luna, under 2% of its cost[4](#n4). *The review the answer key scores last, Claude Code `max` with Sonnet 5.5 agents, is the one the judges rate highest.* The judges credited `max` with findings the answer key does not list: over their 194 recall verdicts on its pairs it won 162, tied 22 and lost 10. It also came last on writing, with 14 wins against 179 losses[11](#n11). Against `high` on Opus 5.5 it rates +148 (95% interval +64 to +241), about a 70% expected win rate under the fitted model, at 61 times the recorded cost; that margin comes mostly from the Opus judge, and the GPT judge on its own does not separate the two[11](#n11). It rates 38 points above Quickchat Review on Opus 5.5, with a 95% interval from −33 to +135: the interval includes zero, so these data do not say which of the two is better[11](#n11). *Among the five main setups, the judges rate Claude Code `max` with Sonnet 5.5 agents best at finding problems and worst at precision and at writing them up.* The two judges agreed on precision 76% of the time and on recall 70%, but on the overall winner only 56%, and each judge’s highest-rated configuration ran on its own model family[12](#n12). GPT favoured lean reviews and Opus thorough ones, so the overall ranking rests on how much a missed bug weighs against a wasted comment; the recall, precision and writing verdicts are where the two judges mostly agree. *The two judges named the same overall winner in only 56% of matchups, but agreed far more often on recall, precision and writing.* ## Start with `/code-review high` on Opus 5.5 For an inexpensive first pass, I would start with `/code-review high` on Opus 5.5. It reviewed every one of the 50 benchmark PRs in 42 to 151 seconds for about 36 cents and matched 51 of the 66 serious answer-key issues, so it also missed 15[5](#n5). On the twenty hard PRs the judges rated it 120 points above the same command on Sonnet 5.5 (1518 against 1398), a gap whose interval only just excludes zero[11](#n11), though the Sonnet 5.5 version costs about a third as much. Run it on your branch before you open a PR, which is the use its documentation describes. It gives no severity, no verdict and no clean split between bugs the PR introduced and bugs that were already there, and its prompt asks for at most ten findings unless you pass `--max-findings`, a request rather than an enforced limit[3](#n3). I would not run `max` on every PR. Anthropic’s model-configuration documentation says the `max` effort level [“may show diminishing returns and is prone to overthinking, so test before adopting it broadly”](https://code.claude.com/docs/en/model-config), and users have reported long, costly multi-agent review runs on Claude Code’s issue tracker[3](#n3). If you do try it headless, cap the budget (the agents can see it and may cut work to fit), set a timeout, keep the transcripts, and check that what came back is a review and not a closing note[9](#n9). ## How Quickchat Review compares, and where it differs Quickchat Review runs a planned set of narrow review passes, each looking for one kind of problem, then a validator meant to check every candidate before anything is posted[4](#n4). In cost and time it sits between one Claude Code agent and the fan-out. On Sonnet 5.5 it produced the fewest unmatched claims of the eight configurations I ran, 6.2 per PR against 11.9 for one Claude Code agent and 23.9 for the fan-out on the same model, and the judges ranked it first of all eight reviewers on precision and on writing; it also matched fewer answer-key issues, 37 against 47 for one Claude Code agent. On Opus 5.5 the judges’ ratings do not separate it from `max`, at $4.61 and 8.2 minutes per PR against $20.95 and 56 minutes, though it produced 16.2 unmatched claims per PR there, more than any other configuration in the five-row table except `max`[7](#n7). On all 50 PRs its Sonnet 5.5 build matched 49 of 66 serious issues at $1.32 a PR, against 51 of 66 at 36 cents for Claude Code `high` on Opus 5.5[5](#n5). Both versions give every finding a severity and end with a verdict, which none of the Claude Code runs do. *Cost per pull request on a log scale against the two judges’ combined Elo for all eight reviewers on the same 20 PRs, with a dashed line joining the best rating at each cost.* Quickchat AI will open-source Quickchat Review in the coming weeks. How its workflow differs from Claude Code’s, and what is still open: - **One finding per root cause, with a severity and a verdict.** It posts blockers, questions and nits, and decides approve or request changes, with approval withheld while CI is red. - **A validator meant to keep old bugs out.** It checks each candidate against the code and drops or flags problems that existed before the PR; on Opus 5.5 it ran on every PR, and on Sonnet 5.5 two of the 20 reviews skipped it[9](#n9) . Claude Code’s verifier checks whether a claim is true, not whether the PR introduced it[6](#n6) . - **Several model families.** The tested builds ran on Claude, GPT, DeepSeek and GLM models; on GLM-5.3 Flash it cost 60 cents a PR on the same twenty[8](#n8) . - **The harness must enforce the review stages it plans.** On Sonnet 5.5 Quickchat Review cut its own plan too: on two PRs it ran none of its review passes, wrote “I skipped the lens passes and the validator and reviewed the diff myself,” and approved one of them[9](#n9) . A step that only a prompt asks for can be skipped, in either tool; the fix is for the harness to launch each planned pass itself, count them and fail the review when one is missing. - **Not tested here:** follow-up rounds, replies to the author and repository-specific setup[15](#n15) . It is not cheaper than Claude Code’s single agent: on the same model it costs 11 to 14 times as much as `high`. On the answer key its F2 on Sonnet 5.5 is 0.8 points above `high`’s (43.9 against 43.1), a gap these data cannot separate, and on Opus 5.5 it trails (35.0 against 43.2). The judges preferred it over `high` on Opus 5.5 (+110, interval +31 to +196), though as with `max` that margin comes mostly from the Opus judge[11](#n11), and on Sonnet 5.5 for precision and writing. Whether that is worth the price depends on what a missed bug and a wasted comment each cost your team. ## Limitations This is one run per configuration on 20 PRs chosen where Quickchat Review had done worst[5](#n5), with costs as each tool records or computes them and times under our concurrency and rate limits[8](#n8)[9](#n9), Claude Code under our sandbox settings rather than an untouched default[2](#n2), LLM judges with no human check that share model families with some of the reviewers[12](#n12), and a builder of a competing reviewer, partly tuned on this benchmark’s PRs, running the tests[14](#n14). The rest are in note 15[15](#n15), and every raw result will be published with the open-source release[16](#n16). ## Frequently asked questions ### How much does Claude Code’s /code-review max cost per pull request? About $21 per finished review in my runs: $20.95 recorded on average over the 19 of 20 benchmark PRs whose review finished, with every subagent forced onto Sonnet 5.5 under a $30 cap, and $15.17 to $26.09 per finished review. Counting the timed-out review and the abandoned attempts that were re-run, the true spend was about $440, or about $22 a PR. The all-Opus 5.5 configuration had spent $40.64 on one 97-line PR when a 30-minute limit stopped it, and one agent at high effort cost $0.11 per PR on Sonnet 5.5 and $0.33 on Opus 5.5, with every review finished. ### How long does a Claude Code /code-review max review take? 56 minutes of wall-clock time on average over the 19 reviews that finished, with three to five reviews running at once on one API key whose rate limits are below Anthropic’s standard tier, and one review was still running at a 90-minute limit. The single-agent high setting took 40 seconds on average on Sonnet 5.5 and 78 seconds on Opus 5.5 on the same 20 PRs, and 42 to 151 seconds on Opus 5.5 across all 50 PRs of the benchmark. ### Does Claude Code’s multi-agent fan-out find more bugs than one agent? On 20 benchmark PRs with Sonnet 5.5 doing the work, the max fan-out matched 23 of the 27 Critical or High answer-key issues and one agent at high matched 21, and both matched 47 of the 65 issues overall. Two issues is about the scoring noise on this benchmark, and with one run per configuration I cannot say whether that difference would repeat. The fan-out produced twice as many claims that matched nothing in the key, 23.9 per PR against 11.9, at 188 times the recorded cost, $398.03 against $2.12. ### What is the difference between /code-review high and max in Claude Code? In Claude Code 2.1.303 they select different review recipes, which the documentation does not describe and which depend on the model as well as the effort level. At high, the recipe asks one agent to work through eight review angles in its own context, deduplicate and report at most ten findings, with no verifier stage. At max, it asks the orchestrator to launch about ten finder subagents, then one verifier subagent per candidate, where a single vote that does not refute a finding keeps it, then a sweep, and to report at most fifteen findings; in my runs the model sometimes launched fewer agents than that. The —max-findings option replaces both limits. ### Is the local /code-review command the same as Anthropic’s managed Code Review? No. The local /code-review command is a skill bundled with the Claude Code CLI that runs in your own session on any plan and bills as usage or to an API key, while Anthropic’s managed Code Review is a GitHub App for Team and Enterprise plans that runs on Anthropic’s infrastructure and adds severity tags. Anthropic’s documentation calls them two separate tools, and this post measured only the local command. ### Which Claude Code review setting is good enough for most pull requests? For an inexpensive first pass, /code-review high on Opus 5.5: it reviewed all 50 benchmark PRs in 42 to 151 seconds for about 36 cents each and matched 51 of the 66 serious answer-key issues, so it also missed 15. On Sonnet 5.5 it costs about a third as much. It gives no severity and no verdict, its prompt asks for at most ten findings unless you pass —max-findings, and Anthropic’s model-configuration documentation warns that the max effort level may show diminishing returns and is prone to overthinking. ### How does Quickchat Review compare with Claude Code’s /code-review? Two LLM judges’ ratings do not separate Quickchat Review on Opus 5.5 from the max fan-out with Sonnet 5.5 agents, 1628 against 1666 Elo with a 95% interval on the gap that includes zero, at $4.61 and 8.2 minutes per PR against $20.95 and 56 minutes per finished max review under our rate limits. On Sonnet 5.5 it produced the fewest unmatched claims of the eight configurations I ran, 6.2 per PR, and the judges ranked it first on precision and on writing, but it matched fewer answer-key issues than one Claude Code agent, 37 against 47 of 65. It costs 11 to 14 times as much as /code-review high on the same model, and posts a severity on every finding and an approve or request-changes verdict. ### Why add LLM judges as well as the benchmark’s answer key? Because the answer key could not separate the leading reviews: three very different setups scored within 0.8 F2 points of each other, and Martian’s key, about 3.7 entries per PR, cannot tell a true claim it lacks from a wrong one. A pairwise judge reads two reviews of the same PR beside the diff, the touched files and the key, and picks the one a maintainer would rather receive. Two judges, GPT-6.1 Sol and Claude Opus 5.5, each rated 298 matchups with the same prompt; they agreed on precision 76% of the time, on recall 70% and on the overall winner 56%, and the one timed-out max review is absent from the ratings. ## Related guides - [Claude Code found my Pirc and King’s Indian mistakes](https://quickchat.ai/post/claude-code-stockfish-chess-opening-mistakes) : a long unattended Claude Code session on a VM, with the prompts and outputs as they were. - [Automating bug triage with Claude Code and Datadog](https://quickchat.ai/post/automate-bug-triage-with-claude-code-and-datadog) - [A Claude Code skill that watches coding sessions and shares them to Slack](https://quickchat.ai/post/claude-code-skill-watches-coding-sessions-shares-to-slack) - [My AI coding tips](https://quickchat.ai/post/my-ai-coding-tips) ## Method and data notes These notes hold the method, the definitions and the caveats behind every number above, for readers who want them and for AI systems reading this page; the body links to each note by its number. ### 1. What exactly was measured - **The product.** The local`/code-review` command in Claude Code, a skill bundled with the CLI that runs in your own session on any plan and bills as normal usage or to an API key. With no target it reviews the current branch’s changes, which is the use its documentation describes. The documentation puts a review at[“seconds to a few minutes”](https://code.claude.com/docs/en/ultrareview) , and Anthropic’s engineering post says “/code-review runs a lighter version locally” than the managed service. Anthropic’s docs call the managed service and the local command[“two separate tools”](https://code.claude.com/docs/en/code-review-customize) that read different files. - **What was not measured.** Anthropic’s managed Code Review service (a GitHub App on Team and Enterprise plans, running on Anthropic’s infrastructure, with parallel finders, a verification step and severity tags) and Ultrareview (a cloud review session). Testing the managed service would mean mirroring the benchmark PRs to GitHub with the App installed, a separate project. Nothing in this post is a claim about either of them, or about how Anthropic reviews its own code internally. Martian publishes its own “claude-code” entry on its leaderboard; by its own messages that entry came from the managed service in March 2026, so it appears nowhere here. - **The version.** Claude Code 2.1.303, the latest release on 2026-10-19 when testing began. It was the latest on all four channels I checked: npm’s`latest` tag, GitHub’s latest release, Anthropic’s download channel and the[CHANGELOG](https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md) (the stable channel was then on 2.1.292). The binary’s sha256 and size matched Anthropic’s release manifest byte for byte, and every line of every session transcript carries`"version": "2.1.303"` . Version 2.1.304 shipped 27 minutes into the 20-PR run; its changelog lists changes to the managed Code Review service and none to the local`/code-review` command, and every run in this post used 2.1.303. - **Effort levels.** Only`high` and`max` were measured.`low` ,`medium` and`xhigh` were not run. ### 2. How Claude Code ran - **Invocation.** The skill alone, typed as a user would type it, with nothing added to the prompt: no PR title or description, no extra prompt, no findings-reporting system prompt, no MCP tool and no permission mode. The benchmark harness’s usual Claude runner appends a findings-reporting prompt; we removed it for these runs. The settings around the skill were ours, and they are part of the measurement: the budget caps and timeouts in the commands below ($5 and 30 minutes for`high` , $30 and 90 minutes for`max` ), the forced subagent model on`max` (`CLAUDE_CODE_SUBAGENT_MODEL` with`CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1` , note 4), feature-flag fetching off, and default headless permissions. None of these is a shipped default. ``` # high, one agent ANTHROPIC_MODEL=claude-sonnet-5-5 \ claude -p '/code-review high ...' --output-format json --max-budget-usd 5 # (and ANTHROPIC_MODEL=claude-opus-5-5 for the Opus 5.5 arm) # max, every subagent forced to Sonnet 5.5 ANTHROPIC_MODEL=claude-opus-5-5 \ CLAUDE_CODE_SUBAGENT_MODEL=claude-sonnet-5-5 \ CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1 \ claude -p '/code-review max ...' --output-format json --max-budget-usd 30 ``` - **Sandbox.** Each review ran in its own locked-down local Docker container, never against GitHub, with no GitHub token and no GitHub writes. The repository copy held no commits beyond the PR’s base and head (on the one PR we counted, 1,230 commits in the copy and 1,230 in the history up to the head), so the sandbox’s repository history held nothing from the project’s future. That check covers the repository copy and nothing else; it says nothing about what any model saw in training. The API key reached the container through an env file, never on the command line. - **Permissions.** Claude Code’s default`-p` permissions, unchanged. Under them WebSearch and WebFetch were refused, and some shell commands needed approval that a headless run cannot give (for example`git -C diff` , while plain`git diff` passes; compound and`python` commands were also refused). The`max` reviews logged 645 refused tool calls in the 14 of the 20 reviews whose output records them, and the two`high` arms 2 and 26 across their 20 reviews each. A refused call costs the agent a turn. Interactive use, where a person approves calls, may behave differently; no permissions-matched run was made. - **Feature flags.** Claude Code’s feature-flag fetching was turned off for the scored runs. On browser-use #857 we ran`high` on Opus 5.5 both ways; the review prompt in the transcripts was byte-identical, and both runs found all five defects. - **Output.** With`--output-format json` , Claude Code returns the review’s final message. Each finding (file, line, text) went into the scoring pipeline in the same shape as a posted review comment. ### 3. The review recipes we observed - **Where they come from.** Each review agent’s prompt in our transcripts opens with a header line naming its recipe. The two we quote are exactly as they appear in 2.1.303 transcripts: - `high effort → 8 inline angles → dedup (no verify) → ≤10 findings` , with the instruction to run the eight angles in sequence in its own context rather than spawn subagents; - `max effort → 5+5 angles × 8 candidates → 1-vote verify → sweep → ≤15 findings` : about ten finder subagents (on browser-use, five correctness angles, three cleanup angles, an altitude angle and a CLAUDE.md conventions angle, which was skipped because the repository has no CLAUDE.md), then one verifier subagent per candidate, where a single vote that does not refute a finding keeps it, then a sweep for gaps. The two Claude Code columns of the body’s first figure draw these recipes as the prompt headers describe them, each as its tool intends it; how much of each recipe actually ran depended on the model (note 6). - **What the `high`/` max` comparison is.**`high` and`max` differ in prompt, scope, verification, output cap and orchestration, so the body compares the two recipes Claude Code ships, under our settings. It does not isolate what parallel agents add to a fixed review procedure. - **Undocumented and version-specific.** Anthropic’s documentation describes effort only in general terms (“From`medium` through`max` , the review broadens coverage”), and does not publish these recipes. Anthropic’s changelog shows that the recipe depends on the model as well as the effort level: 2.1.274 moved models without tuned settings to “leaner inline review prompts” instead of many review subagents, and 2.1.290 names Opus 5.5 and Sonnet 5.5 among those models. In our transcripts Opus 5.5 and Sonnet 5.5 showed the same`high` recipe header, and`max` on Opus 5.5 still fanned out. Recipes have changed between releases before and may change again, so these observations hold for 2.1.303 only. - **What users have reported.** Issues on Claude Code’s own tracker describe the same behaviour one run at a time:`/review` (an alias of`/code-review` ) at high effort growing to about ten times its token cost through a multi-agent fan-out ([#89249](https://github.com/anthropics/claude-code/issues/89249) ),`/code-review max` running about 65 minutes and 432,000 tokens with no result ([#96708](https://github.com/anthropics/claude-code/issues/96708) ), and the separate code-review plugin skipping its review agents on Sonnet 5.5 while still reporting no issues ([#100401](https://github.com/anthropics/claude-code/issues/100401) ). This post adds a scored measurement over 20 PRs with recorded costs. - **The caps are prompt instructions.** “≤10 findings” and “≤15 findings” are text in the prompt, not limits the harness enforces: one`high` review on Sonnet 5.5 returned 12 findings. Claude Code’s`--max-findings` option (a number,`all` or`default` , from 2.1.288) replaces these limits; we did not pass it, so the caps stayed as shipped.`max` hit its 15-finding cap on 14 of 19 finished reviews, and`high` on Opus 5.5 hit its 10-finding cap on 12 of 20. - **No severity, no verdict.** None of the local command’s findings in our runs stated a severity, and no run ended with an approve or request-changes decision. The managed service has severity tags; the local command does not. - **Agents per review.** The`max` reviews ran 8 to 27 agents each, 16.5 on average: 314 agents over the 19 finished reviews, counting each review’s own review agent and the finders and verifiers it launched, or 339 over all 20 reviews with the timed-out one included. Five of the 19 finished reviews launched no verifier subagent; three more launched one batched or bounded verifier instead of one per candidate, and a fourth batched two to four candidates per verifier. - **The budget line.** When`--max-budget-usd` is set, the agents’ context carries running budget records of the form`USD budget: $1.35/$30; $28.65 remaining` (rounded here). We counted 286 to 585 of them per`max` review on the 20 PRs, in 337 of the 339 agents, and none in the uncapped all-Opus run on browser-use #857. Agents mention a budget in 14 of the 20`max` reviews, and none of the 70`high` reviews we ran on Martian mentions a cost budget. Of the five reviews with no verifier, four named the budget in their summary: “Per-candidate verifier agents were skipped because of budget.” (sentry-greptile #3, $22.45), “because the budget didn’t allow it” (cal.com #11059), “to stay inside the budget” (cal.com #10967) and “(budget)” (keycloak #33832); the fifth (grafana #107534) delivered only a closing note, and its orchestrator had told its finders “Budget is tight: please stop exploring”. No review reached its cap (the highest cost $26.09 of $30), so the agents cut work while budget remained.**The $30 cap we set is therefore part of the measurement:** an uncapped`max` might run more verifiers and cost more. ### 4. Every configuration, with its billed model | Configuration | Billed model(s) | Settings | |---|---|---| | Claude Code `high` , Sonnet 5.5 | Sonnet 5.5 on every call | `/code-review high` , $5 cap, 30-minute timeout, 4 reviews at a time | | Claude Code `high` , Opus 5.5 | Opus 5.5 on every call | `/code-review high` , $5 cap, 30-minute timeout, 3 at a time | | Claude Code `max` , Sonnet 5.5 agents | Sonnet 5.5 for every subagent ($396.29); Opus 5.5 main session ($1.74, 0.44%) | main model Opus 5.5, every subagent forced to Sonnet 5.5, $30 cap, 90-minute timeout, 3 to 5 at a time | | Claude Code `max` , all Opus 5.5 (browser-use only) | Opus 5.5 | no subagent override, no budget cap, the benchmark harness’s standard 30-minute limit | | Claude Code `max` , Sonnet 5.5 or Haiku 5.5 agents (browser-use only) | the forced model | main model Opus 5.5, $15 cap, 60-minute timeout | | Quickchat Review, Sonnet 5.5 | Sonnet 5.5, plus GPT-6 Luna for six quality passes ($0.37 of $23.35) | development build of 2026-10-19, generic default profile, 2 PRs at a time | | Quickchat Review, Opus 5.5 | Opus 5.5, plus GPT-6 Luna for six quality passes ($0.41 of $92.26) | the same build and profile | | Quickchat Review, GPT-6.1 Sol | GPT-6.1 Sol | a build of 2026-10-04 | | Quickchat Review, DeepSeek V4.1 Flash; GLM-5.3 Flash | the named model | older builds, runs of 2026-10-14 and 2026-10-15 | - **The forced subagent model.**`CLAUDE_CODE_SUBAGENT_MODEL` is only a default;`CLAUDE_CODE_SUBAGENT_MODEL_FORCE=1` (Claude Code 2.1.257 and later, per its changelog) puts every subagent on one model, the review’s own skill subagent included. The`max` recipe is picked from the main model, so the main model stayed Opus 5.5 to keep the Opus recipe, and every API response in the transcripts confirms the recipe header and the forced model. The main session itself makes no turns in a`-p` review; it billed only when late background agents reported back to it after the review agent had finished (see note 9).**`max` with Sonnet 5.5 agents is not Claude Code’s default** ; the unforced all-Opus configuration appears only in the one-PR case study, and “default” or “unforced” there means our API setup with`ANTHROPIC_MODEL=claude-opus-5-5` and no subagent override, since the model a plan or account starts from can differ. - **Quickchat Review.** The pull-request reviewer Quickchat AI builds: a planned set of narrow review passes (“lenses”, about 15 per PR on these runs) chosen per PR, a validator that checks every candidate against the code, and one main session that posts severity-tagged comments and a verdict. In the Sonnet 5.5 and Opus 5.5 arms every slot ran on the named model except six code-quality lenses, which ran on GPT-6 Luna (31% of lens runs in the Sonnet arm; 1.6% of the Sonnet arm’s cost and 0.44% of the Opus arm’s, from the table above). The GPT-6.1 Sol, DeepSeek and GLM arms are earlier builds, so model and build are confounded across Quickchat Review’s rows; the two same-model comparisons with Claude Code (Sonnet 5.5 and Opus 5.5) are on one build. The cost-against-rating chart in the body plots all eight configurations, the three older builds included. - **What differs beside the model.** A same-model row pair compares two complete configurations that share a primary model, not two harnesses with everything else equal. Quickchat Review read the PR’s title and description and the unpacked source of the third-party packages the PR imports (downloaded, never installed or run), ran six passes on GPT-6 Luna, and ran its tools under its own sandbox permissions. Claude Code read only`...` , because that is what its command takes, under default headless permissions that refused some of its tool calls (note 2). The Elo judges saw the title and description for every pair. ### 5. Data selection - **The benchmark.** Martian’s[Code Review Bench](https://github.com/withmartian/code-review-benchmark) , offline set (MIT licence): 50 PRs from five open-source projects (Sentry, Grafana, Keycloak, Discourse and Cal.com), with 173 answer-key (“golden”) comments, each labelled with a category and a severity. The PRs are the ones the benchmark’s own forks carry, so every reviewer saw the code every published tool reviewed. - **Why 20 and not 50.**`max` was too expensive and too slow to run on all 50. With every subagent on Sonnet 5.5 a finished review cost $20.95 on average and took 56 minutes of wall-clock time on average; the one all-Opus run had spent $40.64 on a 97-line PR when our 30-minute limit stopped it, and we did not measure that configuration’s average cost, so there is no estimate for it over 50 PRs. All 50 would have cost about $1,000 with Sonnet agents (50 × $20.95 = $1,048). Our API key’s rate limits allowed three`max` reviews at once (note 9), so 50 reviews would have taken about 16 hours of non-stop running (50 × 56 minutes ÷ 3; the 20 took about six). So`max` ran on the 20 PRs where Quickchat Review had done worst, and Claude Code`high` on Opus 5.5 and four Quickchat Review configurations also ran on all 50, where the selection plays no part. - **The 20 PRs.** Chosen before any Claude Code run, from earlier runs of Quickchat Review on all 50 PRs with three models (GPT-6.1 Sol, DeepSeek V4.1 Flash and GLM-5.3 Flash). Each PR got a hardness score: 3 times the severity-weighted answer-key issues that the GPT and DeepSeek runs missed (Critical 4, High 3, Medium 2, Low 1, counting only issues that our own earlier reading of the key judged to describe a real defect introduced by the PR), plus 10 times each of those two models’ shortfall below its own mean recall on that PR. The top 20 by that score form the set. They include every PR where either model missed an issue our reading counted as real, and every one has GPT, DeepSeek or both below their mean recall. Median size 306.5 changed lines (287 across all 50), up to 1,282. On these 20 PRs the key holds 74 comments (3.7 per PR), 65 of them in Martian’s default Core profile, and 27 labelled Critical or High. - **What the selection does.** It picks the PRs where Quickchat Review on two earlier configurations (GPT-6.1 Sol and DeepSeek V4.1 Flash) did worst. Regression to the mean makes every other reviewer expected to look better there, Claude Code included, but shared failure modes and build changes mean the size of that effect on the current comparisons is unknown, and it is not a random sample. The body’s “expected to favour every other reviewer” means exactly this: an expected direction, with an unmeasured size. - **A configuration the selection did not use.** Quickchat Review on GLM-5.3 Flash played no part in the selection score, so among my earlier runs it is the one the selection did not act on directly; it shares a harness with the two that did. No Claude Code result was used to choose anything. - **The full 50.** To check the selection, Claude Code`high` on Opus 5.5 ran on all 50 PRs in the same configuration: 50 of 50 finished, 42 to 151 seconds each (82 on average), $18.22 in total ($0.36 a PR), F2 44.9 on the Core profile, with 51 of the 66 Critical or High answer-key issues matched. Whether 51 of 66, which also means 15 missed, is enough for a given repository or risk is not something this benchmark can say. Quickchat Review on Sonnet 5.5 also ran the other 30 PRs, for a 50-PR figure of F2 47.4, 49 of 66 Critical or High and $1.32 a PR (3.6 minutes a review on the other 30), with 3 of its 50 reviews running none of its passes. Quickchat Review’s three earlier runs cover all 50 too (F2 43.5 on DeepSeek V4.1 Flash, 40.9 on GPT-6.1 Sol, 37.9 on GLM-5.3 Flash). There is no same-model pair on the full 50 (Claude Code ran Opus 5.5 there, Quickchat Review Sonnet 5.5), so those rows compare harness and model together. Running all 50 removes the 20-PR selection rule and nothing else: earlier tuning and whatever the models saw in training remain (note 14). The full scores are in note 7. The other configurations (`max` ,`high` on Sonnet 5.5 and Quickchat Review on Opus 5.5) did not run on the other 30. - **One run per configuration.** Each configuration reviewed each PR once. Run-to-run variance was not measured. ### 6. The single-PR case study: browser-use #857 - **The PR.**[browser-use/browser-use#857](https://github.com/browser-use/browser-use/pull/857) , a switch from Playwright to Patchright: 11 files, +39 and −58 lines (97 changed). It comes from[AACR-Bench](https://arxiv.org/abs/2601.19494) , whose dataset is Apache-2.0. The answer key is our own for this PR, not AACR-Bench’s reference comments: a blind hunt for defects from the code, plus every comment our earlier reviews of it posted, each candidate checked against the code by AI agents with no human check. It lists five defects:`channel='chrome'` requires a system Chrome (high); the headful viewport (medium); the Chrome channel is sent to Firefox and WebKit (medium); the Agent overwrites`disable_security` (low); closed shadow roots are hidden (low). - **Why this PR.** It was the PR where an older build of Quickchat Review (on GPT-6.1 Sol) did worst in an earlier benchmark run, 11% severity-weighted recall, while an earlier Claude Code (2.1.283) found all five. It was picked as a PR where Claude Code was known to do well and Quickchat Review badly. - **How it was scored.** By reading each review against the five defects, with no scoring-model call; like the key, that reading had no human check. The result is one PR: an illustration, not a benchmark score. - **Results.** Configuration Cost Time Result `high` , Opus 5.5 (two runs, flags off and on)$0.32 / $0.27 70 s / 61 s 9 findings each, all 5 defects (two of them in one finding) `max` , all Opus 5.5 (no subagent override)$40.64 stopped at 31 min (1,860 s) nothing printed `max` , every subagent Sonnet 5.5$13.31 32 min 13 findings, all 5 defects `max` , every subagent Haiku 5.5$0.53 13.5 min 12 findings, all 5 defects, about 3 of 12 pre-existing Quickchat Review, Opus 5.5 (5 of 15 lens runs on GPT-6 Luna, $0.02) $4.23 7.9 min request changes, 10 comments (2 blockers, 5 questions, 3 nits), all 5 defects - **Inside the all-Opus `max` run.** One orchestrator, 9 finders and 14 verifiers; 608 API calls; 1.1 million output tokens, 36.9 million cache-read tokens and 2.3 million cache-write tokens. The transcripts held 18 verifier-approved candidates (16 confirmed, 2 plausible, none refuted) covering all five defects plus one more real issue the PR introduces (Patchright runs`page.evaluate` in an isolated world, so the DOM script loses`window.getEventListeners` ). Because Claude Code prints the review only when it finishes, none of that reached the output. - **The verify step and old bugs.** Two of the confirmed candidates (`permissions.query` called unbound, and a`TypeError` from`**browser.config` in`BrowserContext.__init__` ) are identical at the merge base, so they existed before the PR. Claude Code’s recipe puts bugs in unchanged lines of a touched function in scope by design; the verifier checks that a claim is true, not that the PR introduced it, so this is a difference of scope rather than a verifier mistake. Quickchat Review’s validator makes that cut, and on this PR it also held back the isolated-world issue as doubtful (kept, not posted) because it had no Patchright source to check against. - **The orchestrator comparison.** The three`max` rows above share the`max` recipe from the Opus 5.5 main model; the forced subagent model, which changes the review agent and every child, is what differs, along with the budget (the Opus run had none) and whether the run finished. One run per configuration, so the comparison illustrates the variation rather than measuring it. The`max` recipe is an instruction the orchestrating model may or may not follow: Opus 5.5 ran it fully and printed nothing before the limit, while Sonnet 5.5 and Haiku 5.5 each skipped part of it and still found all five defects.Review agent and children on Finders Verifiers Cost Time Opus 5.5 9 14 $40.64 stopped at 31 min Sonnet 5.5 5 0 $13.31 32 min Haiku 5.5 0 5 $0.53 13.5 min The Sonnet-forced run launched 5 correctness finders and a gap sweep and no verifiers, and its orchestrator made 59 tool calls itself. The Haiku-forced run launched no finder subagents (its orchestrator found everything itself) and then 5 verifiers. Both of those runs were capped at $15 and carried the budget line from note 3; the all-Opus run was uncapped. ### 7. Scoring on the answer key, with the benchmark’s own numbers - **The pipeline.** Martian’s own steps, run unmodified at a pinned commit of their repository: an LLM extracts the distinct issues from each review’s comments, groups duplicates, then judges every (answer-key comment, extracted issue) pair as the same underlying issue or not. This post calls each extracted issue a claim; one posted comment can hold several claims, and several comments can hold one. Precision is matched claims over all claims; recall is matched answer-key comments over all answer-key comments. F2 weights recall four times as heavily as precision; Martian’s documentation describes it as reflecting the cost asymmetry “where missed bugs are worse than false alarms,” and its dashboard lets the reader move the weight. The body’s “answer-key F2” column is this recall-weighted score. Martian’s answer-key comments carry no file or line, so matching is by meaning. - **The judge.** GPT-6.1 Sol (an OpenAI model) at low reasoning effort, the same judge for every reviewer, in one combined scoring pass over all arms. Low effort follows the published setups (Martian’s published judges run with little or no reasoning, and its 30-second per-call timeout leaves no room for more). - **Comparing with the leaderboard.** Martian’s[public leaderboard](https://codereview.withmartian.com) (data and code in the repository linked in note 5) is scored with other judge models: Claude Opus 4.5, Claude Sonnet 4.5 and GPT-5.2. Our numbers therefore compare with each other and with the re-scored reference rows below, not directly with the leaderboard’s, because scores from different judges should not be ranked together. - **The Core profile.** Martian’s default, the answer-key comments in the categories bug, security, concurrency, data, api, perf, test_gap and doc_defect: 65 of the 74 comments on these 20 PRs (158 of 173 on all 50). Under Martian’s Strict and All profiles the scores move by a few points (table below); the three F2 leaders stay within about three points of each other under all three. - **“Serious”.** The answer-key comments Martian labels Critical or High: 27 on these 20 PRs, 66 on all 50. - **“Unmatched claims”.** Claims that matched no answer-key comment. The body reports them as means per PR (23.9 and 11.9 are 477 and 238 claims over 20 PRs). Martian counts them as false positives. They are not necessarily wrong: a claim can be true and simply absent from a key of about 3.7 entries per PR, and the key cannot tell that case from a wrong claim. That is why the pairwise judge exists (note 10). - **Quickchat Review on Opus 5.5’s unmatched claims.** 16.2 per PR on the 20 PRs (324 over 20), more than every configuration in the body’s main table except`max` ; across all eight configurations, only`max` and the GLM-5.3 Flash build produced more. The same passes post about 15 comments a review on Opus 5.5 against about 6 on Sonnet 5.5, and Martian’s extractor turned the Opus reviews into 18.4 claims per PR. - **Full scores, 20 PRs.** Precision, recall, F1 and F2 on the Core profile, then F2 under the Strict and All profiles.Reviewer Precision Recall F1 F2 Matched (of 65) Unmatched claims Critical/High (of 27) Strict F2 All F2 Quickchat Review, Sonnet 5.5 23.0 56.9 32.7 43.9 37 124 20 45.2 43.5 Claude Code `high` , Opus 5.518.1 66.2 28.4 43.2 43 195 22 42.2 45.4 Claude Code `high` , Sonnet 5.516.5 72.3 26.9 43.1 47 238 21 43.1 42.8 Quickchat Review, GLM-5.3 Flash 12.3 72.3 21.0 36.6 47 335 22 34.7 38.7 Quickchat Review, Opus 5.5 12.0 67.7 20.3 35.0 44 324 22 33.8 36.6 Quickchat Review, DeepSeek V4.1 Flash 13.1 58.5 21.4 34.5 38 252 21 35.8 35.6 Quickchat Review, GPT-6.1 Sol 12.4 49.2 19.8 30.8 32 227 15 30.0 30.5 Claude Code `max` , Sonnet 5.5 agents9.0 72.3 16.0 30.0 47 477 23 27.5 32.6 - **Full scores, all 50 PRs, Core profile.** Every configuration that ran on all 50, scored in one combined pass with the same judge. The first two rows are not our runs: they are the comments Qodo (`qodo-extended-v2` ) and cubic (`cubic-v2` ) posted on these PRs, as Martian’s benchmark data stores them, re-scored by our judge. They post about 3.5 claims per PR, against 8 to 19 for the configurations we ran. Every true but unlisted claim counts against precision, so some of their precision lead is volume; how much of it is volume and how much is fewer wrong claims, this scoring cannot say.Reviewer Precision Recall F1 F2 Matched (of 158) Unmatched claims Critical/High (of 66) Cost (50 PRs) qodo-extended-v2 (its published comments, re-scored by our judge) 56.2 60.1 58.1 59.3 95 74 46 not measured cubic-v2 (its published comments, re-scored by our judge) 54.3 60.1 57.1 58.9 95 80 42 not measured Quickchat Review, Sonnet 5.5 23.6 63.3 34.4 47.4 100 323 49 $66.17 Claude Code `high` , Opus 5.518.4 70.3 29.1 44.9 111 493 51 $18.22 Quickchat Review, DeepSeek V4.1 Flash 16.3 74.7 26.7 43.5 118 607 55 $50.55 Quickchat Review, GPT-6.1 Sol 16.2 65.8 26.1 40.9 104 536 48 $142.03 Quickchat Review, GLM-5.3 Flash 12.7 75.3 21.7 37.9 119 819 53 $34.50 Quickchat Review on Sonnet 5.5’s F1 is computed from its precision and recall; every other value is the pipeline’s output. Martian’s published “claude-code” entry is left out (note 1). - **Noise.** On 20 PRs we treat about ±2 answer-key issues per arm as noise. That is a prior from earlier repeated scoring of this benchmark, not a variance measured on these runs, which have one run per arm and no paired-difference estimate. So 23 against 21 serious issues, or F2 43.9 against 43.1, are differences these data cannot separate, and they say nothing about how reliably such a difference would repeat. Where the body says a gap “cannot be separated” or that it “cannot say whether a difference would repeat”, this is the basis. ### 8. Cost and time accounting - **Claude Code’s figures are its own, with one exception.** Every Claude Code dollar figure except the $40.64 is the client-side list-price estimate Claude Code records for the session, which its[CLI documentation](https://code.claude.com/docs/en/cli-reference) says “can differ from your bill”. List prices per million tokens, from Anthropic’s[pricing page](https://platform.claude.com/docs/en/about-claude/pricing) as of 2026-10-21: Opus 5.5 $4 input, $20 output, $0.20 cache read, $5 cache write (5 minutes); Sonnet 5.5 $2, $10, $0.10, $2.50; Haiku 5.5 $0.10 input and $0.50 output at its standard context tier. - **The `max` run’s totals, and which number is which.** $398.03 recorded for the 19 reviews that finished ($396.29 Sonnet 5.5, $1.74 Opus 5.5). Per finished review that is $20.95, the figure this post’s text and tables use. “$19.90 per review”, which the charts’ cost axes use (the five-panel headline chart and the cost-against-rating scatter, where`max` is plotted at $19.90), is $398.03 over all 20 attempted PRs, which counts the timed-out review as free. The money actually spent was about $440: the timed-out review left no cost record, and the ledger kept during the run put it at about $23 (its transcript’s last budget line reads $24.42, so the chart shows it at about $24); the abandoned attempts that were re-run cost about $20 (note 9). The $440 was built from the ledger’s $23, and either figure for the timeout leaves it at about $440, or about $22 a PR. - **Ratios.** “188 times” is $398.03 over $2.12, recorded subtotal over recorded subtotal on the same 20 PRs; it is not the ratio of complete experiment costs, and the rounded per-review figures ($20.95 or $19.90 over $0.11) do not reproduce it exactly. Against`high` on Opus 5.5 ($6.53) it is 61 times. The body’s “under a quarter of the fan-out’s recorded price” is $4.61 against $20.95 per finished review, and “about a seventh of its wall-clock time” is 8.2 minutes against 56. - **The $40.64.** Claude Code printed no cost when the all-Opus`max` run was stopped, so $40.64 is our recount of its 608 API responses at Opus 5.5 list prices (each response counted once). Of it, output tokens are 54%, cache writes 28% and cache reads 18%. By this recount it passed $24 at about 17 minutes and $39.29 at about 29. The cost-meter chart in the body is this recount plotted over time. - **Cost against PR size.** Finished`max` reviews cost $15.17 to $26.09, with a Pearson correlation of 0.18 against changed lines;`high` on Sonnet 5.5 cost $0.06 to $0.16 (correlation 0.30). On 19 and 20 points these are weak associations, not evidence that cost is independent of size, and changed lines are a coarse proxy for what a review has to read. In the cost-against-size chart, the hollow points are the five`max` reports recovered from transcripts (note 9) and the cross is the timed-out review, plotted at about $24, the last budget line in its transcript rather than a recorded cost. - **Quickchat Review’s costs.** Computed by the benchmark’s own pricer from each run’s token usage at each model’s list API price. The GPT-6.1 Sol arm ran on a ChatGPT subscription login, so its figure is an API-equivalent estimate, not a bill. The “60 cents a PR” for GLM-5.3 Flash is $11.98 over the 20 PRs. - **Time.** Wall-clock time per review inside the sandbox, start to finish. The`max` reviews ran three to five at a time on one API key whose limits are below Anthropic’s standard tier, so some of their time went to waiting out rate limits (note 9); the`high` arms ran three or four at a time and Quickchat Review two at a time. No time in this post was measured at matched concurrency or without throttling, so the ratios between tools are ratios of our runs, not of the tools’ intrinsic speed. - **Not counted.** The cost of judging ([note 13](#n13) ), and of the earlier runs used to select the PRs. ### 9. Failures, and how they were scored - **Closing notes instead of reviews.** In`-p` mode Claude Code returns only the session’s last message. In a`max` review, verifiers still running in the background report to the main session after the review agent has written its report, and the main session (Opus 5.5) then writes a closing note such as “The last background agents have finished. The report above stands, with no findings to add or remove.” That note is all a script receives. It happened on 5 of the 20 reviews, which delivered 2 of their 63 findings.**Scoring rule:** we scored the review agent’s own last findings report, recovered from the transcripts, and kept the delivered count beside it. We computed no separate as-returned score, so the`max` match counts in the body’s tables are what the reviews found, not what a script would have got back. This rule favours`max` ; scored as delivered, those five PRs would have contributed almost nothing. - **The timeout.** sentry #67876 ran into our 90-minute limit at 91 minutes and left no report. Its spend is about $23 by the run’s ledger and $24.42 by the last budget line in its transcript; Claude Code recorded no cost for it.**Scoring rule:** zero findings on the answer key, and in the Elo ratings the PR drops out of that configuration’s pairs (the judge cannot compare a review that does not exist), so`max` is rated on 19 PRs and the Elo carries no penalty for the missing review. - **Rate limits.** Our API key’s limits are 50 requests a minute for Opus 5.5, 1,000 for Sonnet 5.5 and Haiku 5.5, and 500,000 input and 80,000 output tokens a minute per model. Anthropic’s[published rate limits](https://platform.claude.com/docs/en/api/rate-limits) give the standard Start tier 1,000 requests, 2,000,000 input tokens and 400,000 output tokens a minute for these models, and say new organizations may start in an Evaluation tier below the standard limits. On the one-PR trial run, one`max` review with Sonnet 5.5 agents averaged about 20,000 output tokens a minute and peaked near 65,000. We started at 3 reviews at a time, went to 5, then 4, and back to 3 after reviews began failing. 25 of the 339 agents ended on 429 errors that their retries did not recover: 23 of them in 11 finished reviews that still reported success, and 2 in the review that timed out. One review (grafana #90045, $12.27) died on 429s before writing its report and was re-run; in the scored reviews every agent that died on a 429 was a finder or verifier, never the review agent itself. After the drop back to 3 at a time no more reviews died on 429s. These failures come from our key’s limits and our concurrency, not from a defect in Claude Code, but they show that a successful exit status does not mean every agent ran. - **Our own operator errors.** Two reviews (cal.com #10600, about $5 spent, and sentry-greptile #3, about $2.40) were killed when a queue runner was paused and the tool that launched it killed its process group; both were re-run from scratch. Rebuilding the harness image briefly made the queue lose track of running reviews, so 8 ran at once for about a minute; the 4 surplus reviews (33 seconds old) were killed and re-queued. The partial attempts were kept on disk for the record and never scored. - **Quickchat Review’s failures.** On Sonnet 5.5, the main session ran none of its planned passes and no validator on 2 of the 20 PRs (“I skipped the lens passes and the validator and reviewed the diff myself”; one of those went out as an approval) and launched fewer passes than planned on 3 more; it also dropped some candidates “to save effort” before the validator.**Scoring rule:** scored as delivered, the same as Claude Code’s timeout. On Opus 5.5 every planned pass and the validator launched on every PR, and 6 throttled calls were retried. All 20 reviews finished in both arms. This is why the body calls the validator “meant to” keep old bugs out: on these builds the planned passes and the validator are asked for by the prompt and not enforced by the harness, so what was observed (every stage on Opus 5.5, stages skipped on 2 of 20 PRs on Sonnet 5.5) sits beside what is intended. The one finding per root cause, the severity and the verdict were observed on every review. - **Completion.** Claude Code`high` finished 40 of 40 reviews on the 20 PRs (and 70 of 70 Martian reviews in all);`max` finished 19 of 20, and 14 of those 19 returned their report to the caller. ### 10. The pairwise judge - **Why.** The answer key separated`max` (F2 30.0) from the leaders but not the leaders from each other. On the 20 PRs, three very different setups scored within 0.8 points of each other on F2: Quickchat Review on Sonnet 5.5 at 43.9, Claude Code`high` on Opus 5.5 at 43.2 and Claude Code`high` on Sonnet 5.5 at 43.1, a spread these data cannot separate (note 7). The key holds about 3.7 entries per PR (74 on these 20), while the reviews we ran made about 8 to 26 claims per PR, and the key cannot tell a true claim it lacks from a wrong one; both count against precision. A one-line review that matches one key entry and a thorough review that finds three real bugs the key lacks can score alike. The pairwise judge was added after those scores were in, to read whole reviews against the code: it reads two reviews of the same PR side by side and picks a winner, and a rating fitted over many such judgments separates configurations the answer key compresses together. - **One judgment is one model call.** No tools, no agent loop, structured JSON output, and each answer accepted only as one plain final message (checked on every call: a cut-off, tool call or refusal counts as an error and is retried, at most 3 attempts). - **What the prompt holds, in order.** The PR’s title and description; its diff; every file it touches as it is at the head commit, line-numbered; Martian’s answer-key comments; then Review A and Review B (findings and summary). Lockfiles, checksum files, minified bundles and source maps are left out and replaced by one line with their changed-line counts. The prompt ceiling is 400,000 tokens; past it, the largest files fall back to their diff with 60 lines of context (on these 20 PRs no prompt reached the ceiling). The judge sees the touched files only, not the callers, dependencies or tests a reviewer could open, so it cannot check every claim a review makes; it has no “insufficient context” verdict. - **Identity labels removed.** Both reviews are rewritten into one neutral shape: path, lines, text and, where the review stated one, a severity. Quickchat Review’s pass labels, footers and signature are stripped and its tags become a stated severity; Claude Code’s JSON dump of its own findings is removed from its summary. A and B order is random, drawn once per pair, and no pair was judged in both orders. The judge is told not to guess which tool wrote which review, but a stated severity, the number of findings and the writing style remain, so tool identity may stay inferable. - **The answer key is a reference, not truth.** The judge may decide that a finding outside the key is a real defect, or that a key comment is wrong, but only by pointing at code in the prompt (file, line, what it does); without that evidence it defers to the key. An unmatched finding is a false positive only when the code shows it is wrong, the failure cannot happen, or it is noise a maintainer would not want. No sample of these decisions was checked by a human. - **The verdict.** Four categories, each with reasoning first and then a winner (A, B or tie): - **recall:** which review catches more of the real defects that matter, key comments plus real defects the key lacks, weighted by severity; - **precision:** which spends less of the author’s attention on findings that are wrong, unreachable or not worth raising; - **finding quality:** for the real findings, correct severity, precise location, a concrete failure scenario, the root cause and an actionable fix; - **writing:** which a developer can read and act on faster. Then the overall question: “which review a maintainer of this repository would rather have received on this pull request. Catching the serious defects and not wasting the author’s time matter most; finding quality next; writing can decide a close call.” The judge is told that more findings, longer text and confident tone earn nothing on their own, and that several comments about one root cause count as one finding. - **The two judges.** GPT-6.1 Sol and Claude Opus 5.5, both at xhigh reasoning effort, with an identical prompt. Every judgment records the sha256 of the exact prompt it saw. When a PR’s prompt changed (28 GPT judgments were made before we stopped packing lockfiles whole), those judgments were set aside and the pair was judged again by both judges, so both judges’ ratings rest on the same prompts. ### 11. How the Elo ratings were calculated - **What is rated.** A configuration’s review of a PR enters the ratings only if it exists, and the five`max` reports recovered from transcripts enter as recovered. The ratings are therefore conditional on an available report, with transcript recovery; the timed-out`max` review is absent, so`max` is rated on 19 PRs and the other configurations on 20. The body’s “over the reviews that exist” and every Elo it quotes mean exactly this. - **The fit.** Each judgment is one game between two configurations on one PR: 1 for a win, 0.5 for a tie, 0 for a loss. Ratings are a regularized maximum-likelihood fit of the[Bradley-Terry model](https://doi.org/10.2307/2334029) (Bradley and Terry, 1952), computed with[Hunter’s MM iteration](https://doi.org/10.1214/aos/1079120141) (Hunter, 2004), with half a virtual tie between every pair as the regularizer so that a configuration that wins every game still gets a finite rating. Ratings are shown on the Elo scale, 1500 + 400 × log10(strength relative to the geometric mean).[Chatbot Arena](https://arxiv.org/abs/2403.04132) likewise ranks models by Bradley-Terry coefficients fitted to pairwise preferences. Under the fitted model a 100-point gap is an expected score of about 64% for the higher one and 148 points about 70%; those are the model’s expected scores within this evaluation, not measured probabilities that a maintainer prefers a review. The body’s “about a 70% expected win rate” for the +148 gap is this figure. The fitting code ships with the raw results (note 16). - **The intervals.** 95% intervals from 1,000 bootstrap samples over PRs, not over games: judgments on one PR share its difficulty, so the PR is the unit resampled. The ratings’ intervals and every gap’s interval come from the same bootstrap draw. They are pointwise and not adjusted for the number of comparisons made. Because later rounds were allocated after inspecting the earlier ratings (below), they are exploratory and conditional on the matchups that were sampled. A gap is called clear when its interval excludes zero. - **Sampling.** Round 0 judged every pair of the 8 configurations (28 pairs) on the same 5 PRs, 140 judgments per judge. An adaptive round then gave 3 fresh PRs to each of 6 pairs of neighbouring configurations whose gap interval still crossed zero (18 judgments). A final targeted round judged every pair among the top five combined configurations (`max` , Quickchat Review on Opus 5.5, on GPT-6.1 Sol and on Sonnet 5.5, and Claude Code`high` on Opus 5.5) on every PR not yet judged (140 judgments), so those five were compared pairwise on every available PR. That made 298 judgments per judge over all 20 PRs, under a cap of 300. The three lowest configurations (Quickchat Review on DeepSeek V4.1 Flash and on GLM-5.3 Flash, and Claude Code`high` on Sonnet 5.5) were not in the final round, so their ratings rest mostly on round 0 and have wider intervals; that includes Claude Code`high` on Sonnet 5.5, which appears in the body’s main table. - **Combining the judges.** Each judge is fitted on its own, and the combined table counts every PR-pair both judges answered once per judge, at equal weight (596 games). Equal weight is a chosen aggregation rule, not a correction for either judge’s bias. Win-tie-loss counts quoted for the combined table (such as`max` ’s 162-22-10 on recall, over the 194 recall verdicts on its pairs) are over those 596 games. - **The full table.** Conditional on an available report, with transcript recovery;`max` on 19 PRs.Configuration Combined Elo (95% interval) GPT judge Opus judge Answer-key F2 Recorded cost (20 PRs) Claude Code `max` , Sonnet 5.5 agents1666 (1581 to 1794) 1544 1883 30.0 $398.03 (19 finished reviews) Quickchat Review, Opus 5.5 1628 (1567 to 1726) 1566 1729 35.0 $92.26 Quickchat Review, GPT-6.1 Sol 1566 (1486 to 1662) 1738 1372 30.8 $55.15 Claude Code `high` , Opus 5.51518 (1425 to 1629) 1514 1512 43.2 $6.53 Quickchat Review, Sonnet 5.5 1489 (1415 to 1584) 1598 1340 43.9 $23.35 Quickchat Review, DeepSeek V4.1 Flash 1406 (1201 to 1556) 1407 1386 34.5 $20.11 Claude Code `high` , Sonnet 5.51398 (1290 to 1456) 1374 1404 43.1 $2.12 Quickchat Review, GLM-5.3 Flash 1329 (1110 to 1477) 1259 1374 36.6 $11.98 - **Which gaps the intervals exclude zero for (combined).** Gap Combined Elo (95% interval) Expected win rate Head-to-head (won-lost) GPT judge alone Opus judge alone Combined interval excludes zero `max` over Quickchat Review, Opus 5.5+38 (−33 to +135) 55% 22-16 −22 (−142 to +111) +155 (+7 to +353) no: the data do not separate them Quickchat Review Opus 5.5 over Quickchat Review GPT-6.1 Sol +62 (−3 to +129) 59% 23-17 −172 (−325 to −47) +356 (+259 to +513) no: the data do not separate them `max` over Quickchat Review GPT-6.1 Sol+100 (+5 to +207) 64% 24-14 −194 (−385 to −33) +511 (+353 to +761) yes Quickchat Review Opus 5.5 over `high` Opus 5.5+110 (+31 to +196) 65% 29-11 +52 (−68 to +182) +217 (+99 to +377) yes `high` Opus 5.5 over`high` Sonnet 5.5+120 (+7 to +304) 67% 10-6 +140 (−21 to +413) +108 (−99 to +395) yes, narrowly Quickchat Review Opus 5.5 over Quickchat Review Sonnet 5.5 +139 (+77 to +222) 69% 27-13 −32 (−134 to +73) +389 (+291 to +548) yes `max` over`high` Opus 5.5+148 (+64 to +241) 70% 27-11 +30 (−68 to +135) +371 (+188 to +617) yes `max` over Quickchat Review Sonnet 5.5+177 (+76 to +297) 73% 27-11 −54 (−226 to +128) +543 (+374 to +798) yes Expected win rates are the fitted model’s, not observed shares; head-to-head counts are the direct games between the two over both judges, with no ties. Each judge’s own gaps come from that judge’s fit alone. No gap in the table is clear in the same direction under both judges on their own. Every clear combined gap but the +120 is one the Opus judge separates and the GPT judge does not separate in the same direction; the GPT judge separates two gaps the other way, preferring Quickchat Review on GPT-6.1 Sol to both `max` and Quickchat Review on Opus 5.5. The +120 between the two`high` runs is clear only once both judges’ games are pooled, and it rests on 16 games, eight PRs per judge, because`high` on Sonnet 5.5 was not in the final round (Sampling, above); neither judge separates it alone. - **Per category (combined).** These are relative preference ratings in each judged category, each fitted on its own scale; they are not measured precision or recall percentages. Recall:`max` 1827 (162 wins, 22 ties, 10 losses), Quickchat Review on Opus 5.5 1575,`high` on Opus 5.5 1517, Quickchat Review on Sonnet 5.5 last at 1259. Precision: Quickchat Review on Sonnet 5.5 1937 (169-13-6), then Quickchat Review on GPT-6.1 Sol 1596 and on Opus 5.5 1552,`high` on Opus 5.5 1497, and`max` second to last at 1325. Finding quality: Quickchat Review on Opus 5.5 1702,`max` 1638, Quickchat Review on GPT-6.1 Sol 1539,`high` on Sonnet 5.5 last at 1303. Writing: Quickchat Review on Sonnet 5.5 1937 (167-7-14),`high` on Opus 5.5 1698, Quickchat Review on GPT-6.1 Sol 1630,`max` last at 1101 (14-1-179). - **Cost and rating differences.**`max` and Quickchat Review on Opus 5.5 are not separated by the data, at 4.3 times its recorded cost; Quickchat Review on Opus 5.5 rates +110 over`high` on Opus 5.5 at 14 times its cost;`max` rates +148 over`high` on Opus 5.5 at 61 times its cost. ### 12. Judge agreement and self-preference - **Agreement over the 298 matchups both judges answered.** Same overall winner 56%, opposite 44% (no overall ties). Recall 70% same and 7% opposite; precision 76% and 7%; finding quality 50% and 26%; writing 74% and 20%. The rest are matchups where one judge called a tie. These are raw agreement rates, affected by how often each verdict occurs; agreement between two LLM judges says they read the reviews alike, not that either read them correctly, and no human calibration of the verdicts was made. - **Each judge’s top configuration ran on its own family.** The GPT judge’s table is led by Quickchat Review on GPT-6.1 Sol (1738, 73 wins and 21 losses); the Opus judge’s by Claude Code`max` (1883, 88 wins and 9 losses), a Claude-written review that also carries the most findings. The two judges agree almost exactly on Claude Code`high` on Opus 5.5 (1514 and 1512). The pattern fits a family preference; it also fits two judges valuing coverage and brevity differently, and these data cannot separate the two explanations. The body’s “GPT favoured lean reviews and Opus thorough ones” is the observed pattern, not a settled cause. - **How they weigh the overall question.** In 12 matchups we read before the full run, GPT resolved the overall verdict toward precision (“A’s precision problems are decisive”) and Opus toward recall (“A maintainer would get more of the expected defects from A, with only modest noise”). In a handful of spot checks, a third judge (DeepSeek V4.1 Flash, an open model) sided with Opus more often than with GPT. - **What follows.** The per-category results are where the judges mostly agree on which review has more recall, more precision and better writing, so they depend less on which judge one trusts. The overall winner is a weighting of recall against noise, and that is a value judgment. Combining two judges from different providers at equal weight is a chosen rule that limits how far either judge’s preference can carry the table; it does not show that the preference is gone. ### 13. Judging cost - The GPT-6.1 Sol judge ran on a ChatGPT subscription login, which bills quota, not dollars; at API list prices its 298 judgments (plus the 28 set aside) would have cost about $92.96. - The Opus 5.5 judge cost $45.66: the 140 first-round calls went through the Batch API at Batch prices ($2 input, $10 output, $4 one-hour cache write and $0.10 cache read per million tokens), and the 158 calls of the two later rounds ran live at regular API prices. Each call is priced from the service tier it reports. - The answer-key judge’s cost (GPT-6.1 Sol, also on subscription quota) is not counted anywhere in this post. ### 14. Conflicts of interest - Quickchat AI builds Quickchat Review, a pull-request reviewer that competes with Claude Code’s, and I built it. I designed these tests, ran them and wrote this post. - Quickchat Review’s default configuration was tuned in earlier work against public benchmarks, including 20 of Martian’s 50 PRs, so it may have seen some of the PRs in this set. I did not reconstruct how many of those 20 are among this post’s 20, so read the Quickchat Review rows as an evaluation on a benchmark its builder had partly used for tuning. Claude Code was not tuned by us; it ran as its bundled skill under the settings in note 2 (feature-flag fetching off, forced subagent models on `max` , budget caps, timeouts and default headless permissions), not as an untouched default. - The answer-key judge is an OpenAI model scoring an Anthropic tool. Quickchat Review’s default model is also OpenAI’s, and Quickchat AI runs on OpenAI and Google Cloud. - The selection of the 20 PRs is expected to work against Quickchat Review by an amount I cannot measure (note 5), and the closing-note recovery works for `max` (note 9). The pairwise judge was added after the answer-key scores were in (note 10). ### 15. Limitations - 20 PRs from one benchmark, chosen non-randomly, plus Claude Code `high` on Opus 5.5 and four configurations of Quickchat Review on all 50. One run per configuration, so run-to-run variance is unknown, and the “about two issues” of noise is a prior from earlier repeated scoring, not a measurement made here. - Two LLM judges, both from families with a configuration in the pool, and each putting a configuration from its own family first. The answer-key judge is a third LLM. No judgment was checked by a human. - The overall Elo question left the weight of recall against noise to the judges, and they weighed it differently. The Elo ratings are conditional on an available report, with transcript recovery, so the timed-out `max` review costs it nothing there; the later judging rounds were allocated after inspecting earlier ratings, so the intervals are exploratory. - The `max` arm forced Sonnet 5.5 subagents and ran under a $30 cap whose budget line the agents could see and acted on (note 3); the all-Opus configuration was measured on one PR only, where our 30-minute limit stopped it. An uncapped all-Opus`max` on the 20 PRs was not run. - Costs are as each tool records or computes them: Claude Code’s are its own list-price estimates, except the $40.64, which is our recount; Quickchat Review’s are computed at list prices from its token usage; the Opus 5.5 main session added 0.44% of the `max` arm’s cost (note 8). - Our API key’s rate limits are below Anthropic’s standard tier, which caused the 429 failures and limited concurrency; on a higher tier `max` reviews would likely fail less and finish sooner. Every time in this post is wall-clock under those limits and our concurrency, and no run was made at matched concurrency. - Claude Code ran with default `-p` permissions in a sandbox where its web tools and some shell commands were refused, under our sandbox configuration (feature-flag fetching off, forced subagent models on`max` , budget caps of $5 and $30, timeouts of 30 and 90 minutes); interactive use, where a person approves tool calls, may behave differently. - The same-model rows compare complete configurations that share a primary model (note 4): Quickchat Review read the PR title, description and dependency sources, ran six passes on GPT-6 Luna and used its own permissions; Claude Code did none of that. - The recommendation to start with `high` on Opus 5.5 rests on 51 of 66 serious issues matched on all 50 PRs, which also means 15 missed; whether that is enough for a given repository or risk is not something this benchmark can say. - The browser-use answer key (note 6) is our own and was built and checked by AI agents with no human check, and the scoring of that PR had no human check either. - A one-shot benchmark sees only a first review. Neither tool’s follow-up rounds, replies to the author or repository-specific setup were exercised. - The recipe observations (note 3) hold for Claude Code 2.1.303 and may not hold for any other release. ### 16. Raw results - All raw results behind this post (every review each configuration wrote, both judges’ verdicts and explanations, the judge and scoring prompts, the Elo fitting code and the scoring files) will be published when Quickchat Review is open-sourced in the coming weeks. Until then, the tables in notes 7 and 11 are the complete scores.