Don't Trust Anyone's Muse Wonder Benchmark. Race It Against Opus 5.5 on Your Repo

A developer has published a reproducible bake-off method for comparing Meta's Muse Wonder against Claude Opus 5.5 on a user's own repository, after noting that Meta's model card claims Wonder leads on long-horizon coding (78.8 vs 74.2 on DeepSWE, 53.1 vs 46.3 on SWE-Marathon) while Artificial Analysis's Intelligence Index puts Opus 5.5 ahead 58 to 52. The approach runs the same ticket through two agents in separate git worktrees and scores both with a named test command, using the muse-wonder-1.0 API model ID and Meta's documented ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN and ANTHROPIC_MODEL environment variables. The author notes Meta's own footnotes concede competitor runs were "best-effort" with prompts, tools and runtime possibly not optimally tuned.

Meta's chart says Muse Wonder beats Claude Opus 5.5 at long coding jobs: 78.8 to 74.2 on DeepSWE, 53.1 to 46.3 on SWE-Marathon model card https://research.meta.ai/static/muse-wonder-model-card . Artificial Analysis https://artificialanalysis.ai/models/muse-wonder gives Wonder 52 on its Intelligence Index. Opus 5.5 scores 58 https://artificialanalysis.ai/articles/claude-opus-5-5 . Both cost $4 per million input tokens and $20 per million output Wonder's price per TechCrunch https://techcrunch.com/2026/10/12/meta-releases-its-first-frontier-model-muse-wonder/ , Opus's per the same Artificial Analysis article . Meta's chart and the independent index can both hold, because they measure different things, and neither ran on your code. Meta launched Muse Wonder on October 12, 2026 Meta newsroom https://about.fb.com/news/2026/10/introducing-muse-wonder-models/ . In the Hacker News thread https://news.ycombinator.com/item?id=50059115 , one commenter called it "much worse than Opus 5.5" after an image-to-HTML test. That is one prompt. Nothing below is a Muse Wonder result. What follows is the setup to get your own: the same ticket handed to two agents in two git worktrees, and a script that scores both. Wonder wins five of Meta's 14 rows against Opus 5.5, mostly long-horizon coding, legal agent work and Meta's internal memory test. It loses nine, including ProgramBench, where Opus leads 70.5 to 50.0. The row-by-row view is in the scorecard breakdown https://dev.to/gabrielanhaia/metas-muse-wonder-costs-exactly-what-opus-55-costs-it-loses-9-of-metas-own-14-benchmarks-399a . Meta's own footnotes tell you how far to trust any single row. For competitor runs it did in-house, Meta says they "use common settings where practical but remain best-effort: prompts, tools, and runtime may not be optimally tuned for proprietary models." And from the limitations section of the model card https://research.meta.ai/static/muse-wonder-model-card : "Performance can vary with quantization, serving configuration, prompts, tools, and the agent harness." No leaderboard can hold prompts, tools and harness fixed for your setup, but a bake-off on your own repo does exactly that. One ticket, two agents building the same thing, one scale to weigh them. Choose a real ticket with a test that fails today or write one first , a fix that touches a few files, and an outcome you can judge. Save the prompt to a file outside the repo so every agent gets the same words, and name the test command in it. "Make npm test -- slug pass without changing the tests" gives the agents and you the same finish line. Next, create a key for the Meta Model API and export it under the name Meta's coding-agents docs https://dev.meta.ai/docs/coding-agents use. Put it in your shell profile or load it from your secrets manager so any new terminal has it: export MODEL API KEY="your-key-here" The API model ID is muse-wonder-1.0 . The API does not serve Muse Wonder Full, the stronger sibling: it is for Muse Code subscribers only TechCrunch https://techcrunch.com/2026/10/12/meta-releases-its-first-frontier-model-muse-wonder/ . I build Hermes IDE https://hermes-ide.com , a free desktop console for coding agents, so the setup below uses it download it https://hermes-ide.com/download to follow along . The plain-git version follows session B. Press ⌘N Ctrl+Shift+N on Windows and Linux , pick Claude Code in the agent chip, pick Opus 5.5 in the model chip, paste the prompt and press Enter. Claude Code uses its normal sign-in. The task starts in its own git worktree on a new hermes/<task branch, cut from the branch you are on. Meta documents three environment variables that point Claude Code at its API: ANTHROPIC BASE URL , ANTHROPIC AUTH TOKEN and ANTHROPIC MODEL . The docs also tell you to set the Opus, Sonnet and Haiku aliases and the subagent model to the same ID, "so no path falls back to a Claude model". Skip those and some requests ask for a Claude model ID instead of Muse Wonder, and session B stops being a clean Muse run. Seven variables make a long command, so put them in a wrapper. Save this as claude-muse somewhere on your PATH and chmod +x it: bash /usr/bin/env bash claude-muse: Claude Code against Meta's Model API export ANTHROPIC BASE URL="https://api.meta.ai" export ANTHROPIC AUTH TOKEN="$MODEL API KEY" m="muse-wonder-1.0" export ANTHROPIC MODEL="$m" export ANTHROPIC DEFAULT OPUS MODEL="$m" export ANTHROPIC DEFAULT SONNET MODEL="$m" export ANTHROPIC DEFAULT HAIKU MODEL="$m" export CLAUDE CODE SUBAGENT MODEL="$m" exec claude "$@" The base URL has no /v1 , and the key goes in AUTH TOKEN , not API KEY . Both details come from Meta's docs. Their worked example uses muse-spark-1.3 ; muse-wonder-1.0 is the Wonder ID. Now press ⌘N again and pick Custom agent "Any command you type" . Name it Claude Code · Muse Wonder and set the command to claude-muse . The hint under the field says it plainly: "Hermes starts this command in the session's terminal, as if you typed it." Because it's a real shell, an env-prefixed one-liner works too. When the CLI is up, paste the same prompt. Sessions A and B now run the same harness on the same ticket. What differs is the model and the API serving it. Meta's footnote on its SWE-Marathon row calls that row "not a controlled common-harness comparison". Your A-versus-B run is a common-harness one. Real Hermes window, test build with stand-in agents. The same task in two sessions side by side, each on its own worktree branch. In your bake-off, the Custom agent takes the second slot. The isolation part is one command per agent. From your repo: git worktree add -b bakeoff/opus ../bakeoff-opus git worktree add -b bakeoff/muse ../bakeoff-muse Then run claude in ../bakeoff-opus pick Opus 5.5 with /model and claude-muse in ../bakeoff-muse , each in its own terminal. You lose a single place to watch both, and the review step later. Hermes worktrees are plain git branches too, so nothing here locks you in. muse CLI Session B tests the model inside Anthropic's harness. If you want Meta's harness too, install Muse Code. Meta's installer creates a command named muse in ~/.local/bin . Read the script first if you prefer, then run: curl -fsSL https://dev.meta.ai/install.sh | bash Add one more Custom agent with the command muse . Muse Code is a subscription: the Power Usage plan https://dev.meta.ai/products/muse-code is $50 a month and includes Wonder Full. Meta's product page lists macOS and Windows. One early report is negative: in the same Hacker News thread, a commenter said the Muse Code sandbox "never worked across Mac or Linux". Check it on your own machine before you pay for a plan. When both agents say they're done, score them the same way. This script takes the base branch, a test command, and any number of worktree paths. For each one it prints the branch, whether the tests pass, how many files changed and how many lines went in and out against the point where the branch left the base. Save it as compare-worktrees.sh next to your repo and chmod +x it: bash /usr/bin/env bash Score agent worktrees against the branch they started from. usage: compare-worktrees.sh BASE "TEST COMMAND" WORKTREE... set -u base=$1 test cmd=$2 shift 2 for wt in "$@"; do branch=$ git -C "$wt" rev-parse --abbrev-ref HEAD fork=$ git -C "$wt" merge-base "$base" HEAD count untracked files before the tests can add any new=$ git -C "$wt" ls-files -o --exclude-standard | grep -c . cd "$wt" && bash -c "$test cmd" /dev/null 2 &1 code=$? if "$code" -eq 0 ; then result=PASS; else result=FAIL; fi working tree vs fork point: committed and uncommitted edits stats=$ git -C "$wt" diff --numstat "$fork" files=$ printf '%s' "$stats" | grep -c . added=$ printf '%s\n' "$stats" | awk '{s+=$1} END {print s+0}' removed=$ printf '%s\n' "$stats" | awk '{s+=$2} END {print s+0}' echo "== $wt" echo " branch: $branch" echo " tests: $result exit $code " echo " files: $files changed, $new untracked" echo " lines: +$added -$removed vs $base" done It diffs the working tree against the fork point, so it counts edits an agent made but never committed. Brand-new files git doesn't track yet show up in the "untracked" count instead of the line totals. That count runs before the tests, so files the test run writes don't inflate it. Anything the agent's own test runs left behind still counts unless your .gitignore covers it. The output below is not from Muse Wonder or Opus 5.5. It comes from a throwaway toy repo git 2.52, Python 3.14, pycache / in .gitignore with two hand-written fixes for a broken slugify standing in for agent output. Fix A rewrote the function with a regex. Fix B stripped punctuation, still produced double hyphens on extra spaces, and left a TODO in the test file: bash $ git worktree add -b bakeoff/agent-a ../agent-a Preparing worktree new branch 'bakeoff/agent-a' HEAD is now at 759ecee Initial commit $ git worktree add -b bakeoff/agent-b ../agent-b Preparing worktree new branch 'bakeoff/agent-b' HEAD is now at 759ecee Initial commit $ ../compare-worktrees.sh main "python3 -m unittest -q" \ ../agent-a ../agent-b == ../agent-a branch: bakeoff/agent-a tests: PASS exit 0 files: 1 changed, 0 untracked lines: +5 -1 vs main == ../agent-b branch: bakeoff/agent-b tests: FAIL exit 1 files: 2 changed, 0 untracked lines: +6 -1 vs main In Hermes the worktrees are plain git too, so git worktree list in your repo prints the paths to pass in. The script covers the first three rows. The rest you note while you watch: | | A: Claude Code, Opus 5.5 | B: Claude Code, Muse Wonder | C: muse optional | |---|---|---|---| | Tests pass exit code | | | | | Lines +/- vs base | | | | | Files touched | | | | | Wall time to "done" | | | | | Cost | | | | | Review comments before you'd merge | | | | Wall time. Artificial Analysis https://artificialanalysis.ai/models/muse-wonder measured Wonder's time to first token at 19.27 seconds at launch, against a 5.52-second median for comparable models. An agent makes many calls per task, so expect that to show. Cost. Meta's docs warn that agents pointed at a custom provider often show $0 cost, because they lack pricing metadata. Don't read cost from session B's terminal. Take the run's input, cached and output token counts and do the math: input millions × $4, cached input millions × $0.20 the cached rate Handy AI https://handyai.substack.com/p/model-drop-muse-wonder reports , output millions × $20. The per-token prices match Opus 5.5, but per-task cost can still differ. Artificial Analysis counted 130M output tokens for Wonder to run its index, against an 85M median. For session A, Hermes shows an estimated cost per session. Press ⌘G Ctrl+Shift+G on Windows and Linux to open the Review Desk and switch to By turn . You read each agent's work turn by turn instead of as one final diff. Risk flags mark lockfiles, CI workflows, secrets, new binaries and edits an agent makes to its own config. From hermes-ide.com: a recreation of the 2.0 app with demo data. The Review Desk by turn, with a line comment for Claude Code and a drafted commit message. A line comment goes back to the agent that wrote the line. Count those comments: that is the last row of your scorecard. Hermes knows less about a Custom agent than a built-in one, so if session B's turns look thin, read it By file , or fall back to the script's numbers and a plain git diff against your base branch. Land the winner from the Land sheet: commit, open a pull request or merge locally, with undo afterwards. Then close the other session. By hand, remove the losing worktree with git worktree remove ../bakeoff-muse or ../bakeoff-opus . One ticket is an anecdote. Run three to five tickets of different shapes before you call it, because a model that wins a refactor can lose a bug hunt. Hermes has no status signals for Custom agents. It doesn't know the command's approval model, so the attention inbox can't tell when session B or C is waiting on you. Watch their terminals. Session A keeps full status. You're also testing Muse Wonder, not Wonder Full. The API doesn't serve Full, so only session C, on a plan that includes it, reaches it. And session B is Anthropic's harness on Meta's model: fair for comparing models, maybe not how Meta tuned Wonder to work, which is why session C exists. Hermes IDE is not affiliated with or endorsed by Meta, Anthropic, OpenAI or Nous Research. Pick the ticket you'd hand an agent this week, run it through both sessions, and land the one you'd rather review again. I build Hermes IDE. It is free, source-available, and runs on macOS, Windows and Linux. It ships often: 2.2.10 came out on October 12, If you're designing the agents themselves, the AI Agents Pocket Guide covers the patterns behind them.