{"slug": "don-t-trust-anyone-s-muse-wonder-benchmark-race-it-against-opus-5-5-on-your-repo", "title": "Don't Trust Anyone's Muse Wonder Benchmark. Race It Against Opus 5.5 on Your Repo", "summary": "A developer has published a reproducible bake-off method for comparing Meta's Muse Wonder against Claude Opus 5.5 on a user's own repository, after noting that Meta's model card claims Wonder leads on long-horizon coding (78.8 vs 74.2 on DeepSWE, 53.1 vs 46.3 on SWE-Marathon) while Artificial Analysis's Intelligence Index puts Opus 5.5 ahead 58 to 52. The approach runs the same ticket through two agents in separate git worktrees and scores both with a named test command, using the muse-wonder-1.0 API model ID and Meta's documented ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN and ANTHROPIC_MODEL environment variables. The author notes Meta's own footnotes concede competitor runs were \"best-effort\" with prompts, tools and runtime possibly not optimally tuned.", "body_md": "Meta's chart says Muse Wonder beats Claude Opus 5.5 at long coding\n\njobs: 78.8 to 74.2 on DeepSWE, 53.1 to 46.3 on SWE-Marathon\n\n([model card](https://research.meta.ai/static/muse-wonder-model-card)).\n\n[Artificial Analysis](https://artificialanalysis.ai/models/muse-wonder)\n\ngives Wonder 52 on its Intelligence Index. Opus 5.5\n\n[scores 58](https://artificialanalysis.ai/articles/claude-opus-5-5).\n\nBoth cost $4 per million input tokens and $20 per million output\n\n(Wonder's price per\n\n[TechCrunch](https://techcrunch.com/2026/10/12/meta-releases-its-first-frontier-model-muse-wonder/),\n\nOpus's per the same Artificial Analysis article).\n\nMeta's chart and the independent index can both hold, because they\n\nmeasure different things, and neither ran on your code.\n\nMeta launched Muse Wonder on October 12, 2026\n\n([Meta newsroom](https://about.fb.com/news/2026/10/introducing-muse-wonder-models/)).\n\nIn the [Hacker News thread](https://news.ycombinator.com/item?id=50059115),\n\none commenter called it \"much worse than Opus 5.5\" after an\n\nimage-to-HTML test. That is one prompt. Nothing below is a Muse\n\nWonder result. What follows is the setup to get your own: the same\n\nticket handed to two agents in two git worktrees, and a script that\n\nscores both.\n\nWonder wins five of Meta's 14 rows against Opus 5.5, mostly\n\nlong-horizon coding, legal agent work and Meta's internal memory\n\ntest. It loses nine, including ProgramBench, where Opus leads 70.5\n\nto 50.0. The row-by-row view is in\n\n[the scorecard breakdown](https://dev.to/gabrielanhaia/metas-muse-wonder-costs-exactly-what-opus-55-costs-it-loses-9-of-metas-own-14-benchmarks-399a).\n\nMeta's own footnotes tell you how far to trust any single row. For\n\ncompetitor runs it did in-house, Meta says they \"use common settings\n\nwhere practical but remain best-effort: prompts, tools, and runtime\n\nmay not be optimally tuned for proprietary models.\" And from the\n\nlimitations section of the\n\n[model card](https://research.meta.ai/static/muse-wonder-model-card):\n\n\"Performance can vary with quantization, serving configuration,\n\nprompts, tools, and the agent harness.\"\n\nNo leaderboard can hold prompts, tools and harness fixed for your\n\nsetup, but a bake-off on your own repo does exactly that.\n\n*One ticket, two agents building the same thing, one scale to weigh them.*\n\nChoose a real ticket with a test that fails today (or write one\n\nfirst), a fix that touches a few files, and an outcome you can judge.\n\nSave the prompt to a file outside the repo so every agent gets the\n\nsame words, and name the test command in it. \"Make\n\n`npm test -- slug` pass without changing the tests\" gives the agents\n\nand you the same finish line.\n\nNext, create a key for the Meta Model API and export it under the\n\nname Meta's [coding-agents docs](https://dev.meta.ai/docs/coding-agents)\n\nuse. Put it in your shell profile (or load it from your secrets\n\nmanager) so any new terminal has it:\n\n```\nexport MODEL_API_KEY=\"your-key-here\"\n```\n\nThe API model ID is `muse-wonder-1.0`. The API does not serve Muse\n\nWonder Full, the stronger sibling: it is for Muse Code subscribers\n\nonly\n\n([TechCrunch](https://techcrunch.com/2026/10/12/meta-releases-its-first-frontier-model-muse-wonder/)).\n\nI build [Hermes IDE](https://hermes-ide.com), a free desktop console\n\nfor coding agents, so the setup below uses it\n\n([download it](https://hermes-ide.com/download) to follow along). The\n\nplain-git version follows session B.\n\nPress ⌘N (Ctrl+Shift+N on Windows and Linux), pick Claude Code in the\n\nagent chip, pick Opus 5.5 in the model chip, paste the prompt and\n\npress Enter. Claude Code uses its normal sign-in. The task starts in\n\nits own git worktree on a new `hermes/<task>` branch, cut from the\n\nbranch you are on.\n\nMeta documents three environment variables that point Claude Code at\n\nits API: `ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN` and\n\n`ANTHROPIC_MODEL`. The docs also tell you to set the Opus, Sonnet and\n\nHaiku aliases and the subagent model to the same ID, \"so no path\n\nfalls back to a Claude model\". Skip those and some requests ask for\n\na Claude model ID instead of Muse Wonder, and session B stops being\n\na clean Muse run.\n\nSeven variables make a long command, so put them in a wrapper. Save\n\nthis as `claude-muse` somewhere on your `PATH` and `chmod +x` it:\n\n``` bash\n#!/usr/bin/env bash\n# claude-muse: Claude Code against Meta's Model API\nexport ANTHROPIC_BASE_URL=\"https://api.meta.ai\"\nexport ANTHROPIC_AUTH_TOKEN=\"$MODEL_API_KEY\"\nm=\"muse-wonder-1.0\"\nexport ANTHROPIC_MODEL=\"$m\"\nexport ANTHROPIC_DEFAULT_OPUS_MODEL=\"$m\"\nexport ANTHROPIC_DEFAULT_SONNET_MODEL=\"$m\"\nexport ANTHROPIC_DEFAULT_HAIKU_MODEL=\"$m\"\nexport CLAUDE_CODE_SUBAGENT_MODEL=\"$m\"\nexec claude \"$@\"\n```\n\nThe base URL has no `/v1`, and the key goes in `AUTH_TOKEN`, not\n\n`API_KEY`. Both details come from Meta's docs. Their worked example\n\nuses `muse-spark-1.3`; `muse-wonder-1.0` is the Wonder ID.\n\nNow press ⌘N again and pick **Custom agent** (\"Any command you\n\ntype\"). Name it `Claude Code · Muse Wonder` and set the command to\n\n`claude-muse`. The hint under the field says it plainly: \"Hermes\n\nstarts this command in the session's terminal, as if you typed it.\"\n\nBecause it's a real shell, an env-prefixed one-liner works too. When\n\nthe CLI is up, paste the same prompt.\n\nSessions A and B now run the same harness on the same ticket. What\n\ndiffers is the model and the API serving it. Meta's footnote on its\n\nSWE-Marathon row calls that row \"not a controlled common-harness\n\ncomparison\". Your A-versus-B run is a common-harness one.\n\n*Real Hermes window, test build with stand-in agents. The same task in two sessions side by side, each on its own worktree branch. In your bake-off, the Custom agent takes the second slot.*\n\nThe isolation part is one command per agent. From your repo:\n\n```\ngit worktree add -b bakeoff/opus ../bakeoff-opus\ngit worktree add -b bakeoff/muse ../bakeoff-muse\n```\n\nThen run `claude` in `../bakeoff-opus` (pick Opus 5.5 with `/model`)\n\nand `claude-muse` in `../bakeoff-muse`, each in its own terminal.\n\nYou lose a single place to watch both, and the review step later.\n\nHermes worktrees are plain git branches too, so nothing here locks\n\nyou in.\n\n`muse` CLI\nSession B tests the model inside Anthropic's harness. If you want\n\nMeta's harness too, install Muse Code. Meta's installer creates a\n\ncommand named `muse` in `~/.local/bin`. Read the script first if you\n\nprefer, then run:\n\n```\ncurl -fsSL https://dev.meta.ai/install.sh | bash\n```\n\nAdd one more Custom agent with the command `muse`. Muse Code is a\n\nsubscription: the [Power Usage plan](https://dev.meta.ai/products/muse-code)\n\nis $50 a month and includes Wonder Full. Meta's product page lists\n\nmacOS and Windows. One early report is negative: in the same Hacker\n\nNews thread, a commenter said the Muse Code sandbox \"never worked\n\nacross Mac or Linux\". Check it on your own machine before you pay\n\nfor a plan.\n\nWhen both agents say they're done, score them the same way. This\n\nscript takes the base branch, a test command, and any number of\n\nworktree paths. For each one it prints the branch, whether the tests\n\npass, how many files changed and how many lines went in and out\n\nagainst the point where the branch left the base. Save it as\n\n`compare-worktrees.sh` next to your repo and `chmod +x` it:\n\n``` bash\n#!/usr/bin/env bash\n# Score agent worktrees against the branch they started from.\n# usage: compare-worktrees.sh BASE \"TEST COMMAND\" WORKTREE...\nset -u\n\nbase=$1\ntest_cmd=$2\nshift 2\n\nfor wt in \"$@\"; do\n  branch=$(git -C \"$wt\" rev-parse --abbrev-ref HEAD)\n  fork=$(git -C \"$wt\" merge-base \"$base\" HEAD)\n\n  # count untracked files before the tests can add any\n  new=$(git -C \"$wt\" ls-files -o --exclude-standard | grep -c .)\n\n  (cd \"$wt\" && bash -c \"$test_cmd\") >/dev/null 2>&1\n  code=$?\n  if [ \"$code\" -eq 0 ]; then result=PASS; else result=FAIL; fi\n\n  # working tree vs fork point: committed and uncommitted edits\n  stats=$(git -C \"$wt\" diff --numstat \"$fork\")\n  files=$(printf '%s' \"$stats\" | grep -c .)\n  added=$(printf '%s\\n' \"$stats\" | awk '{s+=$1} END {print s+0}')\n  removed=$(printf '%s\\n' \"$stats\" | awk '{s+=$2} END {print s+0}')\n\n  echo \"== $wt\"\n  echo \"   branch:  $branch\"\n  echo \"   tests:   $result (exit $code)\"\n  echo \"   files:   $files changed, $new untracked\"\n  echo \"   lines:   +$added -$removed vs $base\"\ndone\n```\n\nIt diffs the working tree against the fork point, so it counts edits\n\nan agent made but never committed. Brand-new files git doesn't track\n\nyet show up in the \"untracked\" count instead of the line totals. That\n\ncount runs before the tests, so files the test run writes don't\n\ninflate it. Anything the agent's own test runs left behind still\n\ncounts unless your `.gitignore` covers it.\n\nThe output below is not from Muse Wonder or Opus 5.5. It comes from a\n\nthrowaway toy repo (git 2.52, Python 3.14, `__pycache__/` in\n\n`.gitignore`) with two hand-written fixes for a broken `slugify()`\n\nstanding in for agent output. Fix A rewrote the function with a\n\nregex. Fix B stripped punctuation, still produced double hyphens on\n\nextra spaces, and left a TODO in the test file:\n\n``` bash\n$ git worktree add -b bakeoff/agent-a ../agent-a\nPreparing worktree (new branch 'bakeoff/agent-a')\nHEAD is now at 759ecee Initial commit\n$ git worktree add -b bakeoff/agent-b ../agent-b\nPreparing worktree (new branch 'bakeoff/agent-b')\nHEAD is now at 759ecee Initial commit\n$ ../compare-worktrees.sh main \"python3 -m unittest -q\" \\\n> ../agent-a ../agent-b\n== ../agent-a\n   branch:  bakeoff/agent-a\n   tests:   PASS (exit 0)\n   files:   1 changed, 0 untracked\n   lines:   +5 -1 vs main\n== ../agent-b\n   branch:  bakeoff/agent-b\n   tests:   FAIL (exit 1)\n   files:   2 changed, 0 untracked\n   lines:   +6 -1 vs main\n```\n\nIn Hermes the worktrees are plain git too, so `git worktree list` in\n\nyour repo prints the paths to pass in.\n\nThe script covers the first three rows. The rest you note while you\n\nwatch:\n\n|  | A: Claude Code, Opus 5.5 | B: Claude Code, Muse Wonder | C: `muse` (optional) | \n|---|---|---|---|\n| Tests pass (exit code) |  |  |  | \n| Lines +/- vs base |  |  |  | \n| Files touched |  |  |  | \n| Wall time to \"done\" |  |  |  | \n| Cost |  |  |  | \n| Review comments before you'd merge |  |  |  | \n\n**Wall time.**\n\n[Artificial Analysis](https://artificialanalysis.ai/models/muse-wonder)\n\nmeasured Wonder's time to first token at 19.27 seconds at launch,\n\nagainst a 5.52-second median for comparable models. An agent makes\n\nmany calls per task, so expect that to show.\n\n**Cost.** Meta's docs warn that agents pointed at a custom provider\n\noften show $0 cost, because they lack pricing metadata. Don't read\n\ncost from session B's terminal. Take the run's input, cached and\n\noutput token counts and do the math: input millions × $4, cached\n\ninput millions × $0.20 (the cached rate\n\n[Handy AI](https://handyai.substack.com/p/model-drop-muse-wonder)\n\nreports), output millions × $20. The per-token prices match Opus\n\n5.5, but per-task cost can still differ. Artificial Analysis counted\n\n130M output tokens for Wonder to run its index, against an 85M\n\nmedian. For session A, Hermes shows an estimated cost per session.\n\nPress ⌘G (Ctrl+Shift+G on Windows and Linux) to open the Review Desk\n\nand switch to **By turn**. You read each agent's work turn by turn\n\ninstead of as one final diff. Risk flags mark lockfiles, CI\n\nworkflows, secrets, new binaries and edits an agent makes to its own\n\nconfig.\n\n*From hermes-ide.com: a recreation of the 2.0 app with demo data. The Review Desk by turn, with a line comment for Claude Code and a drafted commit message.*\n\nA line comment goes back to the agent that wrote the line. Count\n\nthose comments: that is the last row of your scorecard. Hermes knows\n\nless about a Custom agent than a built-in one, so if session B's\n\nturns look thin, read it **By file**, or fall back to the script's\n\nnumbers and a plain `git diff` against your base branch.\n\nLand the winner from the Land sheet: commit, open a pull request or\n\nmerge locally, with undo afterwards. Then close the other session.\n\nBy hand, remove the losing worktree with\n\n`git worktree remove ../bakeoff-muse` (or `../bakeoff-opus`).\n\nOne ticket is an anecdote. Run three to five tickets of different\n\nshapes before you call it, because a model that wins a refactor can\n\nlose a bug hunt.\n\nHermes has no status signals for Custom agents. It doesn't know the\n\ncommand's approval model, so the attention inbox can't tell when\n\nsession B or C is waiting on you. Watch their terminals. Session A\n\nkeeps full status.\n\nYou're also testing Muse Wonder, not Wonder Full. The API doesn't\n\nserve Full, so only session C, on a plan that includes it, reaches\n\nit. And session B is Anthropic's harness on Meta's model: fair for\n\ncomparing models, maybe not how Meta tuned Wonder to work, which is\n\nwhy session C exists.\n\nHermes IDE is not affiliated with or endorsed by Meta, Anthropic,\n\nOpenAI or Nous Research.\n\nPick the ticket you'd hand an agent this week, run it through both\n\nsessions, and land the one you'd rather review again.\n\nI build Hermes IDE. It is free, source-available, and runs on macOS,\n\nWindows and Linux. It ships often: 2.2.10 came out on October 12,\n\nIf you're designing the agents themselves, the\n\n*AI Agents Pocket Guide*\n\ncovers the patterns behind them.", "url": "https://wpnews.pro/news/don-t-trust-anyone-s-muse-wonder-benchmark-race-it-against-opus-5-5-on-your-repo", "canonical_source": "https://dev.to/gabrielanhaia/dont-trust-anyones-muse-wonder-benchmark-race-it-against-opus-55-on-your-repo-hcb", "published_at": "2026-10-12 20:42:56+00:00", "updated_at": "2026-10-12 20:51:18.748091+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "developer-tools", "ai-tools"], "entities": ["Meta", "Muse Wonder", "Claude Opus 5.5", "Artificial Analysis", "Anthropic", "Claude Code", "Hermes IDE", "TechCrunch"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/don-t-trust-anyone-s-muse-wonder-benchmark-race-it-against-opus-5-5-on-your-repo", "markdown": "https://wpnews.pro/news/don-t-trust-anyone-s-muse-wonder-benchmark-race-it-against-opus-5-5-on-your-repo.md", "text": "https://wpnews.pro/news/don-t-trust-anyone-s-muse-wonder-benchmark-race-it-against-opus-5-5-on-your-repo.txt", "jsonld": "https://wpnews.pro/news/don-t-trust-anyone-s-muse-wonder-benchmark-race-it-against-opus-5-5-on-your-repo.jsonld"}}