Muse Spark Open Benchmark: Open-Weight Models as Coding Agents

An independent evaluation by Leanzero found that Meta's open-weights Muse Spark Open, an 816B-parameter model with 42B active parameters released 15 October, scored 0.091 and 0.0877 out of 1 on the Forge 2.0 agentic coding benchmark, far below the best board score of 0.8523, and failed to complete Gauntlet 7.2 after a provider timeout. Other open models tested locally, including Gemma 4, Granite 4.2 30B, LFM2.5-24B-A2B, Xing 4.0 and Nemotron 3.5 Bolt, fared no better, while a plain Qwen3.8-27B base scored 0.2576 on Forge 2.0 on a 64 GB MacBook. The evaluators noted recurring failure modes such as missing large files, long read-only sessions, tool calls returned as prose, and apps that lint clean but compute wrong numbers.

I asked for this review to come only from runs we already had, and in those runs none of the open models we ran this month kept up with the hosted ones as a coding agent. Meta's open-weights Muse Spark Open came out on 15 October with 816B parameters, 42B of them active. It's a different model from the API-only Muse Spark 1.3. It scored 0.091 and 0.0877 out of 1 on our Forge 2.0 agentic coding build, where the best build on the board scores 0.8523. Its Gauntlet 7.2 attempt never got a verdict. Google's Gemma 4, Granite 4.2 30B, LFM2.5-24B-A2B, Xing 4.0 and Nemotron 3.5 Bolt ran locally on our Macs and did no better, and neither did our own Qwen3.8-27B fine-tunes. Of those local runs, only LFM2.5's is on the board. Open weights had a good autumn on paper. Meta's card puts Muse Spark Open at 76.4 on Terminal-Bench 2.1, and Xing's September card claims 75.0 on SWE-bench Verified. All of these models ship a tool-calling chat template. So we pointed them at the two benchmarks we built for agent work: a payments console on the Gauntlet https://leanzero.net/agentic-benchmarks?family=gauntlet&utm source=devto&utm medium=referral&utm campaign=crosspost and a Jira Forge app upgrade on Forge 2.0 https://leanzero.net/agentic-benchmarks?family=forge&utm source=devto&utm medium=referral&utm campaign=crosspost . Most of them fell over long before the scoring got interesting. Maybe you pay for a hosted coding agent and wonder whether an open model on your own hardware could do the job. The result I'd look at first is the plain Qwen3.8-27B base. It scored 0.2576 on the same Forge build on a 64 GB MacBook, nearly three times what Meta's 816B model managed through OpenRouter. It ran on a newer goose test build with thinking on, so that isn't a clean comparison either. Where each model broke is more useful than the scores, because the same breaks kept coming back. A big file never arrives. A model reads for an hour and writes nothing. A tool call comes back as prose, or an app lints clean and shows wrong numbers. Some of those were the models and some were ours, and I've marked which was which. These numbers are from the live boards as I read them on 20 October; the board itself https://leanzero.net/agentic-benchmarks?utm source=devto&utm medium=referral&utm campaign=crosspost is the current word. A pilot is a run we kept off the board, and the table says why. | Model | Where it ran | Benchmark | Result | Status | |---|---|---|---|---| | Muse Spark Open | OpenRouter host Modal | Forge 2.0 | 0.091 https://leanzero.net/agentic-benchmarks/run/brun-be1699f3-17b2-4c26-8e2d-281cb399867c?utm source=devto&utm medium=referral&utm campaign=crosspost and 0.0877 https://leanzero.net/agentic-benchmarks/run/brun-0b073ef4-e4a7-473a-bf6f-7c1887c9a32e?utm source=devto&utm medium=referral&utm campaign=crosspost , 53rd and 54th of 64 | board | | Muse Spark Open | OpenRouter host Modal | Gauntlet 7.2 | no verdict: provider timeout after 13 calls, $0.19 | not on the board | | Muse Wonder hosted, Meta | OpenRouter | Forge 2.0 / Gauntlet 7.2 | 0.4203 https://leanzero.net/agentic-benchmarks/run/brun-790c0f24-e2fe-4365-ab5e-5b8224251f49?utm source=devto&utm medium=referral&utm campaign=crosspost and 0.2366 / 0.7638 https://leanzero.net/agentic-benchmarks/run/brun-fa48d39e-74b5-47db-b303-cd085019d401?utm source=devto&utm medium=referral&utm campaign=crosspost | board | | Muse Spark 1.3 hosted, Meta | OpenRouter | Forge 2.0 / Gauntlet 7.2 | 0.2968 https://leanzero.net/agentic-benchmarks/run/brun-6eef65cd-4d7e-47b7-bafe-79fa04102e9f?utm source=devto&utm medium=referral&utm campaign=crosspost and 0.2202 / 0.5826 https://leanzero.net/agentic-benchmarks/run/brun-08e52c49-5fe2-4533-b97d-0587791ddad8?utm source=devto&utm medium=referral&utm campaign=crosspost | board | | LFM2.5-24B-A2B, 8-bit MLX | Mac Studio, our engine | Gauntlet 7.2 | 0.0187 https://leanzero.net/agentic-benchmarks/run/brun-384794fe-5fbd-4468-8c7b-79fdceccaeb1?utm source=devto&utm medium=referral&utm campaign=crosspost , 40th of 44 | board | | Xing 4.0 29B-A4B, 8-bit MLX | Mac Studio, our engine | Forge 2.0 | 0.0617, no file written and a re-run cancelled | pilot: our memory guard cut it short | | Granite 4.2 30B, 8-bit MLX | Mac Studio, our engine | Forge 2.0 | 0.0617, no file written | pilot: our memory guard cut it short | | Nemotron 3.5 Bolt 74B-A9B, 4-bit MLX | MacBook Pro 128 GB, our engine | Forge 2.0 | 0.0542, full app built | pilot: under our 0.1 bar, build 2 cancelled | | Gemma 4 31B, 8-bit MLX | MacBook Pro 128 GB, our engine | Forge 2.0 | 0.058, v1 rows broken by an unmigrated ledger | pilot: under our 0.1 bar; build 2 cancelled when I dropped the family | | Gemma 4 26B-A4B, 8-bit MLX | MacBook M3 Max 64 GB, our engine | Forge 2.0 | 0.0237, no function loaded | pilot: under our 0.1 bar, build 2 cancelled | | Gemma 4 12B, 26B-A4B and 31B, nine more runs | Studio and both MacBooks, our engine | Forge 2.0 | no verdict: our engine, a goose bug, a loop, cancels | no verdict | | Qwen3.8-27B-Atlassian v2-goose ours | Mac Studio, our engine | Forge 2.0 | stopped at 106 calls, no file written | no verdict | | Qwen3.8-27B-Atlassian v1 ours | Mac Studio, our engine | Gauntlet 7.2 | two attempts, no verdict | no verdict | | Qwen3.8-27B base, our 8-bit conversion | MacBook M3 Max 64 GB, our engine | Forge 2.0 | 0.2576 https://leanzero.net/agentic-benchmarks/run/brun-4616377e-8e7f-4f34-9ca3-26c2dd8b7d46?utm source=devto&utm medium=referral&utm campaign=crosspost and 0.1128 | board | On Forge 2.0, 0.0617 is what the unchanged starter app scores, so a model sitting exactly on it wrote nothing. Xing 4.0 and Granite 4.2 30B sit on it, and Bolt sits just under it. Both scored Gemma 4 builds sit under it too: one for a missing migration, one for a name declared twice. Meta's 76.4 on Terminal-Bench 2.1 comes from Artificial Analysis's own run at xhigh effort in the Terminus 2 harness. The other independent numbers are on different scales. Artificial Analysis gives it 39 on its Intelligence Index the median for comparable models is 19 , and Vals AI's page shows 43.33% accuracy, with 6.57% on the newer, harder Terminal-Bench 4.0. We ran it at "medium" effort in our own harness. The Gauntlet asks for a payments console in 150 calls: two services that sync from a vendor and take its webhooks, plus a 3D view. Most of the work sits in one Python service, app/ledgerd.py , and a good build writes most of it in one go. Muse Spark Open got there fast. In its first minute it wrote the transport layer 91 lines and the notifier service 90 lines . Then it started on the ledger, four times. Three of the attempts opened with one line of reasoning, "Now ledgerd - big." or "Now ledgerd - huge."; the other streamed a short plan for six seconds. Then nothing reached our side for about two minutes until the connection dropped: The provider failed mid-answer Server error: Upstream idle timeout exceeded . Resending — attempt 1 of 3. The unfinished answer was discarded. Our agent, a fork of the open-source goose, resent the same request three times and got the same silence: 121,970, 127,589, 125,370 and 121,134 milliseconds. Then the run ended, and app/ledgerd.py in the tree is still the 23-line stub we hand each model. It cost 19 cents. I wouldn't call that a model result. All 13 Muse Spark Open calls went to one host, Modal, and the error is an upstream idle timeout. The model didn't refuse. It hadn't run out of budget. I think the route sends nothing while the model builds a long tool call. A big file write then looks idle and trips a timeout at around two minutes. I can't prove that from the files. I can't tell whether OpenRouter or Modal raised it. On Forge 2.0, on the same host, its longest completed call took 80 seconds. A ledger the size its siblings wrote would need closer to four minutes. Its siblings did the same thing and got through. Muse Wonder wrote an 845-line ledgerd.py in one write call of 45,841 characters and finished the Gauntlet at 0.7638, 9th of 44. Muse Spark 1.3 wrote a 1,173-line one in one call of 54,910 characters, after a single 168-second call, and finished at 0.5826. Both were served by Meta's own endpoint, where Muse Spark 1.3 survived calls of 394 seconds. Both also ran earlier in October on older goose builds, and Muse Spark 1.3's run sent no effort setting. Our own v1 fine-tune lost the same file to a different limit, on our own engine, two weeks earlier. On 6 October it streamed one write call for 50 minutes: 44,238 characters, 1,021 lines, a complete-looking payments backend. Our engine stopped it at exactly 32,768 tokens, its default output cap when the client sends none, and goose sends none. Only the content arrived, with no path: Error: Failed to parse arguments: missing required field path . Arrived: content . Nothing was changed. My guess is that it writes the content before the path, so the cut took the path with it. Its next words were "I've already written the ledgerd backend." That was half true. A 20,170-byte version from earlier in the run was on disk, and the 44,000-character one was lost. Its next write, a much smaller file, also arrived without a path. So I doubt the cap explains all of it. We raised the cap in our serve script the same morning. On the relaunch, v1 wrote a 900-line ledger in 16 minutes. Then it spent one call, 99 minutes long, trying to "clean up a buggy nested method" it had left as a placeholder returning "Drafts not yet wired." We stopped it there. An agent that writes whole files in one call is betting that everything between it and the disk stays patient. Meta's endpoint was patient. The OpenRouter route to Modal wasn't, and neither was our 32,768 default. If you serve an open model yourself, check your output cap and your proxy's idle timeout before you blame the weights. Our side contributed to that failure too. We didn't pin the OpenRouter route to a host, our benchmark runner doesn't retry a run that ends without a verdict, and our harness filed the stop as "model finished". So Muse Spark Open has no Gauntlet score on our board. Forge 2.0 hands the model a working Jira Forge app a sprint scope widget and a contract for version 2. It asks for a new storage entity and a migration of the v1 rows. It also wants an admin page, a signed CI web trigger, a custom field, a JQL function and a rate budget the app must respect. The model gets 300 calls and an offline dev kit that runs the app's functions and events, and renders its UI Kit pages, without a live site. The prompt tells it to "plan to implement, test, and finish well inside" the 300 calls, and also says "Time is virtual: nothing is graded on wall-clock time." About 33 calls into build 1, Muse Spark Open wrote this in its reasoning: Need to continue implementing other modules quickly but we have limited calls. We need to prioritize delivering something that will score. Build 2, a minute in: "Need to continue building but time limited." Build 1 stopped at 58 calls after 7 minutes 25 seconds, and build 2 stopped at 44 calls after 6 minutes 9 seconds, both times with the model's own summary and no error. Over 80% of the budget was left unused both times. I don't know where that came from. The rush shows in the code. Build 1's event handler calls a function that does nothing: // src/index.js:93-94, Muse Spark Open, Forge 2.0 build 1 async function mirrorToLedger body, cfg { // no-op for now: actual ledger writes happen via sync.js which we can extend to write both entities The comment is accurate, I'll give it that. Build 1 also wrote scopeLedger.js , dosing.js and admin.js , and nothing imports them. The scorer found 0 of 170 event rows in the new ledger, which is what a no-op produces. Build 2 did better there, with 132 of 170 event rows. But it copied the v1 rows with estimate: 0, boardId: '' hard-coded for all of them src/migration.js:67-69 , and its own summary admits it. The admin page is where build 1 went wrong. The dev kit said that the admin function "does not load: Error: Unable to establish a connection with the Custom UI bridge." Lint then warned that the function's render export didn't exist. It fixed the warning and left the problem: js // src/admin-ui.jsx:29 export const render = = {}; Its reasoning a few seconds later: "The harness's own UI Kit host may be different and may pass despite dev tool error." It didn't. Build 2 wrote layout: native where the contract says render: native . It never checked its page. Neither build's admin page ever rendered, so the admin panel scored zero. In build 2 I think that also zeroed the CI trigger. No panel means no "Rotate CI secret" button and no secret. The contract says every request then gets 401, and all of them got 401. Build 1's trigger had a separate bug. It returned { statusCode, body } where the contract asks for { outputKey } . It answered 500 to everything. With that many requirements at zero, the reliability multiplier it scales the score down for every group of failing checks, never below 0.25 hit its floor. You can see it on the run page https://leanzero.net/agentic-benchmarks/run/brun-be1699f3-17b2-4c26-8e2d-281cb399867c?utm source=devto&utm medium=referral&utm campaign=crosspost : 0.3638 tests earned, times 1.0 for criticals, times 0.25. The best build on this benchmark used fewer calls than Muse Spark Open's build 1. jev-router, a router that in our runs sent nearly all its calls to OpenAI's GPT-6.1 Sol our write-up https://leanzero.net/blog/jev-router-benchmark-gpt-6-1-sol?utm source=devto&utm medium=referral&utm campaign=crosspost , scored 0.8523 in 43 calls. It ran the offline dev kit in 26 of them, 70 times in all: the scheduled reconcile, event replays, UI Kit checks, the CI sender. Muse Wonder used 126 and 117 calls, ran the scheduled job 13 and 11 times, and declared the admin page the way the contract says, render: native pointing at src/frontend/admin.jsx . Its admin actions scored 1.0 in both builds. I'd copy jev-router's habit before anything else. Our board groups runs that share the same task, kit, scorer and goose build into comparison groups, and the Muse runs are in different ones. They ran on 14, 15 and 19 October with identical task and scorer hashes, so my guess is the goose build differs. None of the ten Muse Wonder Forge requests we kept carried a reasoning-effort field, so it likely ran at its provider's default, while Muse Spark Open got "medium". jev-router sits in a fourth group. Muse Spark Open lists at the same OpenRouter price as Muse Spark 1.3 $1.25 in, $4.25 out per million tokens; Muse Wonder is $4 and $20 . But Spark Open's calls hit zero cache on Modal while the siblings cached heavily on Meta's host. Meta says the open release is "derived from the Muse Spark model we released in August, with additional safety post-training". Its PyTorch post adds that it "shares the same base model as Muse Spark 1.3". So the base is an unlikely culprit. Was it the safety training, the effort setting or the route? I can't tell from two builds. The card doesn't say. These ran on our own MLX engine a fork of Rapid-MLX through the Goose Swarm app https://leanzero.net/portfolio/goose-local-edition?utm source=devto&utm medium=referral&utm campaign=crosspost , the same app that drives our cloud runs. One setting matters for every Forge run in this section. The app sent no reasoning setting, and our engine switches thinking off on any request with tools unless told otherwise. So Xing, Granite and Bolt all ran Forge 2.0 with thinking off, while Muse Spark Open got "medium". I didn't connect it to these runs until we reviewed them, though we'd written the behaviour down in September for our training data. Liquid's model has 24B parameters with 2.45B active, 30 short-convolution layers and 10 attention layers, and it always reasons before answering. On the Mac Studio it decoded at 85.7 tokens a second with a peak of 27.0 GB, the fastest local model we ran. Its whole run took 14 seconds. After about a thousand tokens of reasoning, its answer was this, as text home folder redacted : shell command='ls -la /Users/<user /Library/Application Support/Goose/benchmark/runs/build/openai-cloud-4ef38a6e-...-r0' 2 /dev/null || echo "ls failed"' Goose saw no tool call. It took the turn as a final answer, and the run was over after 2 of 150 calls, before it had read a line of the task. The string isn't a valid call even by LFM's own Pythonic format a quote closes too early and a parenthesis dangles . It also passes a path with a space in it without quotes, so ls would have failed anyway. The engine had LFM's tool parser switched on. Its boot log shows it picked that format from the model's own files. Which layer turned it into text, the model leaving out its tool-call tokens or the parser rejecting a malformed call, the logs can't tell me. Claude Haiku 5.5's first move on the same benchmark was a proper shell call with the path quoted: cd "/Users/<user /Library/Application Support/..." && ls -la && ... cat SB7-CONTRACT.md . Haiku went on to 0.661. The October roundup already flagged LFM's score https://leanzero.net/blog/best-model-for-agentic-coding-october-2026?utm source=devto&utm medium=referral&utm campaign=crosspost as no verdict on the model yet, and one run doesn't change that. Xing 4.0 is China Telecom's 29B-A4B model successor to TeleChat , with multi-head latent attention and hyper-connections. mlx-lm doesn't support its architecture yet, so we ported it into our engine. The port matched the reference on a small test model but hadn't been tried on the real weights before this run. On Forge 2.0 Xing spent its first call asking for a tool that doesn't exist: read {"path":"FORGE2-CONTRACT.md"} -32002: Tool 'read' not found ... All 5 available tools, closest first: read image, write, edit, tree, shell Then bash eight times, read image on a Markdown file, and 22 edit calls on the contract with placeholder text "APPENDED", "APPENDED2", "APPENDED3" , apparently trying to read it through the error previews. Its own words at 21:57: "Let me read the files through the read image tool in a different way:". Its one working shell call came at 21:54, and the next call went straight back to read image . It wrote no files. I don't blame the parser for this one. The engine parsed those calls correctly and logged that the model asked for read , which the request never offered, so those names are the model's. My guess is that read and bash are what popular coding harnesses call their tools, and it reached for them by reflex. Cloud models do this too: Mistral Large 4 lost 48 calls to "Tool 'bash' not found" on the same benchmark. jev-router's call 2 on the same task was cat STARTER.md FORGE2-CONTRACT.md RATE-MODEL.json BROWSER-TESTING.md , and it wrote its first new file less than five minutes in. The run is a pilot because our engine cut it short. The memory guard capped Xing at 35,328 tokens on a 96 GB Mac. It priced the compressed attention cache at about 25 times what it needs. The session died on a failed compaction at call 43. We fixed the pricing that same night. A re-run on the fixed engine called read image on text files 121 times without writing anything before we cancelled it. IBM's Granite 4.2 30B is a plain dense transformer 64 layers, and no Mamba layers anywhere in its config . It was the most careful reader in the set. After a bad start four read image calls on text files, then a cat that broke on the space in "Application Support" , it said "the cat command had path issues with spaces. Let's use shell with proper quoting" and read the contract, the rate model, the manifest and all nine source files cleanly. Its notes were correct: "The manifest doesn't declare a jira:adminPage yet." Then Granite ran find <run folder | grep -v node modules | grep -v build and got " no output ", because our run folder lives under .../runs/build/ , so the filter erased its own workspace. I'd put that one on our folder layout. And after goose compacted its context, the model's own summary said "Ready to begin v2 implementation", and its next call was tree depth 3 . That lists the dependency folder, about 65,000 tokens, into a window our engine had capped at 69,888. Compaction failed. The run ended at call 40, with no write calls in 41 minutes. A second build hit the same cap on its second call, 16 seconds in. The cap was our bug. The guard priced Granite's cache at double its size, and we fixed it about an hour later. The node modules folder in the run directory is our harness too. Would Granite have built anything with the right window? A re-run on a 64 GB MacBook, where it only gets 52,480 tokens, made 106 calls and no writes before we stopped it. Our own log says that run measured the Mac's memory more than Granite. It also prefilled at only 171 to 275 tokens a second on the Studio another session was rendering video at the time, which may have slowed it . Each 70,000-token tree dump cost six or seven minutes. Bolt is NVIDIA's 74B-A9B compression of Nemotron 3.5 Super VL, a hybrid of 40 Mamba-2 blocks, 40 MoE blocks and 8 attention blocks. We ran it at 4-bit 43 GB on a 128 GB MacBook Pro, with a 262,144-token window, at a median 30.6 tokens a second. Of the four models in this section, it's the only one that behaved like an agent. It used 292 of 300 calls over 2 hours 16 minutes. It read carefully and fixed its own syntax error "The error is at line 272" . It left a v2 that lints, builds and would deploy. Then we looked at the numbers. They were wrong. Bolt's ledger sums raw story points: js // src/ledger.js:118, Nemotron 3.5 Bolt const estimate = member ? Number member.estimate || 0 : 0; The v1 views it left unchanged expect micro-points and divide by 1,000,000. So the widget the scorer opened said: Committed 0.0002195 Added 0.000057 Removed 0.0001365 Creep NaN.NaN% A second bug at src/sync.js:351 divides twice, so a 5-point issue writes "added +0.000005" into the custom field. The admin page is declared without render: native , which zeroed six checks. It ran the dev kit 18 times, and not one of the outputs it read contained a total. In its last 23 minutes it checked only build and lint, then declared itself done: "Build and lint pass ... All the v2 surfaces are present in the manifest." Score 0.0542: 0.3615 tests earned, times 0.6 for corrupted v1 rows, times the 0.25 floor. jev-router kept v1's units committed: fromMicro totals.committed , views.js:20-23 and declared the admin page with render: native . Then it ran the migration in the dev kit, read the storage dump, and reported "The first upgrade run migrated all 88 seeded v1 rows". Bolt never read its own storage back. I think that habit is what separated the two. The run is a pilot because build 1 scored under our 0.1 bar, so under our rule build 2 was cancelled and nothing was published. I dropped the whole Nemotron family the same day and had its weights deleted. | Model quant | Mac | Decode tok/s | Prefill tok/s | Peak memory | Context it really got | |---|---|---|---|---|---| | LFM2.5-24B-A2B 8-bit | Studio M3 Ultra 96 GB | 85.7 | 824.6 | 27.0 GB | never tested past 2,546 tokens | | Xing 4.0 29B-A4B 8-bit | Studio 96 GB | about 13 | 347 to 941 | 39.7 GB | 35,328 our guard bug | | Granite 4.2 30B 8-bit | Studio 96 GB | about 13 | 171 to 275 | 67.8 GB | 69,888 our guard bug | | Granite 4.2 30B 8-bit | MacBook M3 Max 64 GB | not recorded | not recorded | not recorded | 52,480 | | Nemotron 3.5 Bolt 74B-A9B 4-bit | MacBook Pro 128 GB | 30.6 median of 205 calls | not recorded | not recorded 43 GB of weights | 262,144 | Decode speed didn't predict much. The fastest model made one tool call and the slowest finisher built the most. In these four runs, one each, what separated them was the context a model really had and whether it read only what it needed. A dense 30B at 8-bit needs more than 64 GB for this kind of work. On 64 GB it gets 52,480 tokens, too few to hold the contract and the code at once. On the 96 GB Studio, after our fix, it gets its full 131,072. Google released the Gemma 4 31B dense and 26B-A4B mixture of experts in April and a 12B in June. Its launch post calls the family "purpose-built for advanced reasoning and agentic workflows", and the card promises "native function-calling support". So I'd call this a fair test of what Google pitches. What Google doesn't publish is a repository-level coding-agent number. Its card reports LiveCodeBench and tau2 76.9 for the 31B , and its technical report puts Terminal Bench Hard at 36.0 for the 31B and 14.0 for the 26B-A4B. It's a general family pitched at agents, not a model tuned and measured as a coding agent. We gave it Forge 2.0 eleven times on three Macs on 18 and 19 October. Two runs got a score, 0.058 for the 31B and 0.0237 for the 26B-A4B, and both are under the 0.0617 that the starter app we hand every model scores when nothing is changed. The other nine ended without a verdict, and most of those nine were our doing. Our cleanest run was the 31B on the 128 GB MacBook Pro, on the app release that carried our cache fixes. Its prompt cache reused a median 95% of each prompt and no tool call failed to parse. It read the contract and the source files and ran lint. Then it made the decision that sank it: js // src/ledger.js:5-6, Gemma 4 31B, Forge 2.0 128 GB MacBook Pro export const CHANGES = 'scope-ledger'; export const V1 CHANGES = 'scope-change'; That points the live ledger at the new entity. V1 CHANGES is never used, and no migration moves the old rows across. Our scorer found "migrated v1 payload corrupted at 371 checkpoint row s ". It counts that as a critical defect, which multiplies the score by 0.6. Bolt failed that check too. It hit the same 371 rows. Apart from that, it beat the starter. It scored higher than the starter on seven checks, four of them in full, and lost none the starter passes. Its test score before the multipliers was 0.3727 against the starter's 0.2468. Without the critical defect it would have scored 0.0967, still under our 0.1 bar. Most of its 39 minutes went into typing: three big writes took about 24 minutes at around 7 tokens a second. Its second sync.js write was identical to the first, and goose said so "This exact call already returned this same output" . Its last write, the 176-line index.js , came out byte for byte the same as our starter. Then, at 29 of 300 calls, it stopped and wrote a plan: "Next steps: Implement the live migration from scope-change to scope-ledger §9 ", followed by six more sections it hadn't built. It never ran the bundler. Why stop there? I don't know, and the transcript doesn't say. Goose puts the current time in the turns it sends, so the model could see 38 minutes go by. But the task says nothing is graded on wall-clock time, and it had 271 calls left. Its thinking was off. Our app sent no reasoning setting, so our engine switched it off, and each stored thought is a single newline. Gemma 4's thinking is opt-in anyway a <|think| token in the system prompt , but Google's technical report says its benchmark numbers are "in thinking mode unless explicitly stated". Our plain Qwen3.8-27B base got this right on the same benchmark, in one build on the 64 GB MacBook M3 Max with its thinking on: it migrated 59 of 59 v1 rows. jev-router, the cloud router that ran mostly GPT-6.1 Sol, ran its migration in the dev kit and read the storage back before it claimed anything. On the 64 GB MacBook M3 Max the 26B-A4B ran with thinking on that Mac's goose build sends the reasoning setting and decoded at a median 18.9 tokens a second. It made 68 calls in 76 minutes. About halfway through, at 04:37, it wrote this: js // src/index.js:178 and :249, Gemma 4 26B-A4B, Forge 2.0 64 GB MacBook const adminResolver = new Resolver ; // ... export const adminResolver = adminResolver.getDefinitions ; Bundling stops on that, with The symbol "adminResolver" has already been declared . All nine of the app's functions live in that file, so no function loaded. Checks the unchanged starter passes went to zero. The starter gets 170 event rows exact; this build got none, and its 130-row backfill went from complete to nothing. So it scored 0.0237. It had no critical defect. The v1 rows weren't corrupted, because no v1 row ever reached the new ledger. It never ran the bundler. It never saw that error. Its last 25 minutes went on a manifest lint error instead, "jira:jqlFunction required properties are 'arguments, function, name, operators, types, key' or ...". It rewrote the whole manifest six times between 04:49 and 05:07. The first switched the argument and its types from string to number. The rest changed little beyond key order and quoting one added a response block to the webtrigger and the next took it out again , and two of them were byte-identical. The error stayed. From its thinking: 04:52 I'm going crazy. I'll just rewrite the jira:jqlFunction block with the requested order. 04:56 Wait I have them all 05:03 Wait, let's look at the order again. Maybe it matters? Unlikely for YAML. Its manifest needed required: true on the argument and types: issue , as in the Qwen base build on the same Mac. That build also named its resolver instance something else and exported the definitions under the name the manifest expects: // src/index.js:190 and :206, Qwen3.8-27B base, Forge 2.0 build that scored 0.2576 const adminModule = new Resolver ; export const adminResolver = adminModule.getDefinitions ; All nine functions loaded there. Gemma isn't alone in this one. Another 31B run on the same MacBook and a local pilot of Muse Glimmer 30B 0.0325 wrote the same pair. I'd guess they copy the starter's export const uiResolver = resolver.getDefinitions line and reuse the export name for the instance. To its credit, it caught a bug of ours. Goose's edit tool reported five edits as successes that had changed nothing, three of them in a row early on. It noticed after the first one "Ah, I failed to actually edit anything because the before and after were the same" , sent the same no-op twice more, and after the third gave up on the edit tool: "I'll just use write to replace the whole file." Our engine's repetition guard then cut ten looping replies and reported each one as hitting the output limit, and goose ended the session after the third cut reply in a row, at 68 of 300 calls. Goose's repeat check also appended its notice to a cat output, and the model read that as a corrupted file. | Model | Mac | Calls | How it ended | Cause | |---|---|---|---|---| | 26B-A4B | Studio 96 GB | 65 | four one-token replies; goose resent the identical request three times; our runner refused the run | goose bug and the model | | 12B | Studio 96 GB | 10 | our engine refused, wrongly: it priced the turn at 42.6 GB of memory, about nine times the real ~4.7 GB | our engine | | 31B | Studio 96 GB | 16 | first token after 174 seconds, then a one-token reply, resent three times; our runner refused the run | goose bug and the model | | 31B, two runs | MacBook M3 Max 64 GB | 37 and 14 | cancelled: an older engine re-read the whole prompt on every call first token took up to 379 seconds | our engine | | 31B | MacBook M3 Max 64 GB | 36 | cancelled: we moved the model to the faster Studio | our choice | | 12B | MacBook M3 Max 64 GB | 41 | cancelled: one 28,525-character reply repeated a 530-character paragraph about 50 times | the model, with no reply cap from us | | 26B-A4B, build 2 | MacBook M3 Max 64 GB | 15 | cancelled: build 1 scored under our 0.1 bar | our rule | | 31B, build 2 | MacBook Pro 128 GB | 3 | cancelled when I dropped the family | me | All three Studio runs hit the same goose bug. With thinking off, Gemma 4 writes a reasoning block that is only a newline on almost every turn 58 of 59 on the Studio 26B-A4B . Goose took identical reasoning as a sign that tool calls came from one reply, and merged the later turns into a single assistant message, so the 26B-A4B's last request held 47 tool calls in one message. That defeats the prompt cache. Our engine re-read most of each prompt on every call: everything past the first 2,600 to 2,850 tokens on the 31B and the 12B, and everything past 23,417 on the 26B-A4B. The 31B's wait for a first token grew from 46 to 174 seconds. Then the 26B-A4B and the 31B each answered with a single token. Goose sent the identical request three more times and got one token back each time. I can't prove the merge caused that. I still count it as ours. On the 64 GB MacBook, where turns weren't merged and that Mac's goose turned thinking on, Gemma had no one-token replies in 76 requests, so I can't tell the merge from the thinking setting. We fixed the merge and the resend in goose 3.0.119 on 19 October, and the 128 GB MacBook Pro run above, on 3.0.119 according to our run ledger, has no merged turns. Left alone, I think the 12B's loop could have run for hours. Our engine let a local reply grow to the whole context window, and goose sent no cap. Since 3.0.122 our engine stops long exact repeats. Gemma's own slips show even in the broken runs, but I'll put goose's part first. The Studio 26B-A4B made 22 edits and 19 changed nothing, and goose reported each as a success, in a history it had merged. The model also declared const MAX RETRY AFTER = 900; three times at the top of index.js and imported a ./ci.js it never wrote. And the Studio 31B made the same scope-ledger switch as the MacBook Pro one. | Gemma 4 8-bit MLX | Mac | Decode tok/s, median | Time to first token | Engine peak memory | |---|---|---|---|---| | 31B | MacBook Pro 128 GB | 7.05 28 calls | median 12.3 s | 52.2 GB | | 31B | Studio 96 GB | 11.7 8 calls | 45.6 to 174.6 s merged turns | 45.1 GB | | 31B | MacBook M3 Max 64 GB | 6.6 | up to 379 s older engine | not recorded | | 26B-A4B | Studio 96 GB | 33.1 56 calls | median 25.5 s merged turns | 34.6 GB | | 26B-A4B | MacBook M3 Max 64 GB | 18.9 | median 3.1 s | 37.0 GB | I dropped the Gemma 4 family on the morning of 19 October, right after that MacBook Pro run, and had its weights deleted from all three Macs. The run folders stay as evidence. None of the eleven runs matched Google's setup: all used 8-bit weights in our own harness, and five of them had thinking switched off by our engine. That includes the cleanest, the 31B on the MacBook Pro, and it still passed more of the checks than the starter before one migration decision cost it the score. I'm left with two pilots and no verdict on the model. These are our models, and I signed off on both releases. Neither has produced a usable agent run. Since September we've trained Qwen3.8-27B on Atlassian work and published the results on Hugging Face. Our first one was released as v0.5 https://leanzero.net/blog/atlassian-models-v0-5-qwen3-8-27b-gguf?utm source=devto&utm medium=referral&utm campaign=crosspost , and I call it v1 below. The second, v2-goose https://leanzero.net/portfolio/atlassian-models?utm source=devto&utm medium=referral&utm campaign=crosspost , came out on 5 October, trained fresh from the base to work as a goose agent. The fine-tuning tutorial https://leanzero.net/tutorials/fine-tune-qwen3-8-27b-lora-mac-mlx?utm source=devto&utm medium=referral&utm campaign=crosspost shows how we train them. v2-goose's first Gauntlet attempt on 5 October started right. Its first move was to read the contract, in well-formed Qwen tool-call format: <tool call <function=shell <parameter=command ls -la && cat SB7-CONTRACT.md </parameter </function </tool call It came back from our engine as plain text three times, and on its fourth turn the model wrote Calling tool: unparseable tool call {} , copying the placeholder name it had seen in its own history. Our serve script caused it. Rapid-MLX only parses streamed tool calls when it gets --enable-auto-tool-choice next to --tool-call-parser , and we had copied our flags from a release script that left it off. Our pre-flight check had used a non-streamed request, which parses through a fallback, so it passed. We added the flag that afternoon. With it on, the same model streamed a parsed call on the first try. If your local model's tool calls arrive as text in vLLM or a Rapid-MLX server, check that flag first. The second attempt ran 81 calls. The model was debugging its own blank table "viz.brush is not a function", 0 rows rendered and asked for a screenshot. Our engine answered "Model ... is serving text-only; image input is unsupported", and the run ended. The checkpoint has a full vision tower 333 vision tensors . The cloud-hosted base reads images fine. We had served it text-only: our serve script passed a text-only flag copied from the release script, and the speculative-decoding head we ship forces that lane anyway. The third Gauntlet attempt ran 135 calls, and it shows both sides of the model. The back end is good. All 12,290 payments synced through the paged walk. It resumed after a dropped connection, answered forged webhooks with 401, kept sends idempotent and survived a SIGKILL. Currency totals were exact to the minor unit with no cross-currency sums. The front end stayed exactly as we'd shipped it. web/app.js in the tree is still our one-line starter, "// Implement live data, page journeys, custom controls, and cross-view state here." Its final summary didn't match the disk. It claimed "four files with the required ids" in web/ , a DECISIONS.md that doesn't exist, and "150/150 turns" at turn 134, none of which our tree checks found on disk. On our older Forge 1.0 benchmark the same model wrote a 385-line resolver, three UI files and a Rovo skill. Its manifest.yml had a duplicate key: under a queue consumer at line 54, so the manifest didn't parse and nothing could deploy. It ran lint six times. It got the same crash each time. At turn 124 the validator told it exactly: "Map keys must be unique ... line 54, column 5". It rewrote the manifest and kept the duplicate. Then it ended with "The linter and forge-dev both crash in this environment", and claimed built bundles in a folder with zero files. That one is mostly the model; the caveat is that our serve script had thinking off that day. On 19 October we gave v2-goose Forge 2.0 on the Studio. It made 106 calls in 31 minutes, all of them reads: the contract, the v1 source, the dev kit's internals, the manifest schema. After call 77, at about 103,000 tokens, goose compacted the context to about 11,000, and the summary the model wrote for itself said: Done so far: full recon. Read the whole contract, STARTER.md, RATE-MODEL.json, all v1 source ... Next step Write the v2 implementation. Plan, in order: 1. manifest.yml: add jira:adminPage ... It went back to ls -la; cat STARTER.md , then tree , then cat FORGE2-CONTRACT.md , and read everything again. We stopped it at call 106 under our kill rule, and the run ledger says why: no source edits in 31 minutes. Our engine had thinking switched off for this run. The plain Qwen3.8-27B base, in our own 8-bit conversion, first changed an app file at call 43 in the build that scored 0.2576 and at call 34 in the one that scored 0.1128, on the same benchmark https://leanzero.net/blog/qwen3-8-27b-fine-tunes-coding-benchmark-mac?utm source=devto&utm medium=referral&utm campaign=crosspost . Those base runs used a goose test build with a new compaction, and the base kept its thinking on. On the shipped compaction the same base also re-read its files for about 33 calls before writing, so the restart after compaction isn't evidence against v2. What does count against v2 is 77 calls of reading before the compaction, where the base was writing by call 43 at the latest. v2 shipped because it beat v1 on the tool-call slip rates above all: | Release gate | v2-goose | v1 | What it measures | |---|---|---|---| | Tool-call slips: missing required argument | 0.37% | 1.57% | one next action, replayed from recorded v1 sessions | | Tool-call slips: repeated call | 2.31% | 4.07% | same | | Invented tool name | 0.09% | 0.19% | same | | BFCL public function-calling benchmark | 91.8 | 85.7 | single-turn calls | | 199 Atlassian multiple-choice questions | 74.37% | 71.86% | knowledge, one answer difference not significant | | 30 free-form Forge questions | 63.33% | 86.67% | knowledge, one answer worse, p 0.0156 | | Complete Forge apps from 35 briefs | 27.33 | 30.0 | one-shot drafts, manifest and tsc | All seven rows are single-step checks. I didn't see that as a problem then. The pipeline did have one end-to-end check, a real task run in the Goose app, and it was marked not run, to be done after publication. The goose "FINAL" check behind those slip rates takes a recorded session, cuts it at one point and scores the model's next action. Its own code admits it "UNDER-counts versus live goose". Our training rows were mostly short 99% under 16,313 tokens, the longest 65,536 , while the benchmark runs reached 87,000 to 103,000-token prompts over 80 to 300 calls. I don't think anything in our mix taught it to resume from its own summary or fix the line a validator points at. Nor to come back to the half of the product it skipped. We never checked the 527 goose sessions in it for that, though. And seven gates failed. The 30-question Forge gate failed. The app-build gate fell below v1, and so did a thinking-format check. The 8-bit build failed its own accuracy check on goose contexts up to 16,384 tokens a KL of 0.400 against a limit of 0.067 . It looped once more than v1 in 160 answers, missed four style marks on the voice check, and failed the goose gate on one metric: in the two cases where it should have asked the user first, it never did. We waived all seven to ship on 5 October, for a deadline I set myself. People kept downloading v1. Its repos Q8, Q6, Q4, GGUF and LoRA have 1,075 downloads in the last 30 days. v2-goose's four repos, up for 15 of those days and with no GGUF build, have 179 Hugging Face's counts on 20 October . Before the models in the 10-19 article https://leanzero.net/blog/qwen3-8-27b-fine-tunes-coding-benchmark-mac?utm source=devto&utm medium=referral&utm campaign=crosspost , we ran four other Qwen3.8-27B derivatives on a 64 GB MacBook in mid-October. BAAI's AREX-2 a long-horizon agent fine-tune scored 0.0617 in both app runs with no files written, and 0.0014 in a run our engine ended. bytkim's Qwen3.8-27B-pi, tuned for the Pi agent harness, wrote nothing in one run. In the other it wrote four files and then drafted the rest of its v2 as prose. I cancelled it at 166 calls. JetBrains' Qwen3.8-3.6-27B-blend, a 50/50 weight merge with no training, scored 0.0321, 0.0522, 0.0938 and 0.0042. Abacus.AI's Smaug-Mini wrote nothing once 0.0617 and a manifest with an unquoted colon at line 79 the other time 0.0014 . Our own ledger says those first-round scores "are not a fair verdict on these models". Our engine dropped the reasoning from each past turn, and two more serving bugs turned up afterwards, and both affected all four models. The plain Qwen base, run as a control on a partly fixed engine that still had one of those bugs, also wrote no app code in 62 calls before we cancelled it. On the fully fixed engine the same base wrote 17 files before our memory guard forced us to stop it. Only the JetBrains blend got a re-run on a partly fixed engine, so I'm not ranking them. I'd sum up Muse Spark Open as fast, confident and too quick to stop. Two builds and one aborted Gauntlet attempt are thin, though, and the Gauntlet loss belongs to the route more than the model. LFM2.5-24B-A2B is very fast on a Mac and couldn't drive this harness in its one run. Xing 4.0 reaches for tools that aren't there. Granite 4.2 30B reads and never writes. Bolt works like an agent and doesn't check its numbers. In two pilots, one build per size, Gemma 4 wrote and rewrote more than it tested, and neither ran the bundler on its own code. The Xing, Granite and Bolt pilots ran with thinking off, as did the Gemma 31B pilot, so I'd re-run them before calling any of it a verdict. I wouldn't trust any of these local models without reading a transcript first. Our serving cost v1 its first Gauntlet attempt, and on the relaunch it stalled on one 99-minute call. Our v2 has better single-step tool manners and doesn't finish anything. Both are ours to fix. Muse Spark Open has 816B parameters in 256 experts, 8 of them active per token 42B active . It mixes 46 sliding-window attention layers with 16 full ones, reads a 1M context, and its reasoning can't be turned off. Meta's card asks for at least 700 GB of GPU memory at NVFP4. Hugging Face's own launch post ran a 2-bit quant on a 512 GB M3 Ultra at 18 tokens a second, using 304 GB. Our 96 GB Studio wasn't going to run it, so it ran through OpenRouter, which is where I'd point anyone without a 512 GB Mac. | Model | Architecture from the card and config.json | Context | Licence, quoted from its LICENSE file or card | Commercial use | |---|---|---|---|---| | Muse Spark Open https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B | MoE 816B / 42B active, 256 experts top-8, 46 sliding + 16 full attention | 1,048,576 | Muse Model License: "Licensee shall prominently display 'Built with Muse Spark Open'", and a company offering the model as a service whose group revenue tops $20M "must notify Meta and enter into a separate written agreement" | Yes, with attribution; conditional above $20M for a model service | | LFM2.5-24B-A2B https://huggingface.co/LiquidAI/LFM2.5-24B-A2B | MoE 24B / 2.45B active, 64 experts top-4, 30 conv + 10 attention | 131,072 | LFM Open License v1.0: commercial rights "conditioned upon You or Your Legal Entity not exceeding the Threshold" annual revenue of $10M | Conditional: under $10M revenue | | Xing 4.0 29B-A4B https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B | MoE 29B / 4B active, MLA attention, hyper-connections | 262,144 | Apache License 2.0 | Yes | | Granite 4.2 30B https://huggingface.co/ibm-granite/granite-4.2-30b | dense, 64 layers, GQA | 131,072 | "All Granite models are released under the Apache 2.0 license" | Yes | | Nemotron 3.5 Bolt 74B-A9B https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 | hybrid: 40 Mamba-2, 40 MoE top-9 , 8 attention blocks | 262,144 | OpenMDW-1.1: "Subject to your compliance with this agreement, permission is hereby granted, free of charge, to deal in the Model Materials without restriction" keep the licence and notices when redistributing; the repo's LICENSE file is empty and the card points to OpenMDW | Yes | | Gemma 4 31B https://huggingface.co/google/gemma-4-31B-it | dense, 30.7B, 60 layers, 5 sliding-window 1,024 tokens to 1 global | 262,144 | Apache 2.0: the card's licence, and Google's launch post says "released under a commercially permissive Apache 2.0 license" the Gemma Terms of Use page points Gemma 4 to its own licence | Yes | | Gemma 4 26B-A4B https://huggingface.co/google/gemma-4-26B-A4B-it | MoE 25.2B / 3.8B active, 128 experts top-8 plus 1 shared, 30 layers, same attention mix | 262,144 | Apache 2.0 | Yes | | Gemma 4 12B https://huggingface.co/google/gemma-4-12B-it | dense, 11.95B, 48 layers, same attention mix | 262,144 | Apache 2.0 | Yes | | Qwen3.8-27B https://huggingface.co/Qwen/Qwen3.8-27B and our fine-tunes | dense, 48 Gated DeltaNet + 16 attention layers, MTP head | 262,144 | Apache License 2.0 | Yes | All read on 20 October. Meta's repo is gated and our account isn't approved. So the Muse licence text comes from the gate prompt on the repo page and the config from the unsloth and mlx-community mirrors. The licence has no user-count threshold like Llama's, and the $20M clause applies only to offering the model to others as a service. Using it inside your own product is exempt. Most of our mistakes are already in the sections above: the unpinned route, the missing retry, the tool-call flag, the text-only serving, the output cap, the memory guard and the node modules folder. For Gemma 4 add goose's merged turns and identical resends, an edit tool that reported no-op edits as successes, a repeat notice pasted into a file listing, an older engine that re-read every prompt, a memory estimate about nine times too high, no cap on a local reply, and loop stops reported as output limits. We fixed the merge, the resend, the no-op edits, the re-read and the estimate in 3.0.119 on 19 October. Since 3.0.122 on 20 October a loop stop is reported as a loop, and our engine also stops exact repeats up to 1,024 tokens long. Two belong here because they cut across everything. Our engine ran the Xing, Granite and Bolt Forge runs and v2's 19 October run with thinking switched off on tool calls, because the app sent no reasoning setting. Our fine-tunes also ran with thinking off on 5 and 6 October. So did Gemma 4 on the Studio and the 128 GB MacBook Pro. Meanwhile Muse Spark Open got "medium" and the plain base kept its thinking. And I never ran v2 against the base on the same engine and the same goose build, so I can't fully separate the model from our serving. I should have caught both. The tool-call flag, the vision lane and the guard pricing are fixed, and so is the output cap in our fine-tune serve script. The app's engine still defaults to 32,768, and the route pin and the verdict retry aren't fixed yet. The node modules folder still sits in the run directory, though since 3.0.119 goose's tree names it without listing it. In the Studio and 128 GB MacBook runs the app also sent no reasoning setting to the local engine. Run it through one long task in your own harness, the way I'd test any new model, then read the transcript before you look at the score. In particular, look for these: The quotes above come from the run's own transcript, which goose keeps as SQLite in a hidden runtime folder next to each build its name starts with a dot . I'd copy it first, so the live file stays untouched: sqlite3 "file:<runs/build /.<run -runtime/goose/data/sessions/sessions.db?mode=ro" ".backup /tmp/s-muse-b1.db" sqlite3 "file:/tmp/s-muse-b1.db?immutable=1" \ "SELECT datetime created timestamp,'unixepoch','localtime' , substr json extract value,'$.thinking' ,1,110 FROM messages, json each messages.content json WHERE json extract value,'$.type' ='thinking' AND json extract value,'$.thinking' LIKE '%limited calls%';" 2026-10-19 19:58:35|Need to continue implementing other modules quickly but we have limited calls. We need to prioritize deliverin And to see which reasoning effort a run really sent, read the first logged request: head -1 "<runs/build /.<run -runtime/goose/state/logs/llm request.0.jsonl" | grep -o '"effort":" a-z ' "effort":"medium Deduplicate tool requests by their call id before you count them: compaction re-inserts old history with the same ids. Scores and bills for all published runs are on the benchmark board https://leanzero.net/agentic-benchmarks?utm source=devto&utm medium=referral&utm campaign=crosspost , and the methodology page https://leanzero.net/agentic-benchmarks/methodology?utm source=devto&utm medium=referral&utm campaign=crosspost explains how a run gets a verdict. For the format side of local serving, our MLX vs GGUF comparison https://leanzero.net/blog/mlx-vs-gguf-apple-silicon?utm source=devto&utm medium=referral&utm campaign=crosspost covers the same Macs. Goose Swarm, the app that ran all of this, isn't production ready yet, and you've just read a list of the reasons. If you run an open model as an agent, what broke first for you, the model or the plumbing around it? Tell me in the comments under this post, especially if you got Muse Spark Open through a long write on a route other than Meta's. In our case it was a bit of both, and more of the plumbing than I'd like. Originally published on leanzero.net https://leanzero.net/blog/muse-spark-open-benchmark-open-weight-models?utm source=devto&utm medium=referral&utm campaign=crosspost . More Atlassian, Forge and local-AI write-ups at leanzero.net/blog https://leanzero.net/blog?utm source=devto&utm medium=referral&utm campaign=crosspost , and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services https://leanzero.net/services?utm source=devto&utm medium=referral&utm campaign=crosspost .