The stock Qwen3.8-27B is still the best local 27B we've measured on this benchmark. I'm a bit annoyed about it. Over two days we ran four popular Qwen3.8-27B fine-tunes and merges through Forge 2.0, our long agentic coding benchmark, on one 64 GB M3 Max MacBook, next to the stock base model. The base's median is 0.1852. The closest candidate, nightmedia's MindMeld-OneJev1.6, scored 0.1105 in its one build, which is level with the base's weaker build (0.1128), so I'd call that one close. Two of the four never got a score. I cancelled both.
If you run a local model for agentic coding, you probably picked it the way we did: by downloads, by the LiveCodeBench number on the card, by who made it. The fine-tunes arrive faster than anyone can test them, and people run them anyway, in big numbers: DavidAU's GGUF of one of the four showed 1,762,401 downloads on 18 October. Three of our four cards claim better scores than the base. On this task the cards told us little. In our setup one of the four had its write tool calls come back malformed. Another wrote careful code and then got stuck hunting for a method that doesn't exist. A third finished first, with placeholders where the grader looks. And the merge got the hardest parts right and still lost its admin page. So what is a card worth for this kind of work?
What follows is one task and one build per candidate, on our own 8-bit conversions, our MTP heads and a goose build that isn't released yet. It says what each model did on a 300-call Jira app upgrade on a laptop. It doesn't say any of them is a bad model. Read it as that.
Forge 2.0 hands goose, the open-source coding agent we build on, a working Jira app and a long written spec. Goose has to upgrade the app to v2 with no help. A scorer then runs 105 checks against three seeded Jira sites. The Forge 2.0 board has the cloud models; the base pair sits on it as one local entry.
| model (Hugging Face repo) | what it is, per its card | builds | score | how it ended |
|---|---|---|---|---|
| Qwen/Qwen3.8-27B , stock | the base | 2 | 0.2576, 0.1128 | finished both, 300 and 285 calls |
| nightmedia/Qwen3.8-27B-MindMeld-OneJev1.6 | mergekit nuslerp merge | 1 | 0.1105 | used its 300-call budget in 1 h 50 m |
| DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU | multi-stage tune, then Heretic | 1 | 0.0311 | stopped itself at call 162 |
| ukisai/Swift-1.5-Qwen3.8-27b | RL and OPD post-train | 1 | none | we cancelled it at call 263 of 300 |
| nerkyor/Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-MTP-DFlash2 | SFT and RLOO coding post-train | 1 | none | we cancelled it at call 165 |
I picked all four by hand. I'll call them Coder390, Swift, DAU735 and MindMeld from here on. Hand the scorer the v1 starter app with no model work at all and it gives 0.0617, and one of the four finished under that.
A Forge 2.0 score is three numbers multiplied. The first is the work: how much of the product passes, before penalties (the base's builds earned 0.7079 and 0.7713 of it). The second is a critical multiplier, 0.6 for each class of serious failure such as a data leak or a forged request that changes something. The third is a reliability multiplier: in each of 19 groups of checks, the worst check that falls short costs up to 10%, and the multiplier never drops below 0.25. The base's 0.2576 is 67 of 105 checks fully passed; its 0.1128 is 69 of 105 with two criticals.
We had two days, one Mac and 1 to 9 hours per build. So I set a rule: a candidate whose first build scored below the base's 0.1852 median got no second build. DAU735 and MindMeld fell under it. The base's own build 2 would have fallen under it too, which tells you how rough the rule is. We cancelled Coder390 and Swift before either finished a first build.
The base's two builds have run pages: build 1 at 0.2576 and build 2 at 0.1128. The four candidates aren't on the board, because single or cancelled builds aren't published, so their numbers come from our run ledger, their session logs and the verdict files.
render at 14:34 and went on grepping for attach eleven more times.
I kept the machine, the task and the settings the same for all five. The machine is a team MacBook Pro with an M3 Max and 64 GB of unified memory, with one model loaded at a time and no other apps running, and each build had a 300-call budget at reasoning effort medium with the same registered task payload the cloud baselines get. The agent was a goose test build with Compaction V3 (branch t12/compaction-v3, based on 3.0.111, not in a release yet), and the engine was LeanZero's MLX engine, our fork of Rapid-MLX, text-only.
We converted all four candidates the same way, from the publisher's full BF16 weights to MLX 8-bit with group size 64 (mx.quantize). Each ran with its own MTP weights, converted into our engine's format. MTP, multi-token prediction, is a small extra head trained to guess the next few tokens; the main model then checks the guesses in one pass. The method comes from Gloeckle et al. and DeepSeek-V3's MTP modules, and the checking step is speculative decoding, which Leviathan et al. describe as sampling "without any changes to the outputs". In the pre-launch smoke tests, MTP took Coder390 from 12.3 to 19.9 tokens per second, and Swift, DAU735 and MindMeld accepted about 2.6 to 2.8 tokens per verify step. Coder390's converted 4-bit MTP head was refused by our engine ("7 tensor(s) whose shape disagrees"), so we rebuilt it by hand from the repo's own BF16 MTP weights.
From Swift on, a smoke test made three write tool calls on the live engine before each launch; Swift, DAU735 and MindMeld passed 3 of 3. Coder390 ran before we had that test. While each build ran, a 10-minute check-up loop read the tool calls, diffed the model's changes against the starter and looked for broken imports. It grew during the day: the malformed-call check came in the middle of Coder390's run, after we spotted its broken writes by hand, and the repeat check during Swift's. The kill rule was mine: a run that loops or stops making progress gets cancelled. How we got the base to run at all on this Mac is in the compaction write-up and the MLX MTP speed test.
The benchmark's admin page has to refuse anyone who isn't a Jira administrator, checked on the server. So the natural call is GET /rest/api/3/mypermissions?permissions=ADMINISTER. Its schema says the answer is a map keyed by permission, and the flag is called havePermission. Four models wrote four readings.
Swift, in src/admin.js:
const has = (res?.permissions ?? []).some((p) => p.permission === 'ADMINISTER' && p.hasPermission);
That treats the map as a list. On the real answer .some isn't a function, so the try catches the error and the check returns false, for real admins too. That's my reading of the code. Swift's build was never scored. Its reasoning before it wrote the line: "Also, I need to check the shape of the mypermissions response: {permissions: [{permission: 'ADMINISTER', hasPermission: true}]}." It never did. Its OpenAPI scripts (the first died on a missing bracket, the rewrites ran) printed each endpoint's parameters and request body, and none of its later calls read a response schema.
Coder390 got the map right and the field wrong, if (!res.permissions?.ADMINISTER?.hasPermission), with the same result by my reading. DAU735 wrote six admin resolvers with no check at all. MindMeld got it right:
return res.permissions?.ADMINISTER?.havePermission === true;
None of the four candidates printed the schema first, so MindMeld remembered it right and the other two remembered it wrong. My guess at why the wrong memory is hasPermission: Confluence's permission check uses that name, and so does plenty of example code. The transcripts can't show that.
But the base did better in a way I like more than being right. Build 1 also believed the wrong format. It said so in a comment: "Real Jira answers an array of { permission, hasPermission }; some mirrors answer the OpenAPI map of key -> { havePermission }. Accept both shapes." Then it accepted both, and the scorer gave it 1.0 on both admin checks (admin actions apply, non-admin refused). That comment is the most useful line any of the five wrote.
Our spec asks for the admin panel in UI Kit, Forge's own component system, where your code never touches the DOM. You build the component tree with @forge/react and hand it to the platform with ForgeReconciler.render(<App />); the UI Kit overview shows exactly that call. In @forge/react 12.3.0, the version in the benchmark's starter, ForgeReconciler has two methods, render and addConfig.
Swift wrote this at 12:21 EEST, two hours into its run:
const container = document.getElementById('root');
const root = createRoot(container);
root.render(<App context={context} />);
ForgeReconciler.attach(root, container);
It's a Custom UI bootstrap (React in an iframe) with an invented glue call on the end. Its reasoning at the time: "Standard Forge UI Kit page template (from Forge docs): ... ForgeReconciler.attach(root, document.getElementById('root'))". No such template exists, and the method isn't in the package. It made that up.
At 14:06 it came very close: "The UI Kit host has no document — native pages must just export the component. Let me check the reconciler's API:". Then it grepped for attach. It ran 29 of those greps by 14:37, with a 13-minute compaction in the middle, and they matched one thing, attachFunctionPropId, an internal helper. One of them was cat node_modules/@forge/react/out/reconciler.d.ts | head -30 on a 56-line file that declares render on line 53. At 14:34 it printed the ForgeReconciler object itself from reconciler.js, render and addConfig and no attach, and then grepped for attach eleven more times. I find that part painful to read. No source file changed after 14:03. We cancelled it at 14:38 under the kill rule, and our exact-repeat check said 0 of 20 repeats the whole time because each grep was worded a little differently.
MindMeld got there by reading. Its reasoning: "the reconciler has a render method that takes an element ... So the bundle should: 1. Import the component 2. Call ForgeReconciler.render(<Component/>)". It rebuilt a 10-line title stub into a 168-line page that, in its code, names all eight required controls, ending in ForgeReconciler.render(<AdminPanel />);. The scorer still recorded "admin-page: never reached its first paint". Its UI Kit host held an empty tree. Even the ... line the page shows first never appeared. I see two unusual things in its code. It calls the bridge by hand (self.__bridge.callBridge('invoke', ...)), after deciding that "For UI Kit, the bridge is provided differently". And its backend is a plain function, where the base used a @forge/resolver definition. I don't think either explains an empty first render. I didn't trace it further.
Both base builds ended up with a plain page. They imported invoke from @forge/bridge, built resolvers with new Resolver() and called the default export's render. Build 1 named the import Forge and called Forge.render(<Admin />); build 2 named it render. Build 2 spent more than 20 calls researching how the UI Kit host runs the admin page, the same question Swift got stuck on, and came out with a working page. Both scored 1.0 on the admin boot check, one invoke before first paint.
That empty page cost MindMeld more than one check. With no admin page there was never a CI secret, so validly signed webhooks got 401 too. The scorer marked the replay and unsigned-request checks as vacuous: nothing was exercised, so nothing was earned.
The benchmark's CI webhook checks a signature (HMAC-SHA256, timing-safe), then a timestamp within 300 seconds, then applies each event id at most once, ever. Two copies arriving at once make the last one hard.
Two of them wrote it the safe way. Swift and base build 1 both claimed the event id atomically. Swift:
await kvs.entity(EVENT_KEY).set(eventId, { at: Date.now(), environment: parsed.environment ?? '' }, { keyPolicy: 'FAIL_IF_EXISTS' });
With that policy the second writer fails and one copy wins. Base build 1 did the same with a plain key, and the scorer's pair of simultaneous copies got "one acceptance and no duplicate Jira/deployment effect". Swift's version reads as if it would pass the same test, but it was never scored. Coder390 wrote the same claim in a helper, and never wired a handler to it.
Base build 2 checked and then set:
if (await kvs.get(seenKey)) return { outputKey: 'duplicate' };
// ...
await kvs.set(seenKey, { at: new Date().toISOString() });
Both copies got through. Each read "not seen" before either wrote. The scorer: "both invocations of one concurrent eventId wrote scope-ledger data", a critical failure. Its wrapper in index.js also saved a rate-budget counter on every request, signed or not, which is a second critical (an unauthenticated request writing storage). Two criticals put a 0.36 multiplier on a build with the best work of any local run, 0.7713 before penalties. Without them it would have scored about 0.31, and with only the race about 0.19.
DAU735's first version of the signature check ended with return 'placeholder'; // Simplified for demo. It replaced it with a real Web Crypto HMAC that it called without await, so for a while every signature was compared against the string "sha256=[object Promise]" and every webhook would have been refused, and then it fixed that too. Its replay list keeps "the last 100" like this:
processed.push(eventId);
// Keep last 100
if (processed.length > 100) processed.length = 100;
Once 100 ids are stored, that truncation drops the id it just pushed, so replays after the hundredth event would be accepted. That comes from reading the code; the scorer never got that far, for MindMeld's reason: no admin page, no secret, every event 401.
My guess at why the race keeps showing up: a model testing in a terminal sends one request at a time, and check-then-set passes all the tests you can run that way. I'd test for it on purpose next time.
DAU735 was the fastest model here, and its code shows how. It made its first write at call 27 and declared itself done at call 162, after 1 h 18 m. Recording a deployment only logs a line ("In a real implementation, would look up issue id and update ledger rows"). The comment-group rule is an empty block marked "Simplified for now". Its closing summary says "All admin resolvers check ADMINISTER permission", and none of them do. And its AI explanation cache does this:
const cached = await getFromCache(view.sprint.id, 'viewer');
// ...
await putInCache(view.sprint.id, 'viewer', result);
Its key is the literal string 'viewer', so all viewers of a sprint share one cached explanation, and whatever the first person was allowed to see becomes what the next person is shown. The scorer caught it as a critical leak: "hidden issue data shown to a person" through the "LLM display cache", issues that person can't browse. Its migration corrupted the v1 data at 371 checkpoint rows, a second critical. Score: 0.0311, from 0.3306 of work, a 0.36 critical multiplier and a 0.2611 reliability multiplier.
Base build 1 keyed its cache on the sprint, the viewer's account id and what that viewer can see, and build 2 on the sprint and the viewer's visible content. Both scored 1.0 on the same leak check. MindMeld left stubs too. Deployments and deleted issues are TODO s, the custom-field write posts a body Jira rejects, and "not in" searches return the set instead of its complement. But nothing it shipped leaked, and it got no critical at all. It sits at 0.1105 with a 1.0 critical multiplier.
Goose compacts the conversation when the context gets near its limit, which means it asks the same model to summarise everything so far, throws the old history away and keeps working from that summary plus the most recent turns. On this Mac that took 6 to 20 minutes each time, because the compaction is a full read of the history and a long write on the same 27B. (Token counts below are our watcher's; goose's own notice reads a little differently.)
Swift compacted three times, at about 12:05, 13:17 and 14:22, from 123,686, 116,028 and 114,188 tokens. Between the second and third it ran one identical diagnostic (reset, then eight scheduled reconcile runs) eight times in nine calls with no edit in between. Same output each time. The summary written at the 13:17 compaction lists, among API facts it had checked, "@forge/react page bootstrap: createRoot(container).render(<App/>) then ForgeReconciler.attach(root, container)". About 50 minutes later the attach hunt began. That summary line is evidence that the invention was carried forward as a fact. My guess at the rest: once it read as a fact, the model went looking for an API to match its code instead of doubting the code.
Base build 1 compacted four times, and after the first one each next call was a write. In build 2, three of its five compactions were followed by a write within two calls; after one it read the UI Kit host's source for 19 calls before writing again. MindMeld compacted once, from 119,177 to 45,654 tokens in about 7 minutes, and kept editing until its budget ran out, while DAU735 compacted at 98,188 tokens and wrote the completion summary that declared it done just 25 calls later.
The base wandered sometimes. It never looped, and I'm keeping it on this Mac for that. Its two builds took 6 h 40 m and 8 h 41 m, the longest here.
Coder390 isn't in the moments above. It never got far enough. It read docs, OpenAPI files and the starter's test kit for 135 calls before its first file change (the base's builds first changed an app file at calls 43 and 34). I don't think the reading was wasted. Its manifest mapped the whole spec correctly, with every new trigger, schedule, web trigger and the custom field. Its quota module had the rate rules right. It just couldn't land the files. I wish it had. 6 of its 7 write calls came back like this:
write: {"path": "manifest.yml\nparameter>\n<parameter=content>\nmodules:\n trigger: ...
Qwen3.8's chat template asks for tool calls in an XML form, <parameter=path>, the value, then </parameter>. Here the closing tag lost its </. The path swallowed the next parameter. The write failed. Coder390 fell back to shell heredocs, and one malformed write left a junk folder in src/. At call 165, after 1 h 40 m, index.js imported 14 names from files that didn't exist and src/admin.js didn't parse, and I cancelled it.
Its chat template is byte-identical to the base's, so the template didn't cause it. We don't know if the lost </ comes from the fine-tune or from the MTP head we rebuilt. MTP verification is supposed to be lossless, so a bad head should only cost speed, unless our verify path is wrong too. I'd still suspect a head built by hand first. We deleted the weights before re-running it without MTP. Its session log also stored no reasoning at all, where every other build stored plenty, which is a second unexplained difference, either in the model or in how our stack parsed it. Its card's own claims are about stopping on time and about GPQA, MMLU and LiveCodeBench, none of which this task measures.
On this one task, in two builds: yes, for drafting. It builds most of the product and then a couple of mistakes take most of the score away. Its two builds earned 0.7079 and 0.7713 of work before penalties, against 0.4234 for MindMeld and 0.3306 for DAU735, the two candidates with scores. On the cloud side of the same board the top score is 0.8523, and on this Mac the base's better build took 6 h 40 m to get its 0.2576, which is a long time to wait for a draft you then have to fix by hand. So I'd use it on a Mac to draft an app like this, and I wouldn't let it ship one unattended.
The base, per its card: a dense 27B with 64 layers in a "16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))" layout. Three of every four layers are Gated DeltaNet linear attention, which carries a fixed-size state instead of a growing cache, and one in four is full attention. Its native context is 262,144 tokens. It ships an MTP head "trained with multiple steps" and thinks by default.
Coder390, per its card: "post-trained with several rounds of SFT and RLOO" on top of nerkyor's EfficientThink SFT + SimPO model, "built to fix the original model's habit of failing to stop". The names in the repo id are explained in its table: "Opus5.5, GPT6Astra | Wrote the gold answers and teacher trajectories", "DSV4Pro, K3 | Their trajectories served as teacher-model data". RLOO is a REINFORCE-style RL method and SimPO a preference method. Its card claims 90 of 100 LiveCodeBench problems against 83 for the original. Hypothesis, labelled as one: the card's post-training data are reasoning and coding answers; if little of it exercised Qwen's XML tool-call format, that format could drift. The card doesn't say either way, and we measured nothing that shows it.
Swift 1.5, per its card: UkisAI scaled up "the post-training (RL and OPD)" from Swift 1.0, "this time with the main focus on long-horizon, agentic, and coding tasks". It "uses 58.5% fewer thinking tokens". It reports 81.71% against 76.76% on LiveCodeBench v6 and 72.13% against 69.21% on Terminal-Bench 2.1. Its effort table also shows medium, our setting, at 82.22% for Swift against 84.14% for the base on GPQA-Diamond, with 24.8% fewer tokens. Swift thought the most of the four candidates here (about 295,000 characters), still well under the base's 500,000 and 754,000, which fits its card. One build can't say whether that cost it anything.
DAU735, per its card: a multi-stage build from "a number of my finetunes ... as well as some third parties", with thanks to nightmedia "for working on the first three stages prior to heretic'ing", then Heretic ("KLD: 0.0025"). Heretic is automatic "censorship removal" by directional ablation, after Arditi et al.. The card promises "1/2 to 1/10" the thinking tokens and says the contributing fine-tunes "will be disclosed upon final release". Its GGUF repo showed 1,762,401 downloads on 18 October; we tested the full-weight source. Its session log kept about 28,000 characters of reasoning against 500,000 and 754,000 for the base's builds, which fits the card's promise. Hypothesis: much shorter reasoning fits a model that finishes fast and writes "Simplified for demo". MindMeld also thought little and also left stubs, without the leak, so one build each can't separate reasoning length from the rest of each recipe.
MindMeld-OneJev1.6: the card says it's a merge of nightmedia/Qwen3.8-27B-MindMeld and OmniJev/OneJev1.6-27B, and lists 18 participating models upstream. The repo's mergekit config uses nuslerp with weight 1.6 on a model it names Qwen3.8-27B-Seven-Brainwaves and 0.4 on OneJev1.6. Per mergekit's docs that's t = 0.4 / 2.0 = 0.2, a spherical interpolation one fifth of the way from the first model toward OneJev. I think the method matters here. Hypothesis: a merge moves a model less than a fresh RL post-train does, which would fit its 0 malformed calls and no loops. Also unproven. (nightmedia is a friend of the company, so we had gated access the day it came out.) The card has changed since. Later on 18 October (17:02 UTC) nightmedia added "This model is not for coding." to it; the earlier version we quote here said nothing like that, and both versions tag it coding.
So what do the cards tell you? What each author optimised: stopping on time, fewer thinking tokens with a stated focus on agentic and coding work, uncensoring, a blend. None of them measures a 300-call build against a written spec on a laptop, and on this task the base's habits mattered most.
Our first audit said AUDIT PASS on Coder390. It was failing. It counted 13 starter files the model never changed as the model's work, missed the malformed tool calls, and double-counted calls because compaction re-inserts the kept history (204 rows for 142 calls), and since then it diffs against the starter's file hashes, counts each tool request once and flags tool-syntax tags inside the arguments of any call.
Our repeat check said 0 of 20 during Swift's 29 greps, because it only matched identical calls. The same exact-match count then flagged MindMeld's normal edit, lint, re-test cycle as 5 of 20 repeats. So it was wrong both ways. A repeat now only counts when nothing was edited in between.
Base build 2 first came out at 0.2736 and "unpublishable", because our own scoring probe crashed with a stack overflow. With the probe fixed it re-scored at 0.1128, two real security criticals included, and 0.1128 is what the board shows.
Our older goose build gave the same base 0.1342 and 0.1457. The 0.1342 carries a critical our probe raised by mistake, and it's at least 0.2237 without it. We took that series off the board when we published one entry per setup. I don't compare against it here.
Coder390 ran on an MTP head we rebuilt and before our tool-call smoke test existed, and all five ran on our own 8-bit conversions. None of these models ran in the stack its author recommends.
I'd run these six next time, in this order.
</parameter> or tags inside the arguments. Our Coder390 setup would very likely have failed this in a minute.mypermissions map, and read what it writes. None of our four candidates looked the answer up first.placeholder, Simplified, In a real implementation and TODO at the paths you care about, and compare its final summary with the code.
If you'd rather train your own than trust a card, our Qwen3.8-27B LoRA tutorial on a Mac is where we started; our own fine-tune isn't in this comparison. Next on this Mac is a 30B model from another family, because after five 27Bs in two days I'm sick of 27B variants.
If you run a community fine-tune for agentic coding on your Mac, what made you trust it, and has it ever broken its own tool-call format on you? With one build each I might be wrong about MindMeld, and I'd like to know if it does better for you. Answer in the comments under this post. The agent we used is goose-local-edition; the Compaction V3 build isn't released yet and it's rough in places, which is partly why we test it this hard.
Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.