{"slug": "a-7-5b-model-beat-a-24b-on-my-coding-benchmark", "title": "A 7.5B model beat a 24B on my coding benchmark.", "summary": "A developer built a 56-task coding benchmark with hidden tests and ran 16 model configurations on the same hardware, finding that a 7.5B-parameter model (gemma-4-e4b) scored 42/56, beating a 24B model (devstral-small-2-24b) at 40/56, and that the ranking does not match public leaderboards. The benchmark uses deterministic verification (cargo test, pytest, node) instead of LLM judges, and the developer found that error bars are a property of the model, not the benchmark, with one model swinging 8 points between identical runs.", "body_md": "Every number below was recounted from the committed `SCORES-*.tsv`\n\nfiles, not transcribed from notes. The raw data - 36 rows, one per run, each with its full failure list - is [ RESULTS-q56.csv](https://github.com/mrdushidush/claudette/blob/battery/q50-quality-corpus/runs/eval-2026-05-29/battery/RESULTS-q56.csv).\n\n`gemma-4-e4b`\n\nis **7.5B parameters in a 4.97 GiB file**. It scored **42/56**.\n\n`devstral-small-2-24b`\n\nis 24B in 11.90 GiB. It scored **40/56**. `gpt-oss-20b`\n\nscored 38. Both\n\ngemma-4-12b variants - one of them at Q8_0, more than twice the precision - landed at 42.5 and 41.\n\nThree runs each. The ranges do not overlap. **On a small card, the 5 GiB model is not the\ncompromise option**, and no public leaderboard would have told you that.\n\nEveryone has opinions about which local model to code with, which quantisation to run, and what\n\nKV cache setting costs you. Almost all of those opinions come from a public leaderboard, or from\n\none run of one config on somebody's box.\n\nSo I built a 56-task coding benchmark that grades with hidden tests, and ran **16 model\nconfigurations** through it on the same card at the same constants, with every held-constant\n\nThe ranking does not match any public leaderboard; in one case it inverts one outright. And the\n\nnoise result surprised me more than the ranking did: **the error bar is a property of the model,\nnot of the benchmark.** One model swings 8 points between identical consecutive runs while another\n\n56 medium-hard coding tasks run through a real agent loop - the model reads files, edits them,\n\nruns tests, and iterates until it says it is done. Grading is **hidden verification**:\n\nThat last point matters more than it sounds. The lesson this project inherited from its own\n\nancestors: **LLM judges inflate.** Deterministic ground truth may be *lowered* by a model, never\n\nraised. So there is no judge here at all - just `cargo test`\n\n, `pytest`\n\n, and `node`\n\n.\n\nTasks are **test-first gated**. A task enters the corpus only if its verifier *fails* on the\n\nuntouched fixture and *passes* on fixture + reference solution. That rules out tasks that are\n\naccidentally already passing, and tasks that are impossible.\n\nThe discriminating axis turned out not to be algorithmic difficulty. It is **unstated correctness\ntraps** - the degenerate input, the falsy-vs-absent distinction, the boundary the prompt implies\n\n**Corpus:** 56 tasks - 15 python, 14 rust, 10 js, 9 ts, 8 shell. By kind: 18 implement-spec,\n\n7 boundary, 6 bugfix, 6 api-misuse, 5 error-handling, 4 refactor, 4 perf, 4 multi-file,\n\n2 concurrency.\n\n**Box:** RTX 5060 Ti 16 GB, Windows 11, LM Studio. Held constant on every row: ctx 32768,\n\nKV cache q8_0, `numParallelSessions`\n\n1, one agent session at a time, identical agent binary.\n\nEvery row's actual values are recorded rather than assumed - see \"the method lesson that cost the\n\nmost\", which is the section I would most want another benchmarker to read.\n\nFull 56 tasks on every row. Weak scores are data, not failures - a benchmark that reports only\n\nwinners tells you nothing about the shape of the field.\n\n| model | params | quant | GiB | runs | median | range | n | wall-clock |\n|---|---|---|---|---|---|---|---|---|\ngoogle/gemma-4-26b-a4b-qat |\n26B-A4B | Q4_0 | 13.45 | 55, 54, 55 | 55 |\n1 | 3 | 71–76 min |\n| unsloth/gemma-4-26B-A4B-it | 26B-A4B | UD-Q4_K_M | 15.78 | 54 | 54 | - | ⚠️1 | 71 min |\nqwen3.6-35b-a3b-mtp@iq3_s |\n35B-A3B | IQ3_S 3.06bpw | 12.67 | 49, 50, 50, 50, 48, 52 | 50 |\n4 | 6 | 21–31 min |\n| qwen3.6-35b-a3b@iq4_xs | 35B-A3B | UD-IQ4_XS 4.25bpw | 16.51 | 50 | 50 | - | ⚠️1 | 53 min |\n| qwen3.6-35b-a3b-mtp (GPU-3) | 35B-A3B | IQ4_XS 3.53bpw | 14.59 | 49 | 49 | - | ⚠️1 | 43 min |\n| qwen3.6-35b-a3b-mtp (GPU-4) | 35B-A3B | IQ4_XS 3.97bpw | 16.43 | 49 | 49 | - | ⚠️1 | 51 min |\n| qwen3-coder-30b-a3b-instruct | 30B-A3B | UD-Q4_K_XL | 16.45 | 43 | 43 | - | ⚠️1 | 58 min |\n| google/gemma-4-12b | 12B | Q8_0 | 11.80 | 42, 43 | 42.5 |\n1 | 2 | 44–53 min |\ngoogle/gemma-4-e4b |\n7.5B |\nQ4_K_M | 4.97 |\n43, 42, 42 | 42 |\n1 | 3 | 30–35 min |\n| north-mini-code-1.0 (Cohere) | 30B-A3B | UD-Q3_K_M | 13.24 | 42 | 42 | - | ⚠️1 | 82 min |\n| google/gemma-4-12b-qat | 12B | Q4_0 | 6.50 | 41 | 41 | - | ⚠️1 | 57 min |\n| devstral-small-2-24b-2512 | 24B | IQ4_XS | 11.90 | 40, 39, 40 | 40 |\n1 | 3 | 19–22 min |\n| openai/gpt-oss-20b | 20B-A3.6B | MXFP4 | 11.28 | 40, 38, 32 | 38 |\n★8\n|\n3 | 10–12 min |\n| qwen3.5-4b | 4B | UD-Q8_K_XL | 5.54 | 31, 33, 33 | 33 |\n2 | 3 | 33–42 min |\n| google/gemma-4-e2b | 4.6B | Q4_K_M | 3.19 | 31, 31, 31 | 31 |\n0 |\n3 | 23 min |\n| unsloth/granite-4.1-8b | 8B | Q8_0 | 8.70 | 25 | 25 | - | ⚠️1 | 22 min |\n\nSizes are the measured GGUF file in **GiB**, not a vendor marketing figure and not `lms ls`\n\n(which reports decimal GB and folds in the mmproj projector). Quant names are exact, because\n\n\"gemma-4-12b\" at Q8_0 and at Q4_0 are two different experiments.\n\n**On the rows marked ⚠️1:** a single run carries an error bar this benchmark can now put a number\n\non, and it is not small - see the noise section. Nothing load-bearing in this piece rests on an\n\nn=1 row. Where one does (QAT at 12B), I say so and decline to make the claim.\n\nOne model was excluded rather than scored: **nemotron-3-nano-omni-30b-a3b** took **299 seconds on\nthe corpus's easiest task** (vs 18 s for devstral) and projected to roughly 4.5 hours. At that\n\nThe single most repeated claim in local-model threads is some version of \"just run the biggest\n\nmodel that fits\". Here is what the gemma-4 family does, one architecture, one vendor, four sizes:\n\n| model | params | quant | GiB | median |\n|---|---|---|---|---|\n| gemma-4-e2b | 4.6B | Q4_K_M | 3.19 | 31 |\n| gemma-4-e4b | 7.5B | Q4_K_M | 4.97 | 42 |\n| gemma-4-12b | 12B | Q8_0 |\n11.80 | 42.5 |\n| gemma-4-26b-a4b-qat | 26B-A4B | Q4_0 | 13.45 | 55 |\n\n```\n4.6B → 7.5B    +11.0\n7.5B →  12B     +0.5      ← the plateau\n 12B →  26B    +12.5\n```\n\n**Scaling 7.5B → 12B buys half a point.** That is the flat stretch, and it is flat *despite the\n12B carrying the better quantisation* - Q8_0 against the 7.5B's Q4_K_M, which is the conservative\n\nBut it is bounded on both sides by real cliffs. Below it, dropping to 4.6B costs **11 points**.\n\nAbove it, the 26B MoE gains **12.5**. \"Size buys nothing\" is true in a band, and the band is\n\nnarrower than the slogan.\n\nWhich is where the title comes from. `gemma-4-e4b`\n\nclears `devstral-small-2-24b`\n\non **under a\nthird of its parameters and 42% of its file size** - and it does it across the whole family\n\n`gpt-oss-20b`\n\n(38), `north-mini-code`\n\n**And active parameters do not explain it either.** `gemma-4-26b-a4b`\n\nactivates about 4B\n\nparameters per token - roughly what the 4.6B `e2b`\n\nactivates - and scores **55 against 31**. Same\n\nactive compute, 24 points apart. Whatever is doing the work here, it is neither total size nor\n\nactive size on its own.\n\nNone of Result 1 is visible from a leaderboard, and that is not an accident - the ranking they\n\nproduce is different from mine at the top of the table too.\n\nA vendor comparison table puts **Gemma-4-31B roughly 21 points below Qwen3.6-35B** on SWE-bench\n\nVerified. On this corpus the gemma MoE beats the qwen champion **55 to 50**.\n\n**LiveCodeBench v6 anti-correlates outright**: gemma scores 77.1 there and 55/56 here; qwen scores\n\n80.4 there and 50/56 here. Higher on the board, lower on my tasks - in both directions, on the two\n\nmodels I care most about. And LiveCodeBench's own table for these models is **0-verified and\n53-self-reported**.\n\nIt is worse than \"different benchmarks measure different things\", because several of the obvious\n\nalternatives are simply not usable any more: **BigCodeBench has been frozen since 2025-04-14**\n\n(202 rows, zero 2026 models) and **Aider's polyglot benchmark went dead 2025-10-04**. Meanwhile\n\npublished work shows that **scaffold choice alone swings the same weights about 2×** on\n\nTerminal-Bench - 24.6% on the official board against 51.5% vendor-claimed for the same model.\n\nThat last number is the actual mechanism, and it is why I do not think my ranking is \"righter\"\n\nthan LiveCodeBench's. A leaderboard measures a model answering in a box. What you experience is a\n\nmodel driving *your* agent, with your tool schema, your context window, your file-edit format. The\n\nscaffold is not a detail wrapped around the model; on Terminal-Bench's own numbers it is worth as\n\nmuch as the model choice.\n\n**Use leaderboards to decide what to download. Never use them to predict what will happen inside\nyour own harness.** The ranking above took 34 full runs to discover and could not have been\n\nFour quantisations of one MoE family, same KV cache and context:\n\n| variant | bpw | GiB | score |\n|---|---|---|---|\n| MTP-GPU-2 | 3.06 |\n12.67 |\n50 (median of 6) |\n| MTP-GPU-3 | 3.53 | 14.59 | 49 |\n| MTP-GPU-4 | 3.97 | 16.43 | 49 |\n| UD-IQ4_XS | 4.25 | 16.51 | 50 |\n\n**Flat across the entire 3.06 → 4.25 bpw range**, while file size varies by nearly 4 GiB and\n\nend-to-end speed varies about 3×. The *lowest* bpw of the four is the best deal on this hardware,\n\nand it is not close - it is the only one that stays fully VRAM-resident.\n\n**Caveat, stated plainly:** this is one MoE family on one 16 GB card, and three of those four rows\n\nare single runs. It is not \"quantisation never matters\", and I would not assume it transfers to\n\ndense models, which are materially more quant-sensitive. The claim is narrower and better\n\nsupported: *within the band people actually argue about, on a card this size, for this family, the\nextra bits bought no measurable code quality and cost real speed.*\n\nQuantisation-aware training looks free, and I have two pairs that appear to show it:\n\n| pair | higher precision | QAT | gap |\n|---|---|---|---|\n| gemma-4-26b-a4b | UD-Q4_K_M 15.78 GiB → 54 (n=1) |\nQ4_0 13.45 GiB → 55 (n=3) |\nQ4_K_M vs Q4_0 |\n| gemma-4-12b |\nQ8_0 11.80 GiB → 42.5 (n=2) |\nQ4_0 6.50 GiB → 41 (n=1) |\n★Q8_0 vs Q4_0 |\n\n**These are not the same experiment, and collapsing them into one \"QAT is free\" line is wrong in\nboth directions.** At 26B it compares Q4_K_M against Q4_0 - both 4-bit-class, a near-like-for-like\n\nExcept the 12B QAT arm is **one run**, against a non-QAT arm whose own two runs span 42–43. One\n\npoint of difference, inside both models' measured spread. **So I am not claiming it.** The\n\nexperiment that would settle it is two more runs of `gemma-4-12b-qat`\n\n, and until those exist the\n\nhonest statement is: *QAT at 26B costs nothing measurable; QAT at 12B is untested.*\n\nI am flagging this at length because it is the shape of claim that gets repeated forever after one\n\nrun, and I nearly published it.\n\nThis is the half I did not expect to be interesting.\n\nFor a long time I carried a single global noise figure - \"±3\" - and applied it to every\n\ncomparison. **That was always wrong**, and it took replicating several models to see it.\n\nConsecutive runs, identical config, identical engine, same box, nothing changed between them:\n\n| model | runs | median | range |\n|---|---|---|---|\n| gemma-4-e2b | 31, 31, 31 | 31 | 0 |\n| gemma-4-e4b | 43, 42, 42 | 42 | 1 |\n| devstral-small-2-24b | 40, 39, 40 | 40 | 1 |\n| gemma-4-26b-a4b-qat | 55, 54, 55 | 55 | 1 |\n| qwen3.5-4b | 31, 33, 33 | 33 | 2 |\n| champion 3.06bpw | 49, 50, 50, 50, 48, 52 | 50 | 4 |\nopenai/gpt-oss-20b |\n40, 38, 32 |\n38 | ★8\n|\n\n**gpt-oss-20b swings 8 points across three identical runs.** Its published single-run number -\n\nthe 40 that would have gone in the table if I had run it once, as one normally does - is the\n\n**top** of its range, not its centre. Single-run reporting flattered it by two points and nothing\n\nin that run signalled it.\n\nMeanwhile `gemma-4-e2b`\n\nran three times and produced not just the same score but the **identical\nfailure list** - the same 25 tasks, the same error text. The only diffs between run 2 and run 3\n\nThe obvious explanation for gpt-oss's swing is \"it is borderline on lots of tasks, so it coin-flips\n\nthem\". **That explanation is dead**, and `e2b`\n\nis what killed it: e2b **fails more tasks than\ngpt-oss** - 25 against 16–24 - with\n\nWhat separates them is *how* they fail. gpt-oss loses tasks to **agent mechanics**: a bare\n\ntop-level `return`\n\nproducing `ERR_INVALID_TYPESCRIPT_SYNTAX`\n\n, `SyntaxError: Unexpected end of`\n\n, literal\n\ninput`\\n`\n\nescape sequences written into a `.mjs`\n\ninstead of real newlines. Its worst\n\nrun lost five tasks to *syntax*, not reasoning. Whether a generation comes out as a parseable\n\nfile is close to a coin flip. Missing an edge case, by contrast, is a stable property of what the\n\nmodel knows.\n\n⇒ **A model's error bar is predicted by its failure mode.** This was a hypothesis after the first\n\nreplication night and it then **held as a prediction** on e2b, at the opposite end of the quality\n\nrange.\n\n**The practical consequence: you cannot import someone else's error bar, and you cannot import\nmine.** If you are comparing two local models and the gap is under 3 points, you do not have a\n\nThis one is new, it is a benchmark-design finding, and I have not seen any local-model leaderboard\n\nreport it.\n\n`qwen3.5-4b`\n\nran three times at temperature 0: **31, 33, 33**. Ordinary-looking noise. Then I\n\nlooked at *which* tasks failed:\n\nAnd the confirmation: a fourth run had been started on r1's day and killed partway through. Over\n\nthe 33 tasks it graded, **it matches r1 exactly**. So day one has two agreeing runs, day two has\n\ntwo agreeing runs, and **the two sessions disagree with each other.**\n\n⇒ ★ **Every determinism claim in this campaign is scoped to a single session.** `e2b`\n\n's \"31/56\n\nthree times, range 0, identical failure list\" and `e4b`\n\n's byte-identical replicates were **all\ngathered with the replicates consecutive inside one session**. That design\n\nI do not know the mechanism yet. Candidates: the model server was restarted between sessions and\n\nsomething non-obvious differs in its state; a driver or engine load-order effect; machine load\n\n(see the next result, which accounts for 3 of the 10 flips but not the other 7). What I can say is\n\nthat **running your replicates back-to-back will understate your error bar**, and that is exactly\n\nhow everybody runs replicates, including me.\n\nIf you take one methodological thing from this piece, take this: **space your replicates across\nsessions, or state that you did not.**\n\nThe champion configuration, six runs at matched constants: **49, 50, 50, 50, 48, 52.** Median 50,\n\nrange 4.\n\nNow look at *which* tasks failed:\n\n| run | failures |\n|---|---|\n| 1 | Q03 Q05 Q25 Q45 Q50 Q51 Q52 |\n| 2 | Q03 Q05 Q13 Q25 Q51 Q52 |\n| 3 | Q03 Q05 Q13 Q25 Q51 Q52 |\n| 4 | Q03 Q05 Q13 Q26 Q36 Q51 |\n| 5 | Q01 Q03 Q05 Q13 Q25 Q46 Q49 Q51 |\n| 6 | Q03 Q25 Q46 Q52 |\n\n**Thirteen distinct tasks rotate through the failure slots. Exactly one - Q03 - failed all six\ntimes.** Frequencies: Q03 6/6, Q51 5/6, Q25 5/6, Q05 5/6, Q52 4/6, Q13 4/6, Q46 2/6, and six\n\n**So almost every \"this local model can't handle X\" claim you read is derived from a single run,\nand on this evidence that claim is unfounded roughly 12 times out of 13.**\n\nI know because I made exactly that mistake. Three runs in, I had confidently written down a\n\n\"consistent failure set\" of five tasks. The next run passed two of them and produced two\n\nfirst-time failures I had never seen. My five-task set was a three-sample artifact.\n\nIt cost me two wasted evaluations. I built a cheap screening gate on those five tasks; it\n\ngreen-lit two candidate configurations, both of which then tied the champion exactly on the full\n\nbattery. **The gate was measuring rotation, not quality.**\n\nWorse, when I rebuilt the screen properly - 12 tasks, scored in aggregate, with a rejection\n\nthreshold *derived* from a bound rather than picked - it still failed. The bound assumed a\n\nchallenger would pass everything outside the screen. **Three different models broke it**, one by\n\neight tasks. A subset screen now gets used only to reject fast, never to promote, bound, or crown.\n\n**If you benchmark local models: score in aggregate, over more than one run, and never conclude\nanything from the identity of a failing task.**\n\nA popular claim is that q8 KV cache costs quality against fp16. I tested it twice:\n\n| config | ctx | score | wall-clock |\n|---|---|---|---|\n| KV q8_0 | 32768 | 50/56 | 23.2 min |\n| KV f16 | 32768 | 50/56 | 26.2 min |\n| KV f16 | 65536 | 50/56 | 58.8 min |\n\n**+0 quality at either context size, and fp16 was slower in both** - 1.13× at matched context,\n\n**2.79×** once context went to 65536 and VRAM hit 96.7%.\n\nNote carefully what that 2.8× actually was. It was **not** KV precision - it was context size\n\npushing VRAM to the edge. I only know because the second fp16 run changed *one* variable; the\n\nfirst had moved two at once and would have supported the wrong conclusion, confidently.\n\nThe embarrassing part: I had been daily-driving that slow corner for two days, on the strength of\n\na promising subset-screen result that turned out to be the rotation from Result 6.\n\nThere is a matching cliff on the other side. Splitting MoE experts between GPU and CPU has a\n\n**usable band of about 15.1–15.6 GB on a 16.3 GB card**, and both errors are expensive. Too many\n\nexperts on CPU strands the card: one model at a bad split ran 113 s/task and at a good one 75\n\ns/task - **35 minutes saved on a single row, with no score change**. Too few thrashes into Windows\n\nshared memory: at 97.7% VRAM one task took **306 seconds**, and after re-splitting, **37 seconds.\n8.3×.**\n\nI have a standard probe: three prompts, streaming, temperature 0, median tok/s, against a 15 tok/s\n\nfloor.\n\nIt was wrong twice, in opposite directions. One configuration probed **faster** than the champion\n\n(73 vs 70 tok/s) and then ran **2.48× slower** on same-verdict tasks. Another probed 1.74× faster\n\nafter a tuning change and delivered **2.2×** end-to-end.\n\nThe reason is structural: an agent battery is dominated by **prompt processing over a growing KV\ncache**, not by token generation on short prompts. The probe measures the thing that is not the\n\n**Use generation-speed probes as a floor for rejecting hopeless candidates. Never cite one as\nevidence a configuration is fast.**\n\nFour of the 56 tasks grade performance by wall-clock - naive vs linear-time, sized for a wide\n\nseparation. Comparing `qwen3.5-4b`\n\n's cross-day replicates exposed what that really measures:\n\n| task | day 1 | day 2 | verdict |\n|---|---|---|---|\n| Q12 (perf, rust) | 90 s |\n24 s |\nFAIL → PASS |\n| Q32 (perf, js) | 51 s | 29 s | FAIL → PASS |\n| Q41 (perf, ts) | - | - | flipped the other way |\n\nIdentical weights, identical config, **3.75× on the same task**. Day 1's whole pass ran 27% slower\n\nbecause I was actively using the machine. **Those verdicts measured my box.**\n\nThis is a real defect in my corpus and I would rather report it than quietly widen the thresholds.\n\nAnyone grading agent performance by wall-clock inherits it. The fixes available are: run perf tasks\n\nonly on a verified-idle box, record a load probe per run and reject contaminated runs, or grade\n\ncomplexity structurally instead of by clock. I have not picked one yet.\n\n⚠️ It does **not** explain the session split in Result 5: **7 of the 10 flipped tasks are not perf\ntasks.** Machine load is one contaminant, not the mechanism.\n\nSeveral models lost tasks not to bad reasoning but to **not being able to emit a valid file**:\n\n`gpt-oss-20b`\n\nlost `return`\n\n, one writing literal `\\n`\n\nescape\nsequences into a `.mjs`\n\n.`north-mini-code-1.0`\n\nlost 2 the same way - an unterminated triple-quoted string, and a Rust\n`E0425`\n\n.`granite-4.1-8b`\n\nlikewise.`devstral`\n\nreached for a These are edit-mechanics and environment-awareness failures, not intelligence failures, and they\n\nare invisible in a bare pass rate. They are also the most actionable thing here for anyone building\n\nan agent: **the model that reasons well but cannot reliably write a file is worse in practice than\nits benchmark score suggests** - and, per Result 4, it is also the model whose score you can trust\n\nIt is the strongest argument I have that **agent benchmarks should grade mechanics separately from\nreasoning.** Mine currently does not, and that is a gap.\n\nThe spec for this benchmark had always listed \"held constant for every row: ctx 32768, KV q8_0,\n\nparallel 1\". Nothing ever **recorded** or **verified** it.\n\nThe KV cache setting in LM Studio is sticky, global, per-model, survives unloads, is not a load\n\nflag, and is not reported by `lms ps --json`\n\n. It silently flipped to fp16 on my box and stayed\n\nthere for two days. Rows from before and after were compared as though they were comparable.\n\n**A held-constant nobody measures is not held.** Every run now emits its actual ctx, KV type,\n\nparallelism, VRAM, GPU/CPU expert split, model path, agent version *and binary mtime* into a\n\nmetadata file, and historical rows are explicitly marked `inferred`\n\nrather than `measured`\n\nso a\n\nreconstruction can never be cited as evidence.\n\nEnforcement caught more than it had any right to. A pre-load config writer found:\n\n| model | field | found | should be |\n|---|---|---|---|\n| gpt-oss-20b | `numParallelSessions` |\n4 |\n1 |\n| qwen3-coder-30b | ctx | 8096 |\n32768 |\n| nemotron-3-nano-omni | ctx | 24000 |\n32768 |\n| qwen3.5-4b | ctx | 32800 |\n32768 |\n| qwen3.5-4b | KV cache | unset → f16 |\nq8_0 |\n\nThe `numParallelSessions: 4`\n\none is my favourite, because it splits the KV allocation across four\n\nslots behind an `lms ps`\n\nthat proudly reports `CONTEXT 32768`\n\n. Two older numbers I had been citing\n\nas priors turned out never to have been measured at the benchmark's own constants.\n\nThree ops traps each came within minutes of publishing a fake row:\n\n`lms load`\n\nnever creates the per-model config file.`lms load`\n\nfuzzy-matches when no key matches exactly.`qwen3.6-35b-a3b-mtp`\n\n- `@iq3_s`\n\n. Caught about\n40 seconds into a run that would otherwise have looked completely normal.The general lesson, and the reason this section exists at all:\n\nA benchmark's integrity failures do not look like errors. They look like successful runs.\n\nBetween the early rows and the late ones, LM Studio auto-updated its inference engine from 2.25.2\n\nto 2.27.1 at 01:21, with auto-update on by default. So I owed a control: re-run the champion,\n\nunchanged, on the new engine.\n\nThe first control scored **48/56** against a baseline whose runs had never gone below 49, with two\n\nnever-before-failed tasks and 1.33× slower wall-clock. Two signals pointing the same way. It would\n\nhave been very easy to write \"the engine update cost 2 points\".\n\n**The second control scored 52/56 - the highest the champion has ever scored - in 21m23s, its\nnormal speed.** Neither the deficit nor the slowdown reproduced. Across all six matched runs the\n\nI am reporting the sequence rather than just the conclusion, because the one-run version of this\n\nwas a publishable-looking finding that was simply wrong, and it is the same lesson as Result 6\n\narriving from a different direction. Your inference engine is still a variable: pin it, record it,\n\nand expect it to update itself while you sleep - mine did, and its auto-*delete* setting can\n\ngarbage-collect the old engine you would need to reproduce anything.\n\nThe corpus works because the hidden tests and reference solutions are hidden. Publishing them\n\ndestroys the instrument for every model trained afterwards. Publishing nothing makes every number\n\nabove an unverifiable claim.\n\nI had planned a split - ship 15 tasks complete, seal 41. **That plan is moot: the full corpus has\nbeen publicly cloneable since 2026-07-25**, on a branch of a public repository, hidden tests and\n\nContamination date: 2026-07-25.Every model in the table above was released before that date\n\nand therefore could not have trained on this corpus. All 36 runs stand. Any model released after\n\nit must be treated as potentially contaminated on these 56 tasks.\n\nSo: **everything ships** - all 56 fixtures, all hidden verifiers, all reference solutions, the\n\nharness, the per-run metadata, and `RESULTS-q56.csv`\n\nwith every run's full failure list. Clone it\n\nand check any number in this piece.\n\nThat makes Q56 a one-shot instrument against future models, which is a real cost and I am not going\n\nto dress it up. The successor corpus is being built sealed, and the design lesson is one I would\n\npass on to anyone building a benchmark they intend to keep: **decide your publication posture before\nthe first push, not after 36 runs.** A public repo is public in every branch.\n\nWhat survives publication completely is the part I actually care about: **the methodology and the\nmeasurement results.** That leaderboards anti-correlate, that error bars are model properties, that\n\nThe harness runs from any clone and takes a model id, so \"bench your own model\" is a real ask\n\nrather than an aspiration. The three most useful contributions, in order:\n\nThe corpus, the harness, all 36 runs and every run's failure list are at [github.com/mrdushidush/claudette](https://github.com/mrdushidush/claudette/tree/battery/q50-quality-corpus/runs/eval-2026-05-29/battery) - MIT OR Apache-2.0.", "url": "https://wpnews.pro/news/a-7-5b-model-beat-a-24b-on-my-coding-benchmark", "canonical_source": "https://dev.to/mrdushidush/a-75b-model-beat-a-24b-on-my-coding-benchmark-30o4", "published_at": "2026-08-01 13:33:57+00:00", "updated_at": "2026-08-01 14:11:58.353373+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "developer-tools", "ai-research"], "entities": ["gemma-4-e4b", "devstral-small-2-24b", "gpt-oss-20b", "gemma-4-12b", "LM Studio", "RTX 5060 Ti"], "alternates": {"html": "https://wpnews.pro/news/a-7-5b-model-beat-a-24b-on-my-coding-benchmark", "markdown": "https://wpnews.pro/news/a-7-5b-model-beat-a-24b-on-my-coding-benchmark.md", "text": "https://wpnews.pro/news/a-7-5b-model-beat-a-24b-on-my-coding-benchmark.txt", "jsonld": "https://wpnews.pro/news/a-7-5b-model-beat-a-24b-on-my-coding-benchmark.jsonld"}}