Two open-weight models, one 20 GB GPU, two real development tasks, and a third model as the referee. Here is what actually happened when I made Qwopus 3.6 27B and Meta's Muse Glimmer 30B implement a bug fix and then a full feature in my own project.
Both models ran fully local on an AMD Radeon RX 7900 XT (20 GB VRAM) via a llama.cpp multi-model router (one OpenAI-compatible endpoint, GGUF models, load-mode=dio
— more on why below). Each model was driven by the pi CLI in non-interactive mode with --thinking high
. A third model — Codex, through a disciplined stdin wrapper — reviewed both outputs and gave the verdict.
The fairness method was simple but strict:
The test project: Jeu de Cochons (a "Pass the Pigs" dice game, vanilla JS PWA on Vite + Vitest) — real code, real tests, no toy repo.
| Qwopus 3.6 27B | Muse Glimmer 30B | |
|---|---|---|
| Source | Community fine-tune of Qwen 3.6 | Meta (distilled from Muse Spark) |
| Size | 27B | 29.6B |
| Quant | IQ4_XS (~15 GB) | UD-Q4_K_XL (~14.8 GB) |
The project had a broken PWA: a commit that added a /jeu-de-cochons/
base path for GitHub Pages had broken 3 service-worker tests (manifest, precache, offline navigation fallback). Task: fix the regression without touching the tests, keep the other 84 green.
| Qwopus | Muse | |
|---|---|---|
| PWA tests (11) | 11/11 ✅ | 11/11 ✅ |
| Full suite (87) | 87/87 ✅ | 87/87 ✅ |
| Files touched | 2 | 2 |
| Diff size | +4/−4 | +4/−4 |
| Wall time | ~8.5 min | ~21 min |
| Leftover artifacts | none | one .bak file |
The remarkable result: both models produced a byte-identical diff. Same diagnosis (a lost capture group in the asset-discovery regex that crashed precache), same manifest fix, same minimal solution. Codex called it a perfect tie on code — but rejected both on a latent conflict neither model noticed: reverting the manifest to /
breaks the /jeu-de-cochons/
production deployment (404 icons). The tests hardcode the root path; the deployment lives under a subpath. That contradiction was visible in the task itself, and only the reviewer caught it.
Lesson #1: on a short, well-bounded task, the two models are interchangeable. Muse thought 2.5× longer and produced the same bytes. You need a richer task to separate them.
Add a single-player mode against an AI opponent: menu selection, bot identity in game state, a thoughtful decision strategy (a pure function choosing to re-roll or bank based on turn score and game situation), automatic bot turns with human controls disabled, unit + integration tests, no regressions, build passing.
| Qwopus | Muse | |
|---|---|---|
| Existing tests | 84 ✅ | 84 ✅ |
| New tests written | 18 (12 unit + 6 integration) | 7 |
| Total green | 102/105 | 91/94 |
| Build | ✅ | ✅ |
| Wall time | ~32 min | ~26 min |
| Leftover artifacts | none | uncommitted dist/ |
The AI strategies they designed:
Qwopus — adaptive threshold:
isBot
flag on the playerMuse — fixed threshold:
Codex verdict:
| Criterion | Qwopus | Muse |
|---|---|---|
AI strategy (primary) |
8/10 | 5/10 | | Architecture | 8/10 | 6/10 | | Robustness | 7/10 | 5/10 | | Minimalism | 6/10 | 7/10 | | Tests | 8/10 | 4/10 | | Verdict | CONDITIONAL | CONDITIONAL |
Winner: Qwopus.
What actually separated them:
isBot
from the localized name — meaning a human player named "Ordinateur" in multiplayer would silently become an AI. A real bug, confirmed in the code.not.toBeNull()
), with a comment documenting the contradiction.Qwopus's own flaws, to be fair: its solo menu requires filling two human-name fields before swapping the second for the bot (blocking if you only fill one), the "~78%" claim had no math behind it, and its bot timer isn't cleared on menu exit (guarded, but not cleaned up).
| Round 1 (short fix) | Round 2 (feature) | |
|---|---|---|
| Winner | Tie | Qwopus | | Deciding factor | none (identical diffs) | strategy + tests + robustness | | Time | Qwopus 2.5× faster | Qwopus slightly slower (but more code) |
`--load-mode dio`
(direct I/O), which both avoids the bug and loads 3–8× faster.`pi -p --provider llama-router --model <model> --thinking high "<task>"`
)Honesty clauses: this is a single run per model per task, not a statistical benchmark. Both tasks came from one small project, and the Codex review is one reviewer's judgment. The numbers are real and were verified against the artifacts, but treat them as a data point — not a ranking of these models in general.