I Benchmarked Two Local LLMs on Real Dev Work — Qwopus 27B vs Muse Glimmer 30B A developer benchmarked two open-weight local LLMs, Qwopus 3.6 27B and Meta's Muse Glimmer 30B, on real development tasks in a JavaScript project. Both models produced byte-identical fixes for a PWA regression, but Qwopus outperformed Muse in a feature implementation, with a third model, Codex, serving as referee. The developer found that on short tasks the models are interchangeable, but richer tasks reveal differences in strategy and robustness. Two open-weight models, one 20 GB GPU, two real development tasks, and a third model as the referee. Here is what actually happened when I made Qwopus 3.6 27B and Meta's Muse Glimmer 30B implement a bug fix and then a full feature in my own project. Both models ran fully local on an AMD Radeon RX 7900 XT 20 GB VRAM via a llama.cpp multi-model router one OpenAI-compatible endpoint, GGUF models, load-mode=dio — more on why below . Each model was driven by the pi CLI in non-interactive mode with --thinking high . A third model — Codex, through a disciplined stdin wrapper — reviewed both outputs and gave the verdict. The fairness method was simple but strict: The test project: Jeu de Cochons a "Pass the Pigs" dice game, vanilla JS PWA on Vite + Vitest — real code, real tests, no toy repo. | Qwopus 3.6 27B | Muse Glimmer 30B | | |---|---|---| | Source | Community fine-tune of Qwen 3.6 | Meta distilled from Muse Spark | | Size | 27B | 29.6B | | Quant | IQ4 XS ~15 GB | UD-Q4 K XL ~14.8 GB | The project had a broken PWA: a commit that added a /jeu-de-cochons/ base path for GitHub Pages had broken 3 service-worker tests manifest, precache, offline navigation fallback . Task: fix the regression without touching the tests , keep the other 84 green. | Qwopus | Muse | | |---|---|---| | PWA tests 11 | 11/11 ✅ | 11/11 ✅ | | Full suite 87 | 87/87 ✅ | 87/87 ✅ | | Files touched | 2 | 2 | | Diff size | +4/−4 | +4/−4 | | Wall time | ~8.5 min | ~21 min | | Leftover artifacts | none | one .bak file | The remarkable result: both models produced a byte-identical diff. Same diagnosis a lost capture group in the asset-discovery regex that crashed precache , same manifest fix, same minimal solution. Codex called it a perfect tie on code — but rejected both on a latent conflict neither model noticed: reverting the manifest to / breaks the /jeu-de-cochons/ production deployment 404 icons . The tests hardcode the root path; the deployment lives under a subpath. That contradiction was visible in the task itself, and only the reviewer caught it. Lesson 1: on a short, well-bounded task, the two models are interchangeable. Muse thought 2.5× longer and produced the same bytes. You need a richer task to separate them. Add a single-player mode against an AI opponent : menu selection, bot identity in game state, a thoughtful decision strategy a pure function choosing to re-roll or bank based on turn score and game situation , automatic bot turns with human controls disabled, unit + integration tests, no regressions, build passing. | Qwopus | Muse | | |---|---|---| | Existing tests | 84 ✅ | 84 ✅ | | New tests written | 18 12 unit + 6 integration | 7 | | Total green | 102/105 | 91/94 | | Build | ✅ | ✅ | | Wall time | ~32 min | ~26 min | | Leftover artifacts | none | uncommitted dist/ | The AI strategies they designed: Qwopus — adaptive threshold: isBot flag on the player Muse — fixed threshold: Codex verdict: | Criterion | Qwopus | Muse | |---|---|---| AI strategy primary | 8/10 | 5/10 | | Architecture | 8/10 | 6/10 | | Robustness | 7/10 | 5/10 | | Minimalism | 6/10 | 7/10 | | Tests | 8/10 | 4/10 | | Verdict | CONDITIONAL | CONDITIONAL | Winner: Qwopus. What actually separated them: isBot from the localized name — meaning a human player named "Ordinateur" in multiplayer would silently become an AI. A real bug, confirmed in the code. not.toBeNull , with a comment documenting the contradiction.Qwopus's own flaws, to be fair: its solo menu requires filling two human-name fields before swapping the second for the bot blocking if you only fill one , the "~78%" claim had no math behind it, and its bot timer isn't cleared on menu exit guarded, but not cleaned up . | Round 1 short fix | Round 2 feature | | |---|---|---| | Winner | Tie | Qwopus | | Deciding factor | none identical diffs | strategy + tests + robustness | | Time | Qwopus 2.5× faster | Qwopus slightly slower but more code | --load-mode dio direct I/O , which both avoids the bug and loads 3–8× faster. pi -p --provider llama-router --model