{"slug": "i-benchmarked-two-local-llms-on-real-dev-work-qwopus-27b-vs-muse-glimmer-30b", "title": "I Benchmarked Two Local LLMs on Real Dev Work — Qwopus 27B vs Muse Glimmer 30B", "summary": "A developer benchmarked two open-weight local LLMs, Qwopus 3.6 27B and Meta's Muse Glimmer 30B, on real development tasks in a JavaScript project. Both models produced byte-identical fixes for a PWA regression, but Qwopus outperformed Muse in a feature implementation, with a third model, Codex, serving as referee. The developer found that on short tasks the models are interchangeable, but richer tasks reveal differences in strategy and robustness.", "body_md": "Two open-weight models, one 20 GB GPU, two real development tasks, and a third model as the referee. Here is what actually happened when I made Qwopus 3.6 27B and Meta's Muse Glimmer 30B implement a bug fix and then a full feature in my own project.\n\nBoth models ran fully local on an AMD Radeon RX 7900 XT (20 GB VRAM) via a llama.cpp multi-model router (one OpenAI-compatible endpoint, GGUF models, `load-mode=dio`\n\n— more on why below). Each model was driven by the pi CLI in non-interactive mode with `--thinking high`\n\n. A third model — Codex, through a disciplined stdin wrapper — reviewed both outputs and gave the verdict.\n\nThe fairness method was simple but strict:\n\nThe test project: **Jeu de Cochons** (a \"Pass the Pigs\" dice game, vanilla JS PWA on Vite + Vitest) — real code, real tests, no toy repo.\n\n| Qwopus 3.6 27B | Muse Glimmer 30B | |\n|---|---|---|\n| Source | Community fine-tune of Qwen 3.6 | Meta (distilled from Muse Spark) |\n| Size | 27B | 29.6B |\n| Quant | IQ4_XS (~15 GB) | UD-Q4_K_XL (~14.8 GB) |\n\nThe project had a broken PWA: a commit that added a `/jeu-de-cochons/`\n\nbase path for GitHub Pages had broken **3 service-worker tests** (manifest, precache, offline navigation fallback). Task: fix the regression **without touching the tests**, keep the other 84 green.\n\n| Qwopus | Muse | |\n|---|---|---|\n| PWA tests (11) | 11/11 ✅ | 11/11 ✅ |\n| Full suite (87) | 87/87 ✅ | 87/87 ✅ |\n| Files touched | 2 | 2 |\n| Diff size | +4/−4 | +4/−4 |\n| Wall time | ~8.5 min | ~21 min |\n| Leftover artifacts | none | one `.bak` file |\n\n**The remarkable result: both models produced a byte-identical diff.** Same diagnosis (a lost capture group in the asset-discovery regex that crashed precache), same manifest fix, same minimal solution. Codex called it a perfect tie on code — but rejected both on a latent conflict neither model noticed: reverting the manifest to `/`\n\nbreaks the `/jeu-de-cochons/`\n\nproduction deployment (404 icons). The tests hardcode the root path; the deployment lives under a subpath. That contradiction was visible in the task itself, and only the reviewer caught it.\n\n**Lesson #1:** on a short, well-bounded task, the two models are interchangeable. Muse thought 2.5× longer and produced the same bytes. You need a richer task to separate them.\n\nAdd a **single-player mode against an AI opponent**: menu selection, bot identity in game state, a *thoughtful* decision strategy (a pure function choosing to re-roll or bank based on turn score and game situation), automatic bot turns with human controls disabled, unit + integration tests, no regressions, build passing.\n\n| Qwopus | Muse | |\n|---|---|---|\n| Existing tests | 84 ✅ | 84 ✅ |\n| New tests written | 18 (12 unit + 6 integration) | 7 |\n| Total green | 102/105 | 91/94 |\n| Build | ✅ | ✅ |\n| Wall time | ~32 min | ~26 min |\n| Leftover artifacts | none | uncommitted `dist/`\n|\n\n**The AI strategies they designed:**\n\n*Qwopus — adaptive threshold:*\n\n`isBot`\n\nflag on the player*Muse — fixed threshold:*\n\n**Codex verdict:**\n\n| Criterion | Qwopus | Muse |\n|---|---|---|\nAI strategy (primary) |\n8/10 |\n5/10 |\n| Architecture | 8/10 | 6/10 |\n| Robustness | 7/10 | 5/10 |\n| Minimalism | 6/10 | 7/10 |\n| Tests | 8/10 | 4/10 |\n| Verdict | CONDITIONAL | CONDITIONAL |\n\n**Winner: Qwopus.**\n\nWhat actually separated them:\n\n`isBot`\n\nfrom the localized name — meaning a human player named \"Ordinateur\" in multiplayer would silently become an AI. A real bug, confirmed in the code.`not.toBeNull()`\n\n), with a comment documenting the contradiction.Qwopus's own flaws, to be fair: its solo menu requires filling two human-name fields before swapping the second for the bot (blocking if you only fill one), the \"~78%\" claim had no math behind it, and its bot timer isn't cleared on menu exit (guarded, but not cleaned up).\n\n| Round 1 (short fix) | Round 2 (feature) | |\n|---|---|---|\n| Winner | Tie | Qwopus |\n| Deciding factor | none (identical diffs) | strategy + tests + robustness |\n| Time | Qwopus 2.5× faster | Qwopus slightly slower (but more code) |\n\n`--load-mode dio`\n\n(direct I/O), which both avoids the bug and loads 3–8× faster.`pi -p --provider llama-router --model <model> --thinking high \"<task>\"`\n\n)*Honesty clauses: this is a single run per model per task, not a statistical benchmark. Both tasks came from one small project, and the Codex review is one reviewer's judgment. The numbers are real and were verified against the artifacts, but treat them as a data point — not a ranking of these models in general.*", "url": "https://wpnews.pro/news/i-benchmarked-two-local-llms-on-real-dev-work-qwopus-27b-vs-muse-glimmer-30b", "canonical_source": "https://dev.to/chpomob/i-benchmarked-two-local-llms-on-real-dev-work-qwopus-27b-vs-muse-glimmer-30b-36ni", "published_at": "2026-08-11 15:22:01+00:00", "updated_at": "2026-08-11 15:48:17.301521+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools"], "entities": ["Qwopus", "Muse Glimmer", "Meta", "Codex", "AMD Radeon RX 7900 XT", "llama.cpp", "Jeu de Cochons", "Vite"], "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-two-local-llms-on-real-dev-work-qwopus-27b-vs-muse-glimmer-30b", "markdown": "https://wpnews.pro/news/i-benchmarked-two-local-llms-on-real-dev-work-qwopus-27b-vs-muse-glimmer-30b.md", "text": "https://wpnews.pro/news/i-benchmarked-two-local-llms-on-real-dev-work-qwopus-27b-vs-muse-glimmer-30b.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-two-local-llms-on-real-dev-work-qwopus-27b-vs-muse-glimmer-30b.jsonld"}}