{"slug": "i-benchmarked-claude-code-skills-against-a-placebo-and-half-of-mine-failed", "title": "I benchmarked Claude Code skills against a placebo — and half of mine failed", "summary": "A developer benchmarked Claude Code agent skills against a placebo and a no-instruction baseline, finding that half of the tested skills failed to outperform the placebo. The most-starred 'Karpathy Guidelines' skill beat the placebo but barely improved over no instructions, while the developer's own 20-line 'underkill' skill reduced source code by 23.8% at identical accuracy.", "body_md": "There's a whole ecosystem of \"agent skills\" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything.\n\nThat bothered me, because \"adding a plausible-sounding instruction\" and \"adding an instruction that works\" look *identical* until you measure them. So I built a benchmark with one rule, committed before I ran anything:\n\nNo skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers.\n\nThe placebo arm is the part almost nobody runs, and it turned out to be the most important one.\n\nMost \"battle-tested\" skill collections that measure anything at all compare skill-on vs skill-off. The problem: that comparison can't separate \"this skill works\" from \"adding *any* confident-sounding text changes the model's behavior.\" LLMs are suggestible. If you want to claim your skill did something, you have to show it beats a same-length instruction that contains no actual mechanism — just vibes.\n\nSo every result here is a three-way comparison — off / placebo / on — run K=5–8 times per task per arm, in isolated git workspaces, graded by hold-out acceptance tests the agent never sees, with every raw run log committed to the repo and the README regenerated from those logs in CI. 516 runs total, all on `claude-opus-4-8`\n\n.\n\nMy anti-over-engineering skill (`underkill`\n\n, ~20 lines) cut source LOC by **-23.8% vs baseline** at identical accuracy (60/60 hold-out passes). Good. But the interesting column is the placebo: a same-length \"write clean, minimal, professional code\" instruction didn't reduce code at all — it *increased* it on 8 of 12 tasks (median ~+19%).\n\nRead that again: telling the model to write clean code made it write *more* code. The generic exhortation added nothing but tokens, and the model padded to match the vibe. If I'd only compared on-vs-off, `underkill`\n\nwould have looked good. Compared against the placebo, it looks good *and* I can prove the effect isn't just instruction-presence.\n\nThe most-starred behavioral skill in the ecosystem right now is the \"Karpathy Guidelines\" (~196k stars) — four rules derived from Andrej Karpathy's observations on LLM coding pitfalls, one of which is \"Simplicity First: if you write 200 lines and it could be 50, rewrite it.\" That's a direct, falsifiable claim on the exact metric `underkill`\n\ntargets, so I ran it as a comparison arm on the same 12 tasks.\n\n| arm | median src LOC | hold-out accuracy |\n|---|---|---|\n| off (no instructions) | 10.5 | 60/60 |\n| placebo | 13.0 | 60/60 |\n| underkill (mine, ~20 lines) | 8.0 | 60/60 |\n| Karpathy Guidelines (~196k star) | 10.0 | 60/60 |\n\nThe famous skill **beats the placebo clearly (-23%)** — so its mechanism is real, not just prompt-presence. But against a no-instruction baseline it barely moves (**-4.8%**), because on small tasks Opus is already close to minimal. And my focused 20-line skill writes **20% less code than the 196k-star one**, at equal accuracy and slightly lower cost.\n\nThis isn't a dunk on the Karpathy skill — it addresses four different failure modes and only one is on this axis. The point is narrower and more useful: a rule aimed at one specific failure mode beats a broad, general ruleset that merely mentions it. Instruction *presence* is cheap; instruction *mechanism* is what pays.\n\nPublishing only wins would make the whole exercise worthless, so here are the rejects with their numbers:\n\n`tests-that-bite`\n\n`thrift`\n\nThe `thrift`\n\nsweep had the most counterintuitive result of the whole project: I ran the 92k-star \"caveman\" terse-mode ruleset as a comparison arm, and it measured **more expensive per run than no instructions at all** on every cost task. Agentic cost is input-dominated (files read, tool results, repeated context), so an instruction that compresses the model's *output* can't recoup the input tokens its own prompt adds on every turn. Output-compression advice that's true for chat responses can invert on agentic runs.\n\nThe tasks are small and self-contained (3–50 line solutions) — enough to gate a skill on, not enough to generalize to large codebases. K=5–8 gives you evidence, not proof. Pre-registration here is commit-ordering in a repo I control, self-attested. And hold-out material lives in the same repo, so it's reachable in principle — I run a CI check that scans every committed run log for references to the task material (516/516 clean). All of this is written up in the repo's \"Known Limitations,\" because a results log that only contains good news isn't worth trusting.\n\nTwo skills passed the gate (`underkill`\n\n, `repro-first`\n\n). They install as a Claude Code plugin or via `npx skills add sjh9714/skill-receipts`\n\n, and the installed snippet is byte-identical to what the benchmark measured (CI enforces that).\n\nIf you have a skill you think works — yours, or a famous one you want audited — the gate is the same for everyone: open an issue. I'd genuinely like to be proven wrong on some of these.\n\n**Repo + all raw logs:** [https://github.com/sjh9714/skill-receipts](https://github.com/sjh9714/skill-receipts)", "url": "https://wpnews.pro/news/i-benchmarked-claude-code-skills-against-a-placebo-and-half-of-mine-failed", "canonical_source": "https://dev.to/sjh9714/i-benchmarked-claude-code-skills-against-a-placebo-and-half-of-mine-failed-4okk", "published_at": "2026-07-24 12:50:36+00:00", "updated_at": "2026-07-24 13:02:46.401741+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-agents", "ai-research"], "entities": ["Claude Code", "Cursor", "Copilot", "Andrej Karpathy", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/i-benchmarked-claude-code-skills-against-a-placebo-and-half-of-mine-failed", "markdown": "https://wpnews.pro/news/i-benchmarked-claude-code-skills-against-a-placebo-and-half-of-mine-failed.md", "text": "https://wpnews.pro/news/i-benchmarked-claude-code-skills-against-a-placebo-and-half-of-mine-failed.txt", "jsonld": "https://wpnews.pro/news/i-benchmarked-claude-code-skills-against-a-placebo-and-half-of-mine-failed.jsonld"}}