There's a whole ecosystem of "agent skills" now — reusable instruction files you drop into Claude Code (or Cursor, or Copilot) to make the model write cleaner code, debug more carefully, use fewer tokens, and so on. Some of these repos have tens of thousands of GitHub stars. Almost none of them ship a single number telling you whether the skill actually does anything.
That bothered me, because "adding a plausible-sounding instruction" and "adding an instruction that works" look identical until you measure them. So I built a benchmark with one rule, committed before I ran anything:
No skill gets merged unless it beats both a no-instruction baseline AND a placebo prompt on its pre-registered target metric, measured on hidden hold-out tests, with accuracy not allowed to drop. Skills that fail are published anyway, with their numbers.
The placebo arm is the part almost nobody runs, and it turned out to be the most important one.
Most "battle-tested" skill collections that measure anything at all compare skill-on vs skill-off. The problem: that comparison can't separate "this skill works" from "adding any confident-sounding text changes the model's behavior." LLMs are suggestible. If you want to claim your skill did something, you have to show it beats a same-length instruction that contains no actual mechanism — just vibes.
So every result here is a three-way comparison — off / placebo / on — run K=5–8 times per task per arm, in isolated git workspaces, graded by hold-out acceptance tests the agent never sees, with every raw run log committed to the repo and the README regenerated from those logs in CI. 516 runs total, all on claude-opus-4-8
.
My anti-over-engineering skill (underkill
, ~20 lines) cut source LOC by -23.8% vs baseline at identical accuracy (60/60 hold-out passes). Good. But the interesting column is the placebo: a same-length "write clean, minimal, professional code" instruction didn't reduce code at all — it increased it on 8 of 12 tasks (median ~+19%).
Read that again: telling the model to write clean code made it write more code. The generic exhortation added nothing but tokens, and the model padded to match the vibe. If I'd only compared on-vs-off, underkill
would have looked good. Compared against the placebo, it looks good and I can prove the effect isn't just instruction-presence.
The most-starred behavioral skill in the ecosystem right now is the "Karpathy Guidelines" (~196k stars) — four rules derived from Andrej Karpathy's observations on LLM coding pitfalls, one of which is "Simplicity First: if you write 200 lines and it could be 50, rewrite it." That's a direct, falsifiable claim on the exact metric underkill
targets, so I ran it as a comparison arm on the same 12 tasks.
| arm | median src LOC | hold-out accuracy |
|---|---|---|
| off (no instructions) | 10.5 | 60/60 |
| placebo | 13.0 | 60/60 |
| underkill (mine, ~20 lines) | 8.0 | 60/60 |
| Karpathy Guidelines (~196k star) | 10.0 | 60/60 |
The famous skill beats the placebo clearly (-23%) — so its mechanism is real, not just prompt-presence. But against a no-instruction baseline it barely moves (-4.8%), because on small tasks Opus is already close to minimal. And my focused 20-line skill writes 20% less code than the 196k-star one, at equal accuracy and slightly lower cost.
This isn't a dunk on the Karpathy skill — it addresses four different failure modes and only one is on this axis. The point is narrower and more useful: a rule aimed at one specific failure mode beats a broad, general ruleset that merely mentions it. Instruction presence is cheap; instruction mechanism is what pays.
Publishing only wins would make the whole exercise worthless, so here are the rejects with their numbers:
tests-that-bite
thrift
The thrift
sweep had the most counterintuitive result of the whole project: I ran the 92k-star "caveman" terse-mode ruleset as a comparison arm, and it measured more expensive per run than no instructions at all on every cost task. Agentic cost is input-dominated (files read, tool results, repeated context), so an instruction that compresses the model's output can't recoup the input tokens its own prompt adds on every turn. Output-compression advice that's true for chat responses can invert on agentic runs.
The tasks are small and self-contained (3–50 line solutions) — enough to gate a skill on, not enough to generalize to large codebases. K=5–8 gives you evidence, not proof. Pre-registration here is commit-ordering in a repo I control, self-attested. And hold-out material lives in the same repo, so it's reachable in principle — I run a CI check that scans every committed run log for references to the task material (516/516 clean). All of this is written up in the repo's "Known Limitations," because a results log that only contains good news isn't worth trusting.
Two skills passed the gate (underkill
, repro-first
). They install as a Claude Code plugin or via npx skills add sjh9714/skill-receipts
, and the installed snippet is byte-identical to what the benchmark measured (CI enforces that).
If you have a skill you think works — yours, or a famous one you want audited — the gate is the same for everyone: open an issue. I'd genuinely like to be proven wrong on some of these.
**Repo + all raw logs:** [https://github.com/sjh9714/skill-receipts](https://github.com/sjh9714/skill-receipts)