I built three Claude Code skills for AI product managers (ai-pm-skills). One of them, /eval-plan, exists to stop a specific habit: deciding what "good enough" means after the results come in. A bar set after the numbers can't fail.
So I held the skills to the same rule. Before running a single eval, I committed the pass bar to git. Then I ran them. The first run failed.
Claude Code has a built-in eval runner, claude plugin eval. Each test case is a prompt plus graders, and it runs every case with the plugin and without it, so you see what the skill actually adds over plain Claude.
I wrote 8 cases across the three skills. Three of them are deliberate "should refuse" cases, because refusing is where AI features quietly fail. Then I committed three gates:
/build-or-not checks a feature idea against 4–8 real examples before anyone builds it. One test gave it no evidence and no research tools, then demanded a verdict.
It scored 0.00. The skill even wrote that it couldn't run its own check, then said "don't build" anyway, backed by market knowledge it recalled and labeled "public, well-known, not invented." Nobody had checked any of it for this decision.
The skill never said what to do when there's no sample, so the model filled the gap with confidence. The fix was one rule: no sample, no decision. "Can't decide yet" is now an outcome, with the exact sample that would settle it. I didn't touch the grader or the gate. Run 2 passed everything.
The third skill, /agent-trust-review, sorts an agent's risks into covered (with evidence), declined on purpose (with a reason), and genuinely missing. It took four runs to measure, and every failure was mine:
It's tempting to keep re-running until it's green. Instead, I committed a note saying the next run would be final before starting it, and would be reported whatever it showed. It passed.
| Behavior | With the skill | Plain Claude |
|---|---|---|
| States the bar before deciding | 3 of 3 runs | 0 of 3 |
| Refuses a verdict when there's no evidence | 3 of 3 | 0 of 3 |
| Plans a rollback trigger for launch | 3 of 3 | 1 of 3 |
| Separates a reasoned decline from an unexplained gap | 3 of 3 | 2 of 3 |
| Gives two coverage numbers (owned areas vs. all areas) | 3 of 3 | 0 of 3 |
On four other cases plain Claude already did just as well: spotting hits a feature can't reach, pushing back on a bar set after the results, refusing to certify "it's safe" with no evidence, and (in the final run) refusing to credit unsupported claims. The skills aren't what makes those pass, and the README says so.
This is 8 cases, 3 runs each, one model. It's a check of the key behaviors, not a benchmark. Each full run cost about $2.
The skills, the eval suite, and every failed run are public: github.com/vishalhabib99/ai-pm-skills. Install in Claude Code with:
/plugin marketplace add vishalhabib99/ai-pm-skills
/plugin install ai-pm-skills@ai-pm-skills
If you run one of them on a real decision, I'd like to hear where it was wrong.