{"slug": "i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run-failed", "title": "I set the pass bar before testing my Claude Code skills. The first run failed.", "summary": "A developer built three Claude Code skills for AI product managers and published the eval suite, including every failed run, on GitHub. Committing pass criteria to git before testing, the first run of the /build-or-not skill scored 0.00 after it issued a \"don't build\" verdict with no supporting sample; adding a \"no sample, no decision\" rule made the second run pass. Across 8 cases and 3 runs each on one model, the skills consistently stated decision bars and refused verdicts without evidence where plain Claude did not, at roughly $2 per full run.", "body_md": "I built three Claude Code skills for AI product managers ([ai-pm-skills](https://github.com/vishalhabib99/ai-pm-skills)). One of them, `/eval-plan`, exists to stop a specific habit: deciding what \"good enough\" means *after* the results come in. A bar set after the numbers can't fail.\n\nSo I held the skills to the same rule. Before running a single eval, I committed the pass bar to git. Then I ran them. The first run failed.\n\nClaude Code has a built-in eval runner, `claude plugin eval`. Each test case is a prompt plus graders, and it runs every case **with the plugin and without it**, so you see what the skill actually adds over plain Claude.\n\nI wrote 8 cases across the three skills. Three of them are deliberate \"should refuse\" cases, because refusing is where AI features quietly fail. Then I committed three gates:\n\n`/build-or-not` checks a feature idea against 4–8 real examples before anyone builds it. One test gave it no evidence and no research tools, then demanded a verdict.\n\nIt scored **0.00**. The skill even wrote that it couldn't run its own check, then said \"don't build\" anyway, backed by market knowledge it recalled and labeled \"public, well-known, not invented.\" Nobody had checked any of it for this decision.\n\nThe skill never said what to do when there's no sample, so the model filled the gap with confidence. The fix was one rule: **no sample, no decision.** \"Can't decide yet\" is now an outcome, with the exact sample that would settle it. I didn't touch the grader or the gate. Run 2 passed everything.\n\nThe third skill, `/agent-trust-review`, sorts an agent's risks into covered (with evidence), declined on purpose (with a reason), and genuinely missing. It took four runs to measure, and every failure was mine:\n\nIt's tempting to keep re-running until it's green. Instead, I committed a note saying the next run would be final *before* starting it, and would be reported whatever it showed. It passed.\n\n| Behavior | With the skill | Plain Claude | \n|---|---|---|\n| States the bar before deciding | 3 of 3 runs | 0 of 3 | \n| Refuses a verdict when there's no evidence | 3 of 3 | 0 of 3 | \n| Plans a rollback trigger for launch | 3 of 3 | 1 of 3 | \n| Separates a reasoned decline from an unexplained gap | 3 of 3 | 2 of 3 | \n| Gives two coverage numbers (owned areas vs. all areas) | 3 of 3 | 0 of 3 | \n\nOn four other cases plain Claude already did just as well: spotting hits a feature can't reach, pushing back on a bar set after the results, refusing to certify \"it's safe\" with no evidence, and (in the final run) refusing to credit unsupported claims. The skills aren't what makes those pass, and the README says so.\n\nThis is 8 cases, 3 runs each, one model. It's a check of the key behaviors, not a benchmark. Each full run cost about $2.\n\nThe skills, the eval suite, and every failed run are public: [github.com/vishalhabib99/ai-pm-skills](https://github.com/vishalhabib99/ai-pm-skills). Install in Claude Code with:\n\n```\n/plugin marketplace add vishalhabib99/ai-pm-skills\n/plugin install ai-pm-skills@ai-pm-skills\n```\n\nIf you run one of them on a real decision, I'd like to hear where it was wrong.", "url": "https://wpnews.pro/news/i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run-failed", "canonical_source": "https://dev.to/vishalhabib99/i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run-failed-1ef5", "published_at": "2026-09-23 02:00:12+00:00", "updated_at": "2026-09-23 02:22:36.350683+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Claude Code", "Anthropic", "GitHub", "vishalhabib99", "ai-pm-skills"], "alternates": {"html": "https://wpnews.pro/news/i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run-failed", "markdown": "https://wpnews.pro/news/i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run-failed.md", "text": "https://wpnews.pro/news/i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run-failed.txt", "jsonld": "https://wpnews.pro/news/i-set-the-pass-bar-before-testing-my-claude-code-skills-the-first-run-failed.jsonld"}}