How to Know If Your Claude SKILL.md Actually Works

A developer built skilleval, a CLI tool that provides objective, repeatable scoring for Claude SKILL.md files by running blind A/B tests with randomized judging to eliminate position bias. The tool reports margin-based scores, detects silent regression via run history, and supports multi-turn conversation tasks, giving teams a data-driven alternative to gut-feel evaluation.

I spent today shipping a tool I've wanted for months. If you build with Claude, you've probably written a SKILL.md file. And you've probably shipped it based on gut feel. That changes today. The problem nobody talks about Skills are just system prompt injections. The honest question is: does this skill actually improve Claude's outputs, or does it just feel like it does? Most teams answer this by eyeballing a few responses. That's not evaluation. That's vibes. Three things make vibes-based skill evaluation dangerous: Position bias — if you ask Claude to compare its own outputs, it favors whichever it sees first Silent regression — model updates, skill edits, and context changes can silently make a skill worse No shared rubric — every engineer scores skills differently, so "this skill is good" means nothing What I built skilleval — a CLI that gives you a repeatable, objective score for any SKILL.md in under 2 minutes. bash npx @dileeppandiya/skilleval ./my-skill --tasks ./tasks.yaml Real output from the sample skill in the repo: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ skilleval results - api-design - 2 tasks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Skill effectiveness: +0.3 / 3 Tasks improved: 1 / 2 50% Tasks hurt: 1 / 2 50% Confidence: UNRATED use --runs 3+ for confidence task-003 +2.5 Output A provides more robust API design... task-004 -2.0 Output A is more comprehensive... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Runner: claude-sonnet-4-6 | Judge: gemini-3.5-flash Estimated API cost this run: $0.101 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Notice the mixed signal. The skill helped on task-003 but hurt on task-004. skilleval doesn't inflate scores to make skills look good. It reports what the judge actually found. How it works Blind A/B testing — each task runs twice concurrently, with the skill injected into the system prompt vs. raw context only. Randomized judge — a Gemini Flash judge compares outputs. Which output gets labeled A or B is randomized per task with a seeded RNG, eliminating position bias completely. Margin-based scoring — the judge returns a winner + margin 0–3 : margin 3 gives 3.0/0.0, margin 0 gives a genuine tie at 1.5/1.5. Honest confidence — single runs show UNRATED. One sample tells you nothing about stability. Real confidence HIGH/MEDIUM/LOW only appears at --runs 3+. bash skilleval ./my-skill --tasks ./tasks.yaml --runs 3 Five things that make it different tasks: - id: login-endpoint prompt: "Design a login endpoint" assertions: must contain: - "POST" - "401" must not contain: - "GET /login" min length: 100 Assertion failures automatically count as hurt tasks, no LLM needed to know "missing POST method" is wrong. Multi-turn conversation tasks — most real skills operate across turns, not single prompts. The skill injects into the system prompt for the full conversation, and the judge sees complete context when scoring. Run history + regression detection — every run auto-saves to .skilleval/history/. After two runs: bash skilleval diff ./my-skill ── skilleval diff: api-design ────────────────── vs previous run: 2026-07-11T14:30:00Z Effectiveness: +0.3 → +0.8 +0.5 ↑ Tasks improved: 1 → 2 +1 ↑ Tasks hurt: 1 → 0 -1 ↓ This is "skill hell" prevention in practice — you can see the exact moment a skill started regressing. bash skilleval ./skill-v1 --compare ./skill-v2 --tasks ./tasks.yaml No more "I think v2 is better." Now you know. on: pull request: paths: - ' /SKILL.md' jobs: skilleval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dileepkpandiya/skilleval@main with: skill-path: ./my-skill tasks: ./tasks/tasks.yaml fail-below: '0.3' fail-if-hurt-pct: '50' anthropic-api-key: ${{ secrets.ANTHROPIC API KEY }} gemini-api-key: ${{ secrets.GEMINI API KEY }} plaintext Exit code 0 = pass, 1 = gate failed, 2 = error. Cost Setup Cost 5 tasks, --runs 1, Gemini Flash judge ~$0.10 5 tasks, --runs 3 real confidence ~$0.30 10 tasks, --runs 3 ~$0.60 Use --cost to see an estimate before spending anything. Gemini Flash is the default judge, and the free tier handles casual iteration easily. Quick start bash git clone https://github.com/dileepkpandiya/skilleval cd skilleval npx @dileeppandiya/skilleval ./samples/api-design \ --tasks ./tasks/sample-tasks.yaml skilleval --init ./my-new-skill Install globally npm install -g @dileeppandiya/skilleval You'll need ANTHROPIC API KEY for the Claude runner and GEMINI API KEY for the default judge. What's still missing Honest gaps in v0.3.0: Tool-call evaluation — if your skill affects which tools Claude calls, text-output scoring misses that Visual history dashboard — the diff command is CLI only, no charts yet Local model judge support — no Ollama/local-model judging for fully offline eval yet The repo MIT licensed, open source, TypeScript. 38 unit tests, zero API calls needed to run the test suite, GitHub Action included. 👉 github.com/dileepkpandiya/skilleval What are you using to evaluate your skills today? I'd genuinely love to know what's broken about this for your use case, you can file an issue or drop a comment below.