# How to Know If Your Claude SKILL.md Actually Works > Source: > Published: 2026-07-13 01:54:47+00:00 I spent today shipping a tool I've wanted for months. If you build with Claude, you've probably written a SKILL.md file. And you've probably shipped it based on gut feel. That changes today. **The problem nobody talks about** Skills are just system prompt injections. The honest question is: does this skill actually improve Claude's outputs, or does it just feel like it does? Most teams answer this by eyeballing a few responses. That's not evaluation. That's vibes. Three things make vibes-based skill evaluation dangerous: **Position bias** — if you ask Claude to compare its own outputs, it favors whichever it sees first **Silent regression** — model updates, skill edits, and context changes can silently make a skill worse **No shared rubric**— every engineer scores skills differently, so "this skill is good" means nothing What I built **skilleval** — a CLI that gives you a repeatable, objective score for any SKILL.md in under 2 minutes. ``` bash npx @dileeppandiya/skilleval ./my-skill --tasks ./tasks.yaml ``` Real output from the sample skill in the repo: ``` ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ skilleval results - api-design - 2 tasks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Skill effectiveness: +0.3 / 3 Tasks improved: 1 / 2 (50%) Tasks hurt: 1 / 2 (50%) Confidence: UNRATED (use --runs 3+ for confidence) task-003 +2.5 Output A provides more robust API design... task-004 -2.0 Output A is more comprehensive... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Runner: claude-sonnet-4-6 | Judge: gemini-3.5-flash Estimated API cost this run: $0.101 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ``` Notice the mixed signal. The skill helped on task-003 but hurt on task-004. skilleval doesn't inflate scores to make skills look good. It reports what the judge actually found. How it works **Blind A/B testing** — each task runs twice concurrently, with the skill injected into the system prompt vs. raw context only. **Randomized judge** — a Gemini Flash judge compares outputs. Which output gets labeled A or B is randomized per task with a seeded RNG, eliminating position bias completely. **Margin-based scoring** — the judge returns a winner + margin (0–3): margin 3 gives 3.0/0.0, margin 0 gives a genuine tie at 1.5/1.5. Honest confidence — single runs show UNRATED. One sample tells you nothing about stability. Real confidence (HIGH/MEDIUM/LOW) only appears at --runs 3+. ``` bash skilleval ./my-skill --tasks ./tasks.yaml --runs 3 ``` Five things that make it different ``` tasks: - id: login-endpoint prompt: "Design a login endpoint" assertions: must_contain: - "POST" - "401" must_not_contain: - "GET /login" min_length: 100 ``` Assertion failures automatically count as hurt tasks, no LLM needed to know "missing POST method" is wrong. **Multi-turn conversation tasks** — most real skills operate across turns, not single prompts. The skill injects into the system prompt for the full conversation, and the judge sees complete context when scoring. **Run history + regression detection** — every run auto-saves to .skilleval/history/. After two runs: ``` bash skilleval diff ./my-skill ── skilleval diff: api-design ────────────────── vs previous run: 2026-07-11T14:30:00Z Effectiveness: +0.3 → +0.8 (+0.5 ↑) Tasks improved: 1 → 2 (+1 ↑) Tasks hurt: 1 → 0 (-1 ↓) ``` This is "skill hell" prevention in practice — you can see the exact moment a skill started regressing. ``` bash skilleval ./skill-v1 --compare ./skill-v2 --tasks ./tasks.yaml ``` No more "I think v2 is better." Now you know. ``` on: pull_request: paths: - '**/SKILL.md' jobs: skilleval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dileepkpandiya/skilleval@main with: skill-path: ./my-skill tasks: ./tasks/tasks.yaml fail-below: '0.3' fail-if-hurt-pct: '50' anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }} gemini-api-key: ${{ secrets.GEMINI_API_KEY }} ``` plaintext Exit code 0 = pass, 1 = gate failed, 2 = error. Cost Setup Cost 5 tasks, --runs 1, Gemini Flash judge ~$0.10 5 tasks, --runs 3 (real confidence) ~$0.30 10 tasks, --runs 3 ~$0.60 Use --cost to see an estimate before spending anything. Gemini Flash is the default judge, and the free tier handles casual iteration easily. Quick start bash ``` git clone https://github.com/dileepkpandiya/skilleval cd skilleval npx @dileeppandiya/skilleval ./samples/api-design \ --tasks ./tasks/sample-tasks.yaml skilleval --init ./my-new-skill ## Install globally npm install -g @dileeppandiya/skilleval ``` You'll need ANTHROPIC_API_KEY for the Claude runner and GEMINI_API_KEY for the default judge. **What's still missing** Honest gaps in v0.3.0: **Tool-call evaluation** — if your skill affects which tools Claude calls, text-output scoring misses that **Visual history dashboard** — the diff command is CLI only, no charts yet **Local model judge support** — no Ollama/local-model judging for fully offline eval yet The repo MIT licensed, open source, TypeScript. 38 unit tests, zero API calls needed to run the test suite, GitHub Action included. 👉 github.com/dileepkpandiya/skilleval What are you using to evaluate your skills today? I'd genuinely love to know what's broken about this for your use case, you can file an issue or drop a comment below.