I spent today shipping a tool I've wanted for months.
If you build with Claude, you've probably written a SKILL.md file. And you've probably shipped it based on gut feel.
That changes today.
The problem nobody talks about
Skills are just system prompt injections. The honest question is: does this skill actually improve Claude's outputs, or does it just feel like it does?
Most teams answer this by eyeballing a few responses. That's not evaluation. That's vibes. Three things make vibes-based skill evaluation dangerous:
Position bias — if you ask Claude to compare its own outputs, it favors whichever it sees first
Silent regression — model updates, skill edits, and context changes can silently make a skill worse
No shared rubric— every engineer scores skills differently, so "this skill is good" means nothing
What I built
skilleval — a CLI that gives you a repeatable, objective score for any SKILL.md in under 2 minutes.
bash
npx @dileeppandiya/skilleval ./my-skill --tasks ./tasks.yaml
Real output from the sample skill in the repo:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
skilleval results - api-design - 2 tasks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Skill effectiveness: +0.3 / 3
Tasks improved: 1 / 2 (50%)
Tasks hurt: 1 / 2 (50%)
Confidence: UNRATED (use --runs 3+ for confidence)
task-003 +2.5 Output A provides more robust API design...
task-004 -2.0 Output A is more comprehensive...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Runner: claude-sonnet-4-6 | Judge: gemini-3.5-flash
Estimated API cost this run: $0.101
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Notice the mixed signal. The skill helped on task-003 but hurt on task-004. skilleval doesn't inflate scores to make skills look good. It reports what the judge actually found.
How it works
Blind A/B testing — each task runs twice concurrently, with the skill injected into the system prompt vs. raw context only.
Randomized judge — a Gemini Flash judge compares outputs. Which output gets labeled A or B is randomized per task with a seeded RNG, eliminating position bias completely.
Margin-based scoring — the judge returns a winner + margin (0–3): margin 3 gives 3.0/0.0, margin 0 gives a genuine tie at 1.5/1.5.
Honest confidence — single runs show UNRATED. One sample tells you nothing about stability. Real confidence (HIGH/MEDIUM/LOW) only appears at --runs 3+.
bash
skilleval ./my-skill --tasks ./tasks.yaml --runs 3
Five things that make it different
tasks:
- id: login-endpoint
prompt: "Design a login endpoint"
assertions:
must_contain:
- "POST"
- "401"
must_not_contain:
- "GET /login"
min_length: 100
Assertion failures automatically count as hurt tasks, no LLM needed to know "missing POST method" is wrong.
Multi-turn conversation tasks — most real skills operate across turns, not single prompts. The skill injects into the system prompt for the full conversation, and the judge sees complete context when scoring.
Run history + regression detection — every run auto-saves to .skilleval/history/. After two runs:
bash
skilleval diff ./my-skill
── skilleval diff: api-design ──────────────────
vs previous run: 2026-07-11T14:30:00Z
Effectiveness: +0.3 → +0.8 (+0.5 ↑)
Tasks improved: 1 → 2 (+1 ↑)
Tasks hurt: 1 → 0 (-1 ↓)
This is "skill hell" prevention in practice — you can see the exact moment a skill started regressing.
bash
skilleval ./skill-v1 --compare ./skill-v2 --tasks ./tasks.yaml
No more "I think v2 is better." Now you know.
on:
pull_request:
paths:
- '**/SKILL.md'
jobs:
skilleval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dileepkpandiya/skilleval@main
with:
skill-path: ./my-skill
tasks: ./tasks/tasks.yaml
fail-below: '0.3'
fail-if-hurt-pct: '50'
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
gemini-api-key: ${{ secrets.GEMINI_API_KEY }}
plaintext
Exit code 0 = pass, 1 = gate failed, 2 = error.
Cost
Setup Cost
5 tasks, --runs 1, Gemini Flash judge ~$0.10
5 tasks, --runs 3 (real confidence) ~$0.30
10 tasks, --runs 3 ~$0.60
Use --cost to see an estimate before spending anything. Gemini Flash is the default judge, and the free tier handles casual iteration easily.
Quick start
bash
git clone https://github.com/dileepkpandiya/skilleval
cd skilleval
npx @dileeppandiya/skilleval ./samples/api-design \
--tasks ./tasks/sample-tasks.yaml
skilleval --init ./my-new-skill
## Install globally
npm install -g @dileeppandiya/skilleval
You'll need ANTHROPIC_API_KEY for the Claude runner and GEMINI_API_KEY for the default judge.
What's still missing
Honest gaps in v0.3.0:
Tool-call evaluation — if your skill affects which tools Claude calls, text-output scoring misses that
Visual history dashboard — the diff command is CLI only, no charts yet
Local model judge support — no Ollama/local-model judging for fully offline eval yet
The repo
MIT licensed, open source, TypeScript. 38 unit tests, zero API calls needed to run the test suite, GitHub Action included.
👉 github.com/dileepkpandiya/skilleval
What are you using to evaluate your skills today? I'd genuinely love to know what's broken about this for your use case, you can file an issue or drop a comment below.