{"slug": "how-to-know-if-your-claude-skill-md-actually-works", "title": "How to Know If Your Claude SKILL.md Actually Works", "summary": "A developer built skilleval, a CLI tool that provides objective, repeatable scoring for Claude SKILL.md files by running blind A/B tests with randomized judging to eliminate position bias. The tool reports margin-based scores, detects silent regression via run history, and supports multi-turn conversation tasks, giving teams a data-driven alternative to gut-feel evaluation.", "body_md": "I spent today shipping a tool I've wanted for months.\n\nIf you build with Claude, you've probably written a SKILL.md file. And you've probably shipped it based on gut feel.\n\nThat changes today.\n\n**The problem nobody talks about**\n\nSkills are just system prompt injections. The honest question is: does this skill actually improve Claude's outputs, or does it just feel like it does?\n\nMost teams answer this by eyeballing a few responses. That's not evaluation. That's vibes. Three things make vibes-based skill evaluation dangerous:\n\n**Position bias** — if you ask Claude to compare its own outputs, it favors whichever it sees first\n\n**Silent regression** — model updates, skill edits, and context changes can silently make a skill worse\n\n**No shared rubric**— every engineer scores skills differently, so \"this skill is good\" means nothing\n\nWhat I built\n\n**skilleval** — a CLI that gives you a repeatable, objective score for any SKILL.md in under 2 minutes.\n\n```\nbash\nnpx @dileeppandiya/skilleval ./my-skill --tasks ./tasks.yaml\n```\n\nReal output from the sample skill in the repo:\n\n```\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nskilleval results - api-design - 2 tasks\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nSkill effectiveness: +0.3 / 3\nTasks improved: 1 / 2 (50%)\nTasks hurt: 1 / 2 (50%)\nConfidence: UNRATED (use --runs 3+ for confidence)\ntask-003 +2.5 Output A provides more robust API design...\ntask-004 -2.0 Output A is more comprehensive...\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nRunner: claude-sonnet-4-6 | Judge: gemini-3.5-flash\nEstimated API cost this run: $0.101\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n```\n\nNotice the mixed signal. The skill helped on task-003 but hurt on task-004. skilleval doesn't inflate scores to make skills look good. It reports what the judge actually found.\n\nHow it works\n\n**Blind A/B testing** — each task runs twice concurrently, with the skill injected into the system prompt vs. raw context only.\n\n**Randomized judge** — a Gemini Flash judge compares outputs. Which output gets labeled A or B is randomized per task with a seeded RNG, eliminating position bias completely.\n\n**Margin-based scoring** — the judge returns a winner + margin (0–3): margin 3 gives 3.0/0.0, margin 0 gives a genuine tie at 1.5/1.5.\n\nHonest confidence — single runs show UNRATED. One sample tells you nothing about stability. Real confidence (HIGH/MEDIUM/LOW) only appears at --runs 3+.\n\n```\nbash\nskilleval ./my-skill --tasks ./tasks.yaml --runs 3\n```\n\nFive things that make it different\n\n```\ntasks:\n  - id: login-endpoint\n    prompt: \"Design a login endpoint\"\n    assertions:\n      must_contain:\n        - \"POST\"\n        - \"401\"\n      must_not_contain:\n        - \"GET /login\"\n      min_length: 100\n```\n\nAssertion failures automatically count as hurt tasks, no LLM needed to know \"missing POST method\" is wrong.\n\n**Multi-turn conversation tasks** — most real skills operate across turns, not single prompts. The skill injects into the system prompt for the full conversation, and the judge sees complete context when scoring.\n\n**Run history + regression detection** — every run auto-saves to .skilleval/history/. After two runs:\n\n```\nbash\nskilleval diff ./my-skill\n── skilleval diff: api-design ──────────────────\nvs previous run: 2026-07-11T14:30:00Z\nEffectiveness: +0.3 → +0.8 (+0.5 ↑)\nTasks improved: 1 → 2 (+1 ↑)\nTasks hurt: 1 → 0 (-1 ↓)\n```\n\nThis is \"skill hell\" prevention in practice — you can see the exact moment a skill started regressing.\n\n```\nbash\nskilleval ./skill-v1 --compare ./skill-v2 --tasks ./tasks.yaml\n```\n\nNo more \"I think v2 is better.\" Now you know.\n\n```\non:\n  pull_request:\n    paths:\n      - '**/SKILL.md'\n\njobs:\n  skilleval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: dileepkpandiya/skilleval@main\n        with:\n          skill-path: ./my-skill\n          tasks: ./tasks/tasks.yaml\n          fail-below: '0.3'\n          fail-if-hurt-pct: '50'\n          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}\n          gemini-api-key: ${{ secrets.GEMINI_API_KEY }}\n```\n\nplaintext\n\nExit code 0 = pass, 1 = gate failed, 2 = error.\n\nCost\n\nSetup Cost\n\n5 tasks, --runs 1, Gemini Flash judge ~$0.10\n\n5 tasks, --runs 3 (real confidence) ~$0.30\n\n10 tasks, --runs 3 ~$0.60\n\nUse --cost to see an estimate before spending anything. Gemini Flash is the default judge, and the free tier handles casual iteration easily.\n\nQuick start\n\nbash\n\n```\ngit clone https://github.com/dileepkpandiya/skilleval\ncd skilleval\nnpx @dileeppandiya/skilleval ./samples/api-design \\\n  --tasks ./tasks/sample-tasks.yaml\nskilleval --init ./my-new-skill\n## Install globally\nnpm install -g @dileeppandiya/skilleval\n```\n\nYou'll need ANTHROPIC_API_KEY for the Claude runner and GEMINI_API_KEY for the default judge.\n\n**What's still missing**\n\nHonest gaps in v0.3.0:\n\n**Tool-call evaluation** — if your skill affects which tools Claude calls, text-output scoring misses that\n\n**Visual history dashboard** — the diff command is CLI only, no charts yet\n\n**Local model judge support** — no Ollama/local-model judging for fully offline eval yet\n\nThe repo\n\nMIT licensed, open source, TypeScript. 38 unit tests, zero API calls needed to run the test suite, GitHub Action included.\n\n👉 github.com/dileepkpandiya/skilleval\n\nWhat are you using to evaluate your skills today? I'd genuinely love to know what's broken about this for your use case, you can file an issue or drop a comment below.", "url": "https://wpnews.pro/news/how-to-know-if-your-claude-skill-md-actually-works", "canonical_source": "https://dev.to/dileeppandiya/how-to-know-if-your-claude-skillmd-actually-works-3j4f", "published_at": "2026-07-13 01:54:47+00:00", "updated_at": "2026-07-13 02:14:30.720363+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models", "ai-agents"], "entities": ["Claude", "skilleval", "Gemini", "dileeppandiya"], "alternates": {"html": "https://wpnews.pro/news/how-to-know-if-your-claude-skill-md-actually-works", "markdown": "https://wpnews.pro/news/how-to-know-if-your-claude-skill-md-actually-works.md", "text": "https://wpnews.pro/news/how-to-know-if-your-claude-skill-md-actually-works.txt", "jsonld": "https://wpnews.pro/news/how-to-know-if-your-claude-skill-md-actually-works.jsonld"}}