{"slug": "claude-code-plugin-eval-gate-skills-on-delta-before-you-merge", "title": "Claude Code plugin eval: gate skills on delta before you merge", "summary": "Anthropic shipped a first-party evaluation command for Claude Code, `claude plugin eval`, that runs a test suite with a plugin loaded and again without it to report the delta the plugin actually contributes. The workflow, available in Claude Code v2.1.269+, lets teams create cases under an `evals/` directory, grade outcomes, and fail CI builds when scores drop below a threshold. The documentation warns that a case scoring 1.00 with a plugin loaded can still be useless if the model already solves the prompt unaided, since a near-zero delta means the plugin did not move the outcome.", "body_md": "Claude Code now ships a first-party way to prove a plugin or skill actually helps. The command is `claude plugin eval`. It runs your suite with the plugin loaded and again without it, then reports `WITH`, `W/OUT`, and `Δ` — the lift the plugin contributed. That changes what production teams should do before they merge a skill into a kit repo or pin it in CI.\n\nThis is not another “write better prompts” essay. It is a builder workflow for Claude Code v2.1.269+: create cases under `evals/`, grade outcomes, and fail the build when the score drops. If you already ship agent docs in an owned monorepo, pair this with [Claude Code on an OTF kit: the CLAUDE.md and prompts that ship with the repo](https://dev.to/blog/claude-code-otf-kit-docs). For feature-level evals outside plugins, keep using [A practical LLM evaluation loop for AI features that need to ship](https://dev.to/blog/llm-evaluation-loop) — different layer, different artifact.\n\nAnthropic’s [plugin evals docs](https://code.claude.com/docs/en/plugin-evals) describe a dedicated CLI path for plugin and skill authors. The important claims for builders:\n\n`claude --version`, then `claude update` if needed).`evals/` directory next to your plugin manifest (`plugin.json` or `.claude-plugin/plugin.json`).`Δ` is with-arm score minus without-arm score.`claude plugin eval init` can propose cases and graders interactively; `claude plugin eval init --bare <case>` writes a blank template.`--threshold`, pin `--model` / `--judge-model`, keep reports local with `--no-publish`, and cap spend with `--max-cost-usd`.\nThat is the DO: stop merging skills because a demo chat looked fine. Measure contribution, then gate.\n\nA case that scores `1.00` with the plugin loaded can still be useless. Claude might already solve the prompt without your skill. The docs call this out: if `WITH` and `W/OUT` are both high, `Δ` near zero means the plugin did not move the outcome.\n\nProduction implications:\n\n`Δ ≈ 0` with a `tool_used: Skill` grader failing — Claude never chose your skill on natural phrasing. Fix the skill `description`, re-run, compare.` tool_used` graders on `Skill` are excluded from both arms’ scored totals so you do not invent lift by checking something impossible without the plugin.`regex`, `tool_order`, and `file_exists` are free transcript/file checks. `llm` and `baseline` call a judge model and add to the run’s list-price estimate.`--runs 1 --ablation none`; trust needs the default three.\nIf you only ever run the with-arm, you are grading absolute behavior. That is fine while drafting graders. Before you call a skill “done,” turn the baseline back on and read `Δ`.\n\nFrom the plugin root (the directory that contains the manifest):\n\n```\nclaude --version   # need >= 2.1.269\nclaude plugin eval init --bare commit-message\n```\n\nYou get something shaped like:\n\n```\nevals/commit-message/\n├── prompt.md\n└── graders/\n    └── criteria.md\n```\n\nEdit `prompt.md` so the body is a request a user would type — do not name the skill in the prompt:\n\n```\n---\nmax_turns: 10\nallowed_tools: [Read, Glob, Grep, Skill]\n---\n\nWrite me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.\n```\n\nAdd a result grader (`llm` or `regex`) and a skill-fired grader:\n\n```\n---\ntype: tool_used\ntool: Skill\ninput_match: '\"skill\"\\s*:\\s*\"(?:[\\w-]+:)?your-skill-name\"'\n---\n```\n\nThen run:\n\n```\nclaude plugin eval .\n```\n\nExpect six runs for one case at defaults (3× with, 3× without). Open the HTML report path printed at the end. Iterate on description and graders until `Δ` is positive for the prompts you care about.\n\nFor MCP-backed skills, put mocks under `evals/mocks/<server>/<tool>.md` so CI does not need the real service. Use `--scaffold` only for suites you trust — scaffold scripts run as you, outside the agent sandbox.\n\nThe docs’ recommended CI shape is explicit:\n\n```\nclaude plugin eval . \\\n  --trust-plugin \\\n  --json results.json \\\n  --threshold 0.8 \\\n  --model claude-sonnet-5 \\\n  --judge-model claude-haiku-4-5 \\\n  --no-publish \\\n  --max-cost-usd 20\n```\n\nExit codes that matter:\n\n`0` — every case met `1` — a case scored below threshold, files failed to load, or trust was missing without `--trust-plugin`\n`2` — partial run (`partial: true`\nPin both models so a provider rollout is not mistaken for a plugin regression. Keep every-change suites on free graders when you can; reserve `llm` judges for short outputs with concrete PASS/FAIL rubrics. Leave `partial: true` results out of trend charts.\n\nCredentials and install still belong on the runner (`ANTHROPIC_API_KEY` or your normal Claude Code auth). Without `--trust-plugin`, a non-interactive job against an untrusted checkout exits `1`.\n\nOTF’s wedge is not “another sandboxed chat.” It is an owned repo agents keep editing — `CLAUDE.md`, prompts, and skills that travel with the product. Plugin eval is the missing acceptance gate for those skills:\n\n| Layer | Question | Artifact | \n|---|---|---|\n| Repo conventions | Can an agent find the seams? | `CLAUDE.md` /`.cursorrules` (see the kit docs post above) | \n| Change acceptance | Did this PR do what we asked? | [AI coding agent acceptance checklist](https://dev.to/blog/ai-agent-acceptance-checklist) | \n| Plugin contribution | Did the skill raise the score vs no plugin? | `claude plugin eval` +`Δ` | \n| Product AI features | Does the user-facing model path hold? | LLM evaluation loop | \n\nPut the suite next to the plugin you ship. Fail CI when `Δ` collapses after a description tweak or a model pin change. That is how skills stay production assets instead of chat folklore.\n\nSecurity note from the same docs: pointing `claude plugin eval` at a plugin is the same trust decision as `claude --plugin-dir`. Only evaluate plugins you trust. Hooks and real MCP servers you opt into with `--allow-real-servers` or `--mocks off` run outside the agent sandbox.\n\nIf you are still choosing between a sandboxed MVP host and a repo you own, start with the kit spine, then add this gate. Skills without `Δ` are demos. Skills with a CI threshold are release machinery.", "url": "https://wpnews.pro/news/claude-code-plugin-eval-gate-skills-on-delta-before-you-merge", "canonical_source": "https://dev.to/davekurian/claude-code-plugin-eval-gate-skills-on-delta-before-you-merge-5bh6", "published_at": "2026-09-14 16:06:38+00:00", "updated_at": "2026-09-14 16:25:30.739328+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "mlops", "large-language-models"], "entities": ["Anthropic", "Claude Code", "claude plugin eval"], "alternates": {"html": "https://wpnews.pro/news/claude-code-plugin-eval-gate-skills-on-delta-before-you-merge", "markdown": "https://wpnews.pro/news/claude-code-plugin-eval-gate-skills-on-delta-before-you-merge.md", "text": "https://wpnews.pro/news/claude-code-plugin-eval-gate-skills-on-delta-before-you-merge.txt", "jsonld": "https://wpnews.pro/news/claude-code-plugin-eval-gate-skills-on-delta-before-you-merge.jsonld"}}