# Claude Code plugin eval: gate skills on delta before you merge > Source: > Published: 2026-09-14 16:06:38+00:00 Claude Code now ships a first-party way to prove a plugin or skill actually helps. The command is `claude plugin eval`. It runs your suite with the plugin loaded and again without it, then reports `WITH`, `W/OUT`, and `Δ` — the lift the plugin contributed. That changes what production teams should do before they merge a skill into a kit repo or pin it in CI. This is not another “write better prompts” essay. It is a builder workflow for Claude Code v2.1.269+: create cases under `evals/`, grade outcomes, and fail the build when the score drops. If you already ship agent docs in an owned monorepo, pair this with [Claude Code on an OTF kit: the CLAUDE.md and prompts that ship with the repo](https://dev.to/blog/claude-code-otf-kit-docs). For feature-level evals outside plugins, keep using [A practical LLM evaluation loop for AI features that need to ship](https://dev.to/blog/llm-evaluation-loop) — different layer, different artifact. Anthropic’s [plugin evals docs](https://code.claude.com/docs/en/plugin-evals) describe a dedicated CLI path for plugin and skill authors. The important claims for builders: `claude --version`, then `claude update` if needed).`evals/` directory next to your plugin manifest (`plugin.json` or `.claude-plugin/plugin.json`).`Δ` is with-arm score minus without-arm score.`claude plugin eval init` can propose cases and graders interactively; `claude plugin eval init --bare ` writes a blank template.`--threshold`, pin `--model` / `--judge-model`, keep reports local with `--no-publish`, and cap spend with `--max-cost-usd`. That is the DO: stop merging skills because a demo chat looked fine. Measure contribution, then gate. A case that scores `1.00` with the plugin loaded can still be useless. Claude might already solve the prompt without your skill. The docs call this out: if `WITH` and `W/OUT` are both high, `Δ` near zero means the plugin did not move the outcome. Production implications: `Δ ≈ 0` with a `tool_used: Skill` grader failing — Claude never chose your skill on natural phrasing. Fix the skill `description`, re-run, compare.` tool_used` graders on `Skill` are excluded from both arms’ scored totals so you do not invent lift by checking something impossible without the plugin.`regex`, `tool_order`, and `file_exists` are free transcript/file checks. `llm` and `baseline` call a judge model and add to the run’s list-price estimate.`--runs 1 --ablation none`; trust needs the default three. If you only ever run the with-arm, you are grading absolute behavior. That is fine while drafting graders. Before you call a skill “done,” turn the baseline back on and read `Δ`. From the plugin root (the directory that contains the manifest): ``` claude --version # need >= 2.1.269 claude plugin eval init --bare commit-message ``` You get something shaped like: ``` evals/commit-message/ ├── prompt.md └── graders/ └── criteria.md ``` Edit `prompt.md` so the body is a request a user would type — do not name the skill in the prompt: ``` --- max_turns: 10 allowed_tools: [Read, Glob, Grep, Skill] --- Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites. ``` Add a result grader (`llm` or `regex`) and a skill-fired grader: ``` --- type: tool_used tool: Skill input_match: '"skill"\s*:\s*"(?:[\w-]+:)?your-skill-name"' --- ``` Then run: ``` claude plugin eval . ``` Expect six runs for one case at defaults (3× with, 3× without). Open the HTML report path printed at the end. Iterate on description and graders until `Δ` is positive for the prompts you care about. For MCP-backed skills, put mocks under `evals/mocks//.md` so CI does not need the real service. Use `--scaffold` only for suites you trust — scaffold scripts run as you, outside the agent sandbox. The docs’ recommended CI shape is explicit: ``` claude plugin eval . \ --trust-plugin \ --json results.json \ --threshold 0.8 \ --model claude-sonnet-5 \ --judge-model claude-haiku-4-5 \ --no-publish \ --max-cost-usd 20 ``` Exit codes that matter: `0` — every case met `1` — a case scored below threshold, files failed to load, or trust was missing without `--trust-plugin` `2` — partial run (`partial: true` Pin both models so a provider rollout is not mistaken for a plugin regression. Keep every-change suites on free graders when you can; reserve `llm` judges for short outputs with concrete PASS/FAIL rubrics. Leave `partial: true` results out of trend charts. Credentials and install still belong on the runner (`ANTHROPIC_API_KEY` or your normal Claude Code auth). Without `--trust-plugin`, a non-interactive job against an untrusted checkout exits `1`. OTF’s wedge is not “another sandboxed chat.” It is an owned repo agents keep editing — `CLAUDE.md`, prompts, and skills that travel with the product. Plugin eval is the missing acceptance gate for those skills: | Layer | Question | Artifact | |---|---|---| | Repo conventions | Can an agent find the seams? | `CLAUDE.md` /`.cursorrules` (see the kit docs post above) | | Change acceptance | Did this PR do what we asked? | [AI coding agent acceptance checklist](https://dev.to/blog/ai-agent-acceptance-checklist) | | Plugin contribution | Did the skill raise the score vs no plugin? | `claude plugin eval` +`Δ` | | Product AI features | Does the user-facing model path hold? | LLM evaluation loop | Put the suite next to the plugin you ship. Fail CI when `Δ` collapses after a description tweak or a model pin change. That is how skills stay production assets instead of chat folklore. Security note from the same docs: pointing `claude plugin eval` at a plugin is the same trust decision as `claude --plugin-dir`. Only evaluate plugins you trust. Hooks and real MCP servers you opt into with `--allow-real-servers` or `--mocks off` run outside the agent sandbox. If you are still choosing between a sandboxed MVP host and a repo you own, start with the kit spine, then add this gate. Skills without `Δ` are demos. Skills with a CI threshold are release machinery.