Claude Code Plugin Evals: Read the Delta, Not the Score

Anthropic shipped the `claude plugin eval` command in Claude Code v2.1.269, a tool that scores plugins against test cases and reports a delta metric comparing results with and without the plugin loaded. Anthropic's documentation shows a healthy first run scoring 1.00 with the plugin versus 0.33 without it, a delta of +0.67, and warns that the most common first finding is a delta near zero with the `tool_used: Skill` grader failing, meaning Claude is not selecting the skill. Each case runs three times with the plugin and three times without, costing six runs per case, and the command offers six grader types — four free (`regex`, `tool_used`, `tool_order`, `file_exists`) and two paid (`llm`, `baseline`).

Most Claude Code plugin authors ship their first skill and assume it works because it installed. That assumption is usually wrong. The skill is loaded, the plugin is active, and Claude is silently ignoring it for every prompt that doesn’t match its description precisely. No errors. No warnings. Just a plugin that does nothing. Anthropic shipped claude plugin eval in v2.1.269 this week to surface exactly that failure — and the feature inside it called the delta metric is the thing you actually need to understand. What Plugin Evals Do claude plugin eval runs your plugin against a suite of test cases and scores the results. Each case is a realistic user prompt plus one or more graders — checks on what Claude produced. A grader might verify Claude used the right tool, match a regex against the reply, or ask a judge model whether the answer met a rubric. The command outputs a scored summary and an HTML report. Setup is low-friction. Run claude plugin eval init from your plugin’s root directory, and Claude opens an interactive session: it reads your plugin, proposes test cases and graders, pilots them once to confirm they behave, and writes the files. You can also write cases manually, but the init path is faster for a first suite. The Delta Metric: Read This Part Here’s where most developers will go wrong: they’ll look at the WITH score, see 1.0, and call it done. That score means nothing on its own. By default, every test case runs twice — once with your plugin loaded and once without it. The difference between those two scores is Δ delta . A Δ of +0.67 means your plugin raised the score by 0.67. A Δ near zero means the base model handled the task without your plugin — your skill is decoration. The official docs show this result from a first run: CASE WITH W/OUT Δ RUNS COST first-case 1.00 0.33 +0.67 6 $0.41 That’s a healthy result. The plugin is doing real work. But Anthropic is explicit about the most common first finding: a Δ near zero, with the tool used: Skill grader failing. Translation: Claude isn’t picking your skill when users phrase things naturally. The fix is to update the skill’s description frontmatter, rerun, and compare the delta. The 6 Grader Types Of the six grader types, four are free and two cost money: - Free graders computed from transcripts and files : regex , tool used , tool order , file exists - Paid graders judge model calls added to your bill : llm , baseline The recommended setup for most cases: one result-checking grader llm or regex on the output and one tool used: Skill grader to confirm the skill actually fired. The llm grader takes a PASS/FAIL rubric in prose — keep rubrics concrete and use it only for short outputs. For generated files or long output, a regex grader over file contents is more stable than asking a judge model to read the whole thing. A minimal skill-fired grader looks like this: --- type: tool used tool: Skill input match: '"skill"\s :\s " ?: \w- +: ?your-skill-name"' --- Getting Started in Under 10 Minutes From your plugin’s root directory: claude plugin eval init Answer Claude’s questions about your plugin, exit when it says the suite is ready, then run: claude plugin eval . Each case runs three times with the plugin and three times without, so one case costs six runs. You’ll see the summary table with WITH, W/OUT, and Δ columns, plus the location of the HTML report. Open the report to see per-run grader verdicts and the judge’s reasoning for any llm graders. When iterating on a specific case and don’t need the baseline comparison, halve the cost with: claude plugin eval . --case first-case --runs 1 --ablation none Locking It Into CI The two-gate approach: run claude plugin validate on every commit free, instant schema check , and run the full eval suite on release tags where you’re willing to pay for model calls. The CI command Anthropic recommends: claude plugin eval . \ --trust-plugin \ --json results.json \ --threshold 0.8 \ --model claude-sonnet-5 \ --judge-model claude-haiku-4-5 \ --no-publish \ --max-cost-usd 20 Pin both --model and --judge-model explicitly. If you don’t, a model rollout will look like a plugin regression — and you’ll spend hours debugging something that isn’t broken. Exit codes: 0 is a pass, 1 means a case fell below the threshold, 2 means your cost ceiling was hit mid-run. Set --max-cost-usd or you’ll have no upper bound on what a CI job spends. One Thing to Get Right Before You Write Cases Write prompts the way users actually phrase things — not the way you named the skill. “Generate a commit message for this diff” is better than “use the commit-message skill.” If your prompt names the skill, you’re testing whether Claude responds to explicit invocation, not whether the skill’s description is good enough to trigger on natural language. One more gotcha: skills have a 1% context budget for their descriptions. If your plugin has many skills and descriptions overflow that budget, skills get silently dropped and will never fire. Check the Skills listing in the Claude Code UI and keep descriptions tight. The complete reference lives in the official plugin evals documentation https://code.claude.com/docs/en/plugin-evals — every grader option, CI flag, and mock server format is covered there. If you haven’t built a plugin yet, the plugin creation guide https://code.claude.com/docs/en/plugins is the right place to start. For the full list of what else shipped in v2.1.269, Matthew Wong’s practical walkthrough https://www.matthewswong.com/en/blog/claude-code-plugin-eval-test-suite/ covers the eval workflow end-to-end with a real plugin example.