{"slug": "claude-code-plugin-evals-read-the-delta-not-the-score", "title": "Claude Code Plugin Evals: Read the Delta, Not the Score", "summary": "Anthropic shipped the `claude plugin eval` command in Claude Code v2.1.269, a tool that scores plugins against test cases and reports a delta metric comparing results with and without the plugin loaded. Anthropic's documentation shows a healthy first run scoring 1.00 with the plugin versus 0.33 without it, a delta of +0.67, and warns that the most common first finding is a delta near zero with the `tool_used: Skill` grader failing, meaning Claude is not selecting the skill. Each case runs three times with the plugin and three times without, costing six runs per case, and the command offers six grader types — four free (`regex`, `tool_used`, `tool_order`, `file_exists`) and two paid (`llm`, `baseline`).", "body_md": "Most Claude Code plugin authors ship their first skill and assume it works because it installed. That assumption is usually wrong. The skill is loaded, the plugin is active, and Claude is silently ignoring it for every prompt that doesn’t match its description precisely. No errors. No warnings. Just a plugin that does nothing. Anthropic shipped `claude plugin eval` in v2.1.269 this week to surface exactly that failure — and the feature inside it called the delta metric is the thing you actually need to understand.\n\n## What Plugin Evals Do\n\n`claude plugin eval` runs your plugin against a suite of test cases and scores the results. Each case is a realistic user prompt plus one or more graders — checks on what Claude produced. A grader might verify Claude used the right tool, match a regex against the reply, or ask a judge model whether the answer met a rubric. The command outputs a scored summary and an HTML report.\n\nSetup is low-friction. Run `claude plugin eval init` from your plugin’s root directory, and Claude opens an interactive session: it reads your plugin, proposes test cases and graders, pilots them once to confirm they behave, and writes the files. You can also write cases manually, but the init path is faster for a first suite.\n\n## The Delta Metric: Read This Part\n\nHere’s where most developers will go wrong: they’ll look at the WITH score, see 1.0, and call it done. That score means nothing on its own.\n\nBy default, every test case runs twice — once with your plugin loaded and once without it. The difference between those two scores is `Δ` (delta). A Δ of +0.67 means your plugin raised the score by 0.67. A Δ near zero means the base model handled the task without your plugin — your skill is decoration.\n\nThe official docs show this result from a first run:\n\n```\nCASE        WITH  W/OUT Δ      RUNS COST\nfirst-case  1.00  0.33  +0.67  6    $0.41\n```\n\nThat’s a healthy result. The plugin is doing real work. But Anthropic is explicit about the most common first finding: a Δ near zero, with the `tool_used: Skill` grader failing. Translation: Claude isn’t picking your skill when users phrase things naturally. The fix is to update the skill’s `description` frontmatter, rerun, and compare the delta.\n\n## The 6 Grader Types\n\nOf the six grader types, four are free and two cost money:\n\n- **Free graders** (computed from transcripts and files):`regex` ,`tool_used` ,`tool_order` ,`file_exists`\n- **Paid graders** (judge model calls added to your bill):`llm` ,`baseline`\n\nThe recommended setup for most cases: one result-checking grader (llm or regex on the output) and one `tool_used: Skill` grader to confirm the skill actually fired. The `llm` grader takes a PASS/FAIL rubric in prose — keep rubrics concrete and use it only for short outputs. For generated files or long output, a `regex` grader over file contents is more stable than asking a judge model to read the whole thing.\n\nA minimal skill-fired grader looks like this:\n\n```\n---\ntype: tool_used\ntool: Skill\ninput_match: '\"skill\"\\s*:\\s*\"(?:[\\w-]+:)?your-skill-name\"'\n---\n```\n\n## Getting Started in Under 10 Minutes\n\nFrom your plugin’s root directory:\n\n```\nclaude plugin eval init\n```\n\nAnswer Claude’s questions about your plugin, exit when it says the suite is ready, then run:\n\n```\nclaude plugin eval .\n```\n\nEach case runs three times with the plugin and three times without, so one case costs six runs. You’ll see the summary table with WITH, W/OUT, and Δ columns, plus the location of the HTML report. Open the report to see per-run grader verdicts and the judge’s reasoning for any `llm` graders.\n\nWhen iterating on a specific case and don’t need the baseline comparison, halve the cost with:\n\n```\nclaude plugin eval . --case first-case --runs 1 --ablation none\n```\n\n## Locking It Into CI\n\nThe two-gate approach: run `claude plugin validate` on every commit (free, instant schema check), and run the full eval suite on release tags where you’re willing to pay for model calls.\n\nThe CI command Anthropic recommends:\n\n```\nclaude plugin eval . \\\n  --trust-plugin \\\n  --json results.json \\\n  --threshold 0.8 \\\n  --model claude-sonnet-5 \\\n  --judge-model claude-haiku-4-5 \\\n  --no-publish \\\n  --max-cost-usd 20\n```\n\nPin both `--model` and `--judge-model` explicitly. If you don’t, a model rollout will look like a plugin regression — and you’ll spend hours debugging something that isn’t broken.\n\nExit codes: `0` is a pass, `1` means a case fell below the threshold, `2` means your cost ceiling was hit mid-run. Set `--max-cost-usd` or you’ll have no upper bound on what a CI job spends.\n\n## One Thing to Get Right Before You Write Cases\n\nWrite prompts the way users actually phrase things — not the way you named the skill. “Generate a commit message for this diff” is better than “use the commit-message skill.” If your prompt names the skill, you’re testing whether Claude responds to explicit invocation, not whether the skill’s description is good enough to trigger on natural language.\n\nOne more gotcha: skills have a 1% context budget for their descriptions. If your plugin has many skills and descriptions overflow that budget, skills get silently dropped and will never fire. Check the Skills listing in the Claude Code UI and keep descriptions tight.\n\nThe complete reference lives in the [official plugin evals documentation](https://code.claude.com/docs/en/plugin-evals) — every grader option, CI flag, and mock server format is covered there. If you haven’t built a plugin yet, the [plugin creation guide](https://code.claude.com/docs/en/plugins) is the right place to start. For the full list of what else shipped in v2.1.269, [Matthew Wong’s practical walkthrough](https://www.matthewswong.com/en/blog/claude-code-plugin-eval-test-suite/) covers the eval workflow end-to-end with a real plugin example.", "url": "https://wpnews.pro/news/claude-code-plugin-evals-read-the-delta-not-the-score", "canonical_source": "https://byteiota.com/claude-code-plugin-evals-read-the-delta-not-the-score/", "published_at": "2026-09-12 06:09:17+00:00", "updated_at": "2026-09-12 06:28:17.280729+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-products"], "entities": ["Anthropic", "Claude Code", "claude plugin eval", "claude plugin validate"], "alternates": {"html": "https://wpnews.pro/news/claude-code-plugin-evals-read-the-delta-not-the-score", "markdown": "https://wpnews.pro/news/claude-code-plugin-evals-read-the-delta-not-the-score.md", "text": "https://wpnews.pro/news/claude-code-plugin-evals-read-the-delta-not-the-score.txt", "jsonld": "https://wpnews.pro/news/claude-code-plugin-evals-read-the-delta-not-the-score.jsonld"}}