{"slug": "anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-a", "title": "Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills", "summary": "Anthropic published a plugin evals workflow for Claude Code that runs the `claude plugin eval` command to score a plugin against realistic prompts and compare results with a run where the plugin is not loaded. The workflow ships 6 grader types — regex, tool_used, tool_order, file_exists, llm, and baseline — of which the two judge-model graders add to the bill, and it requires Claude Code v2.1.269 or later. Anthropic reports the key metric is Δ, the difference between the with-arm and without-arm scores, and warns that a Δ near zero with a failing `tool_used: Skill` grader means the skill never triggers on natural phrasing.", "body_md": "Anthropic has published a new [plugin evals workflow for Claude Code](https://code.claude.com/docs/en/plugin-evals). The `claude plugin eval` command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run where the plugin is not loaded. It answers 3 questions plugin developers could not previously measure: does the skill trigger, does it survive an edit or a new model, and does it beat a bare model.\n\n**Deployable:** Yes. It runs on Claude Code v2.1.269 or later against any directory with a `plugin.json` or `.claude-plugin/plugin.json` manifest, or a skills-directory plugin. Every eval run and judge grader is a real model call billed to your plan or API account.\n\n## **What a case looks like**\n\nAn eval suite lives in an `evals/` directory inside the plugin. Each case is a subdirectory holding a `prompt.md` and a `graders/` folder. The prompt body goes to Claude exactly as written, and `@path` mentions are not expanded. Frontmatter on `prompt.md` can set `max_turns` (default 10), `timeout_seconds` (default 300), `model`, `tags`, and `allowed_tools`.\n\nGraders are markdown files whose frontmatter sets a `type`, an optional `weight`, and an optional `arm`. There are 6 types. Four cost nothing because they are computed from the transcript and the files on disk: `regex`, `tool_used`, `tool_order`, and `file_exists`. Two call a judge model and add to the bill: `llm`, which scores the reply against prose criteria you write, and `baseline`, which compares it against a reference answer.\n\n`claude plugin eval init` reads the plugin, asks what a good result looks like, proposes cases and graders, tries them, and writes the files. In CI, `--bare <name>` writes a blank template instead.\n\n## **The number that matters is Δ**\n\nBy default every case runs twice: a with-arm where the plugin is loaded and a without-arm where it is not. Their difference, Δ, is what the plugin contributed. If a case scores 1.0 in both arms, the plugin is not why it passed. The docs example output shows a single case at WITH 1.00, W/OUT 0.33, Δ +0.67 across 6 runs, costing an estimated $0.41 and taking 74 seconds. A grader marked with-only, typically `tool_used: Skill`, is reported as an indicator and excluded from the score, since the without-arm has no skill to fire.\n\nAnthropic calls out the most common first finding: a Δ near zero with the `tool_used: Skill` grader failing, which means Claude is not choosing the skill on natural phrasing. That is the defect [`claude plugin validate`](https://code.claude.com/docs/en/plugins) cannot see, because it checks manifest syntax and schema rather than behavior.\n\nResults land under `evals/results/<timestamp>/report.html` with per-grader verdicts and judge votes. Where the account supports it, the report is also published to claude.ai unless `--no-publish` is set.\n\n## **Cost and CI**\n\nA suite makes roughly cases × runs × arms agent runs, plus 3 short judge calls per `llm` or `baseline` grader per run, and results vary between runs. The documented CI invocation is:\n\n```\nclaude plugin eval . \\\n  --trust-plugin \\\n  --json results.json \\\n  --threshold 0.8 \\\n  --model claude-sonnet-5 \\\n  --judge-model claude-haiku-4-5 \\\n  --no-publish \\\n  --max-cost-usd 20\n```\n\nThe runner needs a Claude Code install and credentials such as `ANTHROPIC_API_KEY`. Without `--trust-plugin`, an untrusted checkout is refused with exit 1 when there is no terminal. Report problems never change the exit code, and `--json` suppresses progress output.\n\n## **Interactive explainer**\n\n## **Key Takeaways**\n\n- `claude plugin eval` scores realistic prompts with 6 grader types; 4 are free,`llm` and`baseline` bill a judge model.\n- Every case runs with and without the plugin by default; Δ is the only number that proves the plugin did the work.\n- A Δ near zero with a failing `tool_used: Skill` grader means the skill never triggers on natural phrasing.\n- `--threshold` ,`--max-cost-usd` , and`--trust-plugin` turn it into a CI gate; usage-limit errors can fake a regression.\n- Requires Claude Code v2.1.269+; `claude plugin eval init` writes the first suite for you.\n\nCheck out the **[Technical details](https://code.claude.com/docs/en/plugin-evals)**. Also, feel free to follow us on **[Twitter](https://x.com/intent/follow?screen_name=marktechpost)** and don’t forget to join our **[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)** and Subscribe to **[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})**. Wait! are you on telegram? [now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)\n\nNeed to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)", "url": "https://wpnews.pro/news/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-a", "canonical_source": "https://www.marktechpost.com/2026/09/11/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-and-a-ci-gate-for-skills/", "published_at": "2026-09-11 21:05:55+00:00", "updated_at": "2026-09-11 21:24:06.120446+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "developer-tools", "ai-agents"], "entities": ["Anthropic", "Claude Code", "claude plugin eval", "claude plugin validate", "claude.ai", "ANTHROPIC_API_KEY", "claude-sonnet-5", "claude-haiku-4-5"], "alternates": {"html": "https://wpnews.pro/news/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-a", "markdown": "https://wpnews.pro/news/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-a.md", "text": "https://wpnews.pro/news/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-a.txt", "jsonld": "https://wpnews.pro/news/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-a.jsonld"}}