# Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills

> Source: <https://www.marktechpost.com/2026/09/11/anthropic-adds-plugin-evals-to-claude-code-6-grader-types-a-no-plugin-baseline-and-a-ci-gate-for-skills/>
> Published: 2026-09-11 21:05:55+00:00

Anthropic has published a new [plugin evals workflow for Claude Code](https://code.claude.com/docs/en/plugin-evals). The `claude plugin eval` command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run where the plugin is not loaded. It answers 3 questions plugin developers could not previously measure: does the skill trigger, does it survive an edit or a new model, and does it beat a bare model.

**Deployable:** Yes. It runs on Claude Code v2.1.269 or later against any directory with a `plugin.json` or `.claude-plugin/plugin.json` manifest, or a skills-directory plugin. Every eval run and judge grader is a real model call billed to your plan or API account.

## **What a case looks like**

An eval suite lives in an `evals/` directory inside the plugin. Each case is a subdirectory holding a `prompt.md` and a `graders/` folder. The prompt body goes to Claude exactly as written, and `@path` mentions are not expanded. Frontmatter on `prompt.md` can set `max_turns` (default 10), `timeout_seconds` (default 300), `model`, `tags`, and `allowed_tools`.

Graders are markdown files whose frontmatter sets a `type`, an optional `weight`, and an optional `arm`. There are 6 types. Four cost nothing because they are computed from the transcript and the files on disk: `regex`, `tool_used`, `tool_order`, and `file_exists`. Two call a judge model and add to the bill: `llm`, which scores the reply against prose criteria you write, and `baseline`, which compares it against a reference answer.

`claude plugin eval init` reads the plugin, asks what a good result looks like, proposes cases and graders, tries them, and writes the files. In CI, `--bare <name>` writes a blank template instead.

## **The number that matters is Δ**

By default every case runs twice: a with-arm where the plugin is loaded and a without-arm where it is not. Their difference, Δ, is what the plugin contributed. If a case scores 1.0 in both arms, the plugin is not why it passed. The docs example output shows a single case at WITH 1.00, W/OUT 0.33, Δ +0.67 across 6 runs, costing an estimated $0.41 and taking 74 seconds. A grader marked with-only, typically `tool_used: Skill`, is reported as an indicator and excluded from the score, since the without-arm has no skill to fire.

Anthropic calls out the most common first finding: a Δ near zero with the `tool_used: Skill` grader failing, which means Claude is not choosing the skill on natural phrasing. That is the defect [`claude plugin validate`](https://code.claude.com/docs/en/plugins) cannot see, because it checks manifest syntax and schema rather than behavior.

Results land under `evals/results/<timestamp>/report.html` with per-grader verdicts and judge votes. Where the account supports it, the report is also published to claude.ai unless `--no-publish` is set.

## **Cost and CI**

A suite makes roughly cases × runs × arms agent runs, plus 3 short judge calls per `llm` or `baseline` grader per run, and results vary between runs. The documented CI invocation is:

```
claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20
```

The runner needs a Claude Code install and credentials such as `ANTHROPIC_API_KEY`. Without `--trust-plugin`, an untrusted checkout is refused with exit 1 when there is no terminal. Report problems never change the exit code, and `--json` suppresses progress output.

## **Interactive explainer**

## **Key Takeaways**

- `claude plugin eval` scores realistic prompts with 6 grader types; 4 are free,`llm` and`baseline` bill a judge model.
- Every case runs with and without the plugin by default; Δ is the only number that proves the plugin did the work.
- A Δ near zero with a failing `tool_used: Skill` grader means the skill never triggers on natural phrasing.
- `--threshold` ,`--max-cost-usd` , and`--trust-plugin` turn it into a CI gate; usage-limit errors can fake a regression.
- Requires Claude Code v2.1.269+; `claude plugin eval init` writes the first suite for you.

Check out the **[Technical details](https://code.claude.com/docs/en/plugin-evals)**. Also, feel free to follow us on **[Twitter](https://x.com/intent/follow?screen_name=marktechpost)** and don’t forget to join our **[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)** and Subscribe to **[our Newsletter](https://magic.beehiiv.com/v1/f5e63dd4-5653-4f09-83e2-321a8b1ba526?email={{email}})**. Wait! are you on telegram? [now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)