Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills Anthropic published a plugin evals workflow for Claude Code that runs the `claude plugin eval` command to score a plugin against realistic prompts and compare results with a run where the plugin is not loaded. The workflow ships 6 grader types — regex, tool_used, tool_order, file_exists, llm, and baseline — of which the two judge-model graders add to the bill, and it requires Claude Code v2.1.269 or later. Anthropic reports the key metric is Δ, the difference between the with-arm and without-arm scores, and warns that a Δ near zero with a failing `tool_used: Skill` grader means the skill never triggers on natural phrasing. Anthropic has published a new plugin evals workflow for Claude Code https://code.claude.com/docs/en/plugin-evals . The claude plugin eval command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run where the plugin is not loaded. It answers 3 questions plugin developers could not previously measure: does the skill trigger, does it survive an edit or a new model, and does it beat a bare model. Deployable: Yes. It runs on Claude Code v2.1.269 or later against any directory with a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin. Every eval run and judge grader is a real model call billed to your plan or API account. What a case looks like An eval suite lives in an evals/ directory inside the plugin. Each case is a subdirectory holding a prompt.md and a graders/ folder. The prompt body goes to Claude exactly as written, and @path mentions are not expanded. Frontmatter on prompt.md can set max turns default 10 , timeout seconds default 300 , model , tags , and allowed tools . Graders are markdown files whose frontmatter sets a type , an optional weight , and an optional arm . There are 6 types. Four cost nothing because they are computed from the transcript and the files on disk: regex , tool used , tool order , and file exists . Two call a judge model and add to the bill: llm , which scores the reply against prose criteria you write, and baseline , which compares it against a reference answer. claude plugin eval init reads the plugin, asks what a good result looks like, proposes cases and graders, tries them, and writes the files. In CI, --bare