A skill that works when you try it has passed one test, graded by you, once.
For months, this is how I tested my Claude Code skills. Open a sample project, create a clean branch, run the skill, read what it wrote. If it looked right, ship it.
That isn't nothing. It's a real test with a human grader: you set up the project, you write the prompt, you judge the result. The problem is everything around it:
The example I use throughout this series is twd-ai, the Claude Code plugin for TWD, a browser testing library I maintain. It has a skill that sets the library up in a project, one that writes the CI workflow, and one that writes, runs and fixes tests. Nothing in these articles depends on it: swap in your own plugin and every step applies.
An eval is a test for an agent's behaviour. Instead of checking that a function returns a value, it checks that Claude, with your skill loaded, does what the skill promises. It runs for real: Claude reads files, writes code and runs commands, and when it's done, a set of checks scores the result.
Claude Code ships one: claude plugin eval. Each case has three parts:
A trimmed case from my suite:
name: setup-vite-react-query
context:
scaffold_script: scaffold.sh
execution:
allowed_tools: [Read, Glob, Grep, Skill]
prompt: /twd:setup
graders:
- name: vite-plugin-imported
type: regex
pattern: 'from\s+["'']twd-js/vite-plugin["'']'
target: { source: file, path: vite.config.ts }
- name: no-entry-file-code
type: regex
pattern: 'initTWD|twd-js'
match: not_contains
target: { source: file, path: src/main.tsx }
- name: patterns-file-attempted
type: tool_used
tool: Write
input_match: '\.claude/twd-patterns\.md'
The last grader reads the agent's actions, not the disk: eval runs refuse writes into .claude/, so it checks that the skill tried to write its notes file there.
The scaffold writes a small Vite + React app with heredocs. It installs nothing, so the case needs no network.
Graders come in two families. Some check what was written: a pattern is in a file, or isn't. Others check what the agent did: it called a tool, with this input, before that other call. For what a regex can't decide, there's an llm grader: a judge model reads the file against a criterion you write and votes pass or fail, three times.
This is the part I didn't expect to matter most. Every case runs twice: with your plugin and without it. The difference, Δ, is what your skill adds on top of what the model already knows.
My setup case on Opus:
| Score | |
|---|---|
| With the plugin | 1.00 |
| Without the plugin | 0.10 |
| Δ | +0.90 |
Without the skill, the model reads the same project files, says the setup command "isn't installed in this session", and does it by hand from what it knows. It wires the library into the app's entry file, which the library no longer needs, and never writes the config file.
A Δ near zero doesn't mean a case is useless. It means the model already knows how to do it. The case still guards against an edit to the skill that breaks it: if that happens, the score with the plugin drops.
Because an agent doesn't behave the same every time, each case runs three times and the score is the average. A run's score is the share of graders that passed. --threshold 0.9 decides what counts as a pass: it tolerates a grader failing once in three runs, not one that fails every time.
claude plugin eval . --scaffold --runs 1 --ablation none --allow-tools Write Edit
claude plugin eval . --scaffold --threshold 0.9 \
--model claude-opus-5-5 --judge-model claude-sonnet-5-5 --allow-tools Write Edit
The full suite, eight cases on Opus, costs about $10 and 25 minutes. The quick mode is under $2.
The eval sandbox has no browser and no network, and one of my skills runs tests in a browser. So the scaffold writes a stand-in test runner: it honours the filter flag, writes a report in the same format as the real one, and fails a test that mocks a URL the app never requests.
It's the same idea as mocking an API in a test. You're testing how the skill reacts to the tool, not the tool. The real runner has its own tests in its own repo.
A failing grader is fine: it tells you something. The dangerous one passes when it shouldn't, because nobody looks at a green check. It happened to me three times:
beforeEach\([\s\S]*?twd\.clearComponentMocks was meant to check that mocks are cleared in beforeEach. The lazy match kept going into the afterEach below it and passed a file that cleared them in the wrong place.
Before trusting a new grader, run it against a file that should pass and one that should fail.
I still run my skills by hand on real projects. The eval doesn't replace that; it tells me when to. When a case drops after an edit, or after a model release, I know exactly where to look.
The scenarios twd-ai guards, and their scores per model, are on twd.dev.
Next in this series: reading the trace, the file that turns a score into a reason.