{"slug": "i-tried-it-and-it-worked-is-not-a-test-for-your-claude-code-skills", "title": "\"I tried it and it worked\" is not a test for your Claude Code skills", "summary": "A developer maintaining the twd-ai Claude Code plugin described building an eval suite with Claude Code's `claude plugin eval` command to test agent skills rather than relying on manual spot checks. Each case pairs a scaffolded project and prompt with regex, tool-use and LLM graders, and runs both with and without the plugin so the delta measures what the skill adds; the setup case scored 1.00 with the plugin versus 0.10 without on Opus, a +0.90 delta. The full eight-case suite runs about $10 and 25 minutes on Opus, with a quick mode under $2.", "body_md": "A skill that works when you try it has passed one test, graded by you, once.\n\nFor months, this is how I tested my Claude Code skills. Open a sample project, create a clean branch, run the skill, read what it wrote. If it looked right, ship it.\n\nThat isn't nothing. It's a real test with a human grader: you set up the project, you write the prompt, you judge the result. The problem is everything around it:\n\nThe example I use throughout this series is [twd-ai](https://github.com/BRIKEV/twd-ai), the Claude Code plugin for [TWD](https://twd.dev), a browser testing library I maintain. It has a skill that sets the library up in a project, one that writes the CI workflow, and one that writes, runs and fixes tests. Nothing in these articles depends on it: swap in your own plugin and every step applies.\n\nAn eval is a test for an agent's behaviour. Instead of checking that a function returns a value, it checks that Claude, with your skill loaded, does what the skill promises. It runs for real: Claude reads files, writes code and runs commands, and when it's done, a set of checks scores the result.\n\nClaude Code ships one: [`claude plugin eval`](https://code.claude.com/docs/en/plugin-evals). Each case has three parts:\n\nA trimmed case from my suite:\n\n```\nname: setup-vite-react-query\ncontext:\n  scaffold_script: scaffold.sh\nexecution:\n  allowed_tools: [Read, Glob, Grep, Skill]\n  prompt: /twd:setup\ngraders:\n  - name: vite-plugin-imported\n    type: regex\n    pattern: 'from\\s+[\"'']twd-js/vite-plugin[\"'']'\n    target: { source: file, path: vite.config.ts }\n  - name: no-entry-file-code\n    type: regex\n    pattern: 'initTWD|twd-js'\n    match: not_contains\n    target: { source: file, path: src/main.tsx }\n  - name: patterns-file-attempted\n    type: tool_used\n    tool: Write\n    input_match: '\\.claude/twd-patterns\\.md'\n```\n\nThe last grader reads the agent's actions, not the disk: eval runs refuse writes into `.claude/`, so it checks that the skill tried to write its notes file there.\n\nThe scaffold writes a small Vite + React app with heredocs. It installs nothing, so the case needs no network.\n\nGraders come in two families. Some check **what was written**: a pattern is in a file, or isn't. Others check **what the agent did**: it called a tool, with this input, before that other call. For what a regex can't decide, there's an `llm` grader: a judge model reads the file against a criterion you write and votes pass or fail, three times.\n\nThis is the part I didn't expect to matter most. Every case runs twice: **with** your plugin and **without** it. The difference, Δ, is what your skill adds on top of what the model already knows.\n\nMy setup case on Opus:\n\n|  | Score | \n|---|---|\n| With the plugin | 1.00 | \n| Without the plugin | 0.10 | \n| Δ | +0.90 | \n\nWithout the skill, the model reads the same project files, says the setup command \"isn't installed in this session\", and does it by hand from what it knows. It wires the library into the app's entry file, which the library no longer needs, and never writes the config file.\n\nA Δ near zero doesn't mean a case is useless. It means the model already knows how to do it. The case still guards against an edit to the skill that breaks it: if that happens, the score with the plugin drops.\n\nBecause an agent doesn't behave the same every time, each case runs three times and the score is the average. A run's score is the share of graders that passed. `--threshold 0.9` decides what counts as a pass: it tolerates a grader failing once in three runs, not one that fails every time.\n\n```\n# While editing a skill: one run, with the plugin only\nclaude plugin eval . --scaffold --runs 1 --ablation none --allow-tools Write Edit\n\n# Before a release: three runs, with and without the plugin\nclaude plugin eval . --scaffold --threshold 0.9 \\\n  --model claude-opus-5-5 --judge-model claude-sonnet-5-5 --allow-tools Write Edit\n```\n\nThe full suite, eight cases on Opus, costs about $10 and 25 minutes. The quick mode is under $2.\n\nThe eval sandbox has no browser and no network, and one of my skills runs tests in a browser. So the scaffold writes a stand-in test runner: it honours the filter flag, writes a report in the same format as the real one, and fails a test that mocks a URL the app never requests.\n\nIt's the same idea as mocking an API in a test. You're testing how the skill reacts to the tool, not the tool. The real runner has its own tests in its own repo.\n\nA failing grader is fine: it tells you something. The dangerous one passes when it shouldn't, because nobody looks at a green check. It happened to me three times:\n\n`beforeEach\\([\\s\\S]*?twd\\.clearComponentMocks` was meant to check that mocks are cleared in `beforeEach`. The lazy match kept going into the `afterEach` below it and passed a file that cleared them in the wrong place.\nBefore trusting a new grader, run it against a file that should pass and one that should fail.\n\nI still run my skills by hand on real projects. The eval doesn't replace that; it tells me when to. When a case drops after an edit, or after a model release, I know exactly where to look.\n\nThe scenarios twd-ai guards, and their scores per model, are on [twd.dev](https://twd.dev/twd-ai/evals).\n\nNext in this series: reading the trace, the file that turns a score into a reason.", "url": "https://wpnews.pro/news/i-tried-it-and-it-worked-is-not-a-test-for-your-claude-code-skills", "canonical_source": "https://dev.to/kevinccbsg/i-tried-it-and-it-worked-is-not-a-test-for-your-claude-code-skills-2h39", "published_at": "2026-10-29 09:11:13+00:00", "updated_at": "2026-10-29 09:16:55.932587+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Claude Code", "Anthropic", "twd-ai", "TWD", "Opus", "Sonnet"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-tried-it-and-it-worked-is-not-a-test-for-your-claude-code-skills", "markdown": "https://wpnews.pro/news/i-tried-it-and-it-worked-is-not-a-test-for-your-claude-code-skills.md", "text": "https://wpnews.pro/news/i-tried-it-and-it-worked-is-not-a-test-for-your-claude-code-skills.txt", "jsonld": "https://wpnews.pro/news/i-tried-it-and-it-worked-is-not-a-test-for-your-claude-code-skills.jsonld"}}