Your SKILL.md is production config. Test it like one.

A developer released skilldiff, an open-source tool that tests agent skill files like SKILL.md for behavioral regressions by running them in a real agent harness against a fixture repo and asserting on observed files changed, commands run, and tool calls. The tool, which supports Claude Code, Cursor, Codex, and other harnesses, caught a regression where a revised note-taking skill still produced the expected output but also modified a forbidden TODO.md file that a text diff would have missed. CI mode replays recorded traces deterministically without an API key, and failures can gate merges.

You maintain agent skills — the SKILL.md instruction files that steer Claude Code, Cursor, Codex, or any coding agent. Careless edits feel cheap: rewrite one instruction line, done. The problem is what happens next. That one line silently changes how the agent behaves on the next run. Files you didn't ask it to touch. A tool call sequence that skips a step. A task that used to complete in one shot now stumbles through four attempts. Usually you find out weeks later. Not from a test — from a user. A git diff of SKILL.md shows you which words changed. It can't show you what the agent will do differently, because agent behavior is emergent. The same prompt line can produce completely different tool trajectories depending on context. Offline evals score a final answer against a gold label. They don't see the trajectory: the extra tool hop, the silent regression where the agent still prints the right string but took a forbidden shortcut to get there. Here's a real example. I dogfooded skilldiff on itself — it tests its own skills. Two versions of a note-taking skill were run against the same scenario: old skill: 2 tool calls, 1 file s changed, 0 command s run files: NOTES.md new skill: 3 tool calls, 2 file s changed, 0 command s run files: NOTES.md, TODO.md Assertions: ✓ files changed NOTES.md ✓ tool calls read ✓ tool calls write ✗ must not files changed does not include TODO.md actual new : VIOLATED — TODO.md was changed note: this is a REGRESSION — old skill passed, new skill fails Both versions appended SPIKE RAN OK to NOTES.md — the standard diff would look fine. But the new version also created TODO.md, which the scenario explicitly forbids. A text diff would never have caught that. That's behavioral regression. skilldiff runs your skill in a real agent harness against a fixture repo — twice: Then it captures what the agent actually did : files changed, commands run, tool calls made. And it asserts on those observations. | Kind | Meaning | |---|---| | files changed | paths the agent modified | | commands run | commands the agent executed | | tool calls | tools invoked normalized across harnesses | | must not | forbidden files / commands / tools | | output contains | substrings in the final output | A scenario looks like this: skill: notes-helper fixture: repo/notes-helper-scratch expect: files changed: NOTES.md tool calls: read, write must not: files changed: TODO.md Drop it into your PR flow npx skilldiff init discovers .claude/skills, skills/, .agents/skills and writes a starter scenario per skill npx skilldiff run skilldiff/notes-helper.scenario.yaml --live --base origin/main For CI it replays recorded traces, so it's deterministic and needs no API key or credits. The PR that touched a skill gets a comment with the behavior report; failures gate the merge. Live runs use whatever harness and login you already have. There's no shared API key shipped with the tool — each contributor runs on their own account opencode, Claude Code CLI, Cursor, Codex, Freebuff . Honest limitations v0.1 Skills are emergent, so a single run is a sample. Recorded/CI mode is deterministic; live runs vary run to run. Five assertion kinds is deliberately few. They cover the 80% case: files, commands, tools, forbidden behavior, output markers. This is not a replacement for full eval harnesses like Inspect or Promptfoo. It answers one question: did this change regress agent behavior I'm relying on? Contribute If your harness is missing, an adapter is roughly 40 lines — and the highest-value way to contribute. Repo: https://github.com/scs0209/skilldiff https://github.com/scs0209/skilldiff If you maintain skills and this doesn't match your workflow, I'd genuinely like to hear how you review skill changes today.