You maintain agent skills — the SKILL.md instruction files that steer Claude Code, Cursor, Codex, or any coding agent. Careless edits feel cheap: rewrite one instruction line, done.
The problem is what happens next. That one line silently changes how the agent behaves on the next run. Files you didn't ask it to touch. A tool call sequence that skips a step. A task that used to complete in one shot now stumbles through four attempts.
Usually you find out weeks later. Not from a test — from a user.
A git diff of SKILL.md shows you which words changed. It can't show you what the agent will do differently, because agent behavior is emergent. The same prompt line can produce completely different tool trajectories depending on context.
Offline evals score a final answer against a gold label. They don't see the trajectory: the extra tool hop, the silent regression where the agent still prints the right string but took a forbidden shortcut to get there.
Here's a real example. I dogfooded skilldiff on itself — it tests its own skills. Two versions of a note-taking skill were run against the same scenario:
old skill: 2 tool calls, 1 file(s) changed, 0 command(s) run files: NOTES.md
new skill: 3 tool calls, 2 file(s) changed, 0 command(s) run files: NOTES.md, TODO.md
Assertions: ✓ [files_changed] NOTES.md ✓ [tool_calls] read ✓ [tool_calls] write ✗ [must_not] files_changed does not include TODO.md actual (new): VIOLATED — TODO.md was changed note: this is a REGRESSION — old skill passed, new skill fails
Both versions appended SPIKE RAN OK to NOTES.md — the standard diff would look fine. But the new version also created TODO.md, which the scenario explicitly forbids. A text diff would never have caught that. That's behavioral regression.
skilldiff runs your skill in a real agent harness against a fixture repo — twice:
Then it captures what the agent actually did: files changed, commands run, tool calls made. And it asserts on those observations.
| Kind | Meaning |
|---|---|
files_changed |
paths the agent modified |
commands_run |
commands the agent executed |
tool_calls |
tools invoked (normalized across harnesses) |
must_not |
forbidden files / commands / tools |
output_contains |
substrings in the final output |
A scenario looks like this:
skill: notes-helper
fixture: repo/notes-helper-scratch
expect:
files_changed: [NOTES.md]
tool_calls: [read, write]
must_not:
files_changed: [TODO.md]
Drop it into your PR flow
npx skilldiff init # discovers .claude/skills, skills/, .agents/skills
npx skilldiff run skilldiff/notes-helper.scenario.yaml --live --base origin/main
For CI it replays recorded traces, so it's deterministic and needs no API key or credits. The PR that touched a skill gets a comment with the behavior report; failures gate the merge.
Live runs use whatever harness and login you already have. There's no shared API key shipped with the tool — each contributor runs on their own account (opencode, Claude Code CLI, Cursor, Codex, Freebuff).
Honest limitations (v0.1)
Skills are emergent, so a single run is a sample. Recorded/CI mode is deterministic; live runs vary run to run.
Five assertion kinds is deliberately few. They cover the 80% case: files, commands, tools, forbidden behavior, output markers.
This is not a replacement for full eval harnesses like Inspect or Promptfoo. It answers one question: did this change regress agent behavior I'm relying on?
Contribute
If your harness is missing, an adapter is roughly 40 lines — and the highest-value way to contribute.
Repo: https://github.com/scs0209/skilldiff
If you maintain skills and this doesn't match your workflow, I'd genuinely like to hear how you review skill changes today.