{"slug": "your-skill-md-is-production-config-test-it-like-one", "title": "Your SKILL.md is production config. Test it like one.", "summary": "A developer released skilldiff, an open-source tool that tests agent skill files like SKILL.md for behavioral regressions by running them in a real agent harness against a fixture repo and asserting on observed files changed, commands run, and tool calls. The tool, which supports Claude Code, Cursor, Codex, and other harnesses, caught a regression where a revised note-taking skill still produced the expected output but also modified a forbidden TODO.md file that a text diff would have missed. CI mode replays recorded traces deterministically without an API key, and failures can gate merges.", "body_md": "You maintain agent skills — the SKILL.md instruction files that steer Claude Code, Cursor, Codex, or any coding agent. Careless edits feel cheap: rewrite one instruction line, done.\n\nThe problem is what happens next. That one line silently changes how the agent behaves on the next run. Files you didn't ask it to touch. A tool call sequence that skips a step. A task that used to complete in one shot now stumbles through four attempts.\n\nUsually you find out weeks later. Not from a test — from a user.\n\nA git diff of SKILL.md shows you which words changed. It can't show you what the *agent* will do differently, because agent behavior is emergent. The same prompt line can produce completely different tool trajectories depending on context.\n\nOffline evals score a final answer against a gold label. They don't see the trajectory: the extra tool hop, the silent regression where the agent still prints the right string but took a forbidden shortcut to get there.\n\nHere's a real example. I dogfooded skilldiff on itself — it tests its own skills. Two versions of a note-taking skill were run against the same scenario:\n\nold skill: 2 tool calls, 1 file(s) changed, 0 command(s) run files: NOTES.md\n\nnew skill: 3 tool calls, 2 file(s) changed, 0 command(s) run files: NOTES.md, TODO.md\n\nAssertions: ✓ [files_changed] NOTES.md ✓ [tool_calls] read ✓ [tool_calls] write ✗ [must_not] files_changed does not include TODO.md actual (new): VIOLATED — TODO.md was changed note: this is a REGRESSION — old skill passed, new skill fails\n\nBoth versions appended `SPIKE RAN OK` to NOTES.md — the standard diff would look fine. But the new version *also* created TODO.md, which the scenario explicitly forbids. A text diff would never have caught that. That's behavioral regression.\n\nskilldiff runs your skill in a real agent harness against a fixture repo — twice:\n\nThen it captures what the agent *actually did*: files changed, commands run, tool calls made. And it asserts on those observations.\n\n| Kind | Meaning | \n|---|---|\n| `files_changed` | paths the agent modified | \n| `commands_run` | commands the agent executed | \n| `tool_calls` | tools invoked (normalized across harnesses) | \n| `must_not` | forbidden files / commands / tools | \n| `output_contains` | substrings in the final output | \n\nA scenario looks like this:\n\n```\nskill: notes-helper\nfixture: repo/notes-helper-scratch\nexpect:\n  files_changed: [NOTES.md]\n  tool_calls: [read, write]\n  must_not:\n    files_changed: [TODO.md]\nDrop it into your PR flow\nnpx skilldiff init   # discovers .claude/skills, skills/, .agents/skills\n                     # and writes a starter scenario per skill\n\nnpx skilldiff run skilldiff/notes-helper.scenario.yaml --live --base origin/main\nFor CI it replays recorded traces, so it's deterministic and needs no API key or credits. The PR that touched a skill gets a comment with the behavior report; failures gate the merge.\n\nLive runs use whatever harness and login you already have. There's no shared API key shipped with the tool — each contributor runs on their own account (opencode, Claude Code CLI, Cursor, Codex, Freebuff).\n\nHonest limitations (v0.1)\nSkills are emergent, so a single run is a sample. Recorded/CI mode is deterministic; live runs vary run to run.\nFive assertion kinds is deliberately few. They cover the 80% case: files, commands, tools, forbidden behavior, output markers.\nThis is not a replacement for full eval harnesses like Inspect or Promptfoo. It answers one question: did this change regress agent behavior I'm relying on?\nContribute\nIf your harness is missing, an adapter is roughly 40 lines — and the highest-value way to contribute.\n```\n\nRepo: [https://github.com/scs0209/skilldiff](https://github.com/scs0209/skilldiff)\n\nIf you maintain skills and this doesn't match your workflow, I'd genuinely like to hear how you review skill changes today.", "url": "https://wpnews.pro/news/your-skill-md-is-production-config-test-it-like-one", "canonical_source": "https://dev.to/scs0209/your-skillmd-is-production-config-test-it-like-one-d91", "published_at": "2026-09-18 11:06:59+00:00", "updated_at": "2026-09-18 11:22:51.184461+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "mlops"], "entities": ["skilldiff", "Claude Code", "Cursor", "Codex", "GitHub", "Inspect", "Promptfoo", "opencode"], "alternates": {"html": "https://wpnews.pro/news/your-skill-md-is-production-config-test-it-like-one", "markdown": "https://wpnews.pro/news/your-skill-md-is-production-config-test-it-like-one.md", "text": "https://wpnews.pro/news/your-skill-md-is-production-config-test-it-like-one.txt", "jsonld": "https://wpnews.pro/news/your-skill-md-is-production-config-test-it-like-one.jsonld"}}