I measured what my Claude.md, skills and hooks are worth

James Komo released config-drift-checker, an open-source tool that tests coding-agent setups (CLAUDE.md, skills, hooks) against real agent runs to detect when they stop working, citing that Claude Code shipped 25 versions in 30 days and model changes can occur without changelogs. The tool generates eval cases from existing configuration, runs them on every Claude Code release and PR, and provides scores, diffs, and alerts via GitHub Actions, with a demo showing a guard hook reliably stops destructive commands while a conventions skill added no value.

CI for your agent setup. · site https://jameskomo.github.io/config-drift-checker/ · demo dashboard https://jameskomo.github.io/config-drift-checker-demo/ · demo report https://jameskomo.github.io/config-drift-checker-demo/report.html A tool that tells your team when its coding-agent setup — CLAUDE.md , skills, hooks — stopped doing what it should, why , and what changed : a model change behind an alias, a Claude Code release, or a teammate's edit. Your CLAUDE.md, skills and hooks are how your code gets written now. They are configuration that other people change underneath you: Claude Code shipped 25 versions in the 30 days before this was written, the model behind sonnet changes server-side with no changelog, and a colleague can edit a skill in a PR nobody tests. Today you find out when a developer notices "it stopped running the tests before committing" — often weeks later. config-drift-checker turns what your setup must do into test cases, runs the real agent against them, and keeps score over time: Cases in Anthropic's own claude plugin eval format — a prompt, graders regex, tool-use, file, LLM rubric , an optional scaffold that sets up a scratch repo or copies your real source. /config-drift-checker:setup writes the first ones from your actual setup , so you don't start from a blank page. Runs on every Claude Code release a watcher polls npm , on every PR that touches the setup, and on demand — in a throwaway workspace, with your plugin loaded, several times per case. Scores and reasons : every grader's verdict and the judge's explanation, the tool calls, the full response — not just a number. Diff against your baseline → red or green check, PR comment, Slack alert, HTML report, and a dashboard of every case over every Claude Code version, served from your results branch by GitHub Pages. Two things ride on top of that core: Ablation — the same cases run with and without your plugin. The delta tells you what each skill or hook is actually worth: in our demo the guard hook is the only thing that reliably stops a destructive command, and a conventions skill turned out to add nothing the codebase and CLAUDE.md didn't already carry. Generated cases — the setup skill reads your configuration and writes real-code cases, negative-trigger cases and hook cases for it, then repairs its own graders until the smoke run passes. What it is not: a linter for CLAUDE.md it runs the real agent , a test of the model's general quality it tests your configuration on your tasks , or a hosted service it runs on your machine and your CI with your key; nothing is sent anywhere . Works with any codebase — it tests the agent's behaviour, not your app. Uses the official runner automatically where claude plugin eval is enabled, a bundled runner otherwise. claude plugin marketplace add jameskomo/config-drift-checker claude plugin install config-drift-checker@jameskomo Then, in the repo whose setup you want protected: claude /config-drift-checker:setup It finds your CLAUDE.md, skills and hooks, writes starter eval cases from them, smoke-runs them, and writes the GitHub workflow. You add one secret ANTHROPIC API KEY and push. From then on the suite runs on every Claude Code release and every PR that touches your setup: red check, PR comment, Slack alert, and an HTML report with every grader's reason. Already have a suite, or just want the stage? One step: - uses: jameskomo/config-drift-checker/action@v0 with: { plugin-dir: . } See it end to end on a small Spring Boot service: config-drift-checker-demo . config-drift-checker/ the plugin: skills setup · run · write-case and the tools they use tools/eval-shim.mjs runs a suite via claude -p when the official runner is gated tools/eval-diff.mjs baseline vs current → table, exit 1 on regression tools/eval-report.mjs aggregate-result.json → self-contained HTML report tools/eval-dashboard.mjs results history → dashboard score per case over versions, run list tools/release-watch.mjs "did Claude Code publish a new version?" tools/safety-net.mjs PreToolUse hook injected into every eval run action/ composite GitHub Action: install → run → diff → store → report → alert examples/komo-stack/ an example plugin with a full eval suite and baseline results docs/ user guide · architecture · eval format & runner · runbook · security Start with the user guide /jameskomo/config-drift-checker/blob/main/docs/user-guide.md . The full index is in docs/ /jameskomo/config-drift-checker/blob/main/docs/README.md . FSL-1.1-Apache-2.0 /jameskomo/config-drift-checker/blob/main/LICENSE : free to use, modify and self-host; not to be offered as a competing commercial service; each release becomes Apache-2.0 two years after publication.