{"slug": "i-measured-what-my-claude-md-skills-and-hooks-are-worth", "title": "I measured what my Claude.md, skills and hooks are worth", "summary": "James Komo released config-drift-checker, an open-source tool that tests coding-agent setups (CLAUDE.md, skills, hooks) against real agent runs to detect when they stop working, citing that Claude Code shipped 25 versions in 30 days and model changes can occur without changelogs. The tool generates eval cases from existing configuration, runs them on every Claude Code release and PR, and provides scores, diffs, and alerts via GitHub Actions, with a demo showing a guard hook reliably stops destructive commands while a conventions skill added no value.", "body_md": "**CI for your agent setup.** · [site](https://jameskomo.github.io/config-drift-checker/) · [demo dashboard](https://jameskomo.github.io/config-drift-checker-demo/) · [demo report](https://jameskomo.github.io/config-drift-checker-demo/report.html)\n\nA tool that tells your team **when** its coding-agent setup — `CLAUDE.md`\n\n, skills, hooks — stopped\ndoing what it should, **why**, and **what changed**: a model change behind an alias, a Claude Code\nrelease, or a teammate's edit.\n\nYour CLAUDE.md, skills and hooks are how your code gets written now. They are configuration that\nother people change underneath you: Claude Code shipped 25 versions in the 30 days before this was\nwritten, the model behind `sonnet`\n\nchanges server-side with no changelog, and a colleague can edit\na skill in a PR nobody tests. Today you find out when a developer notices \"it stopped running the\ntests before committing\" — often weeks later.\n\nconfig-drift-checker turns *what your setup must do* into test cases, runs the real agent against\nthem, and keeps score over time:\n\n**Cases** in Anthropic's own`claude plugin eval`\n\nformat — a prompt, graders (regex, tool-use, file, LLM rubric), an optional scaffold that sets up a scratch repo or copies your real source.`/config-drift-checker:setup`\n\nwrites the first ones*from your actual setup*, so you don't start from a blank page.**Runs** on every Claude Code release (a watcher polls npm), on every PR that touches the setup, and on demand — in a throwaway workspace, with your plugin loaded, several times per case.**Scores and reasons**: every grader's verdict and the judge's explanation, the tool calls, the full response — not just a number.** Diff against your baseline**→ red or green check, PR comment, Slack alert, HTML report, and a dashboard of every case over every Claude Code version, served from your results branch by GitHub Pages.\n\nTwo things ride on top of that core:\n\n**Ablation**— the same cases run*with*and*without*your plugin. The delta tells you what each skill or hook is actually worth: in our demo the guard hook is the only thing that reliably stops a destructive command, and a conventions skill turned out to add nothing the codebase and CLAUDE.md didn't already carry.**Generated cases**— the setup skill reads your configuration and writes real-code cases, negative-trigger cases and hook cases for it, then repairs its own graders until the smoke run passes.\n\n**What it is not:** a linter for CLAUDE.md (it runs the real agent), a test of the model's general\nquality (it tests *your* configuration on *your* tasks), or a hosted service (it runs on your\nmachine and your CI with your key; nothing is sent anywhere).\n\nWorks with any codebase — it tests the agent's behaviour, not your app. Uses the official runner\nautomatically where `claude plugin eval`\n\nis enabled, a bundled runner otherwise.\n\n```\nclaude plugin marketplace add jameskomo/config-drift-checker\nclaude plugin install config-drift-checker@jameskomo\n```\n\nThen, in the repo whose setup you want protected:\n\n```\nclaude\n> /config-drift-checker:setup\n```\n\nIt finds your CLAUDE.md, skills and hooks, writes starter eval cases from them, smoke-runs them,\nand writes the GitHub workflow. You add one secret (`ANTHROPIC_API_KEY`\n\n) and push. From then on\nthe suite runs on every Claude Code release and every PR that touches your setup: red check, PR\ncomment, Slack alert, and an HTML report with every grader's reason.\n\nAlready have a suite, or just want the stage? One step:\n\n```\n- uses: jameskomo/config-drift-checker/action@v0\n  with: { plugin-dir: . }\n```\n\nSee it end to end on a small Spring Boot service: ** config-drift-checker-demo**.\n\n```\nconfig-drift-checker/     the plugin: skills (setup · run · write-case) and the tools they use\n  tools/eval-shim.mjs       runs a suite via `claude -p` when the official runner is gated\n  tools/eval-diff.mjs       baseline vs current → table, exit 1 on regression\n  tools/eval-report.mjs     aggregate-result.json → self-contained HTML report\n  tools/eval-dashboard.mjs  results history → dashboard (score per case over versions, run list)\n  tools/release-watch.mjs   \"did Claude Code publish a new version?\"\n  tools/safety-net.mjs      PreToolUse hook injected into every eval run\naction/                   composite GitHub Action: install → run → diff → store → report → alert\nexamples/komo-stack/      an example plugin with a full eval suite and baseline results\ndocs/                     user guide · architecture · eval format & runner · runbook · security\n```\n\nStart with the [user guide](/jameskomo/config-drift-checker/blob/main/docs/user-guide.md). The full index is in [docs/](/jameskomo/config-drift-checker/blob/main/docs/README.md).\n\n[FSL-1.1-Apache-2.0](/jameskomo/config-drift-checker/blob/main/LICENSE): free to use, modify and self-host; not to be offered as a competing\ncommercial service; each release becomes Apache-2.0 two years after publication.", "url": "https://wpnews.pro/news/i-measured-what-my-claude-md-skills-and-hooks-are-worth", "canonical_source": "https://github.com/jameskomo/config-drift-checker", "published_at": "2026-08-27 18:12:32+00:00", "updated_at": "2026-08-27 18:50:18.242871+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "mlops", "ai-tools"], "entities": ["James Komo", "config-drift-checker", "Claude Code", "Anthropic", "GitHub Pages", "GitHub Actions", "Slack"], "alternates": {"html": "https://wpnews.pro/news/i-measured-what-my-claude-md-skills-and-hooks-are-worth", "markdown": "https://wpnews.pro/news/i-measured-what-my-claude-md-skills-and-hooks-are-worth.md", "text": "https://wpnews.pro/news/i-measured-what-my-claude-md-skills-and-hooks-are-worth.txt", "jsonld": "https://wpnews.pro/news/i-measured-what-my-claude-md-skills-and-hooks-are-worth.jsonld"}}