# I measured what my Claude.md, skills and hooks are worth

> Source: <https://github.com/jameskomo/config-drift-checker>
> Published: 2026-08-27 18:12:32+00:00

**CI for your agent setup.** · [site](https://jameskomo.github.io/config-drift-checker/) · [demo dashboard](https://jameskomo.github.io/config-drift-checker-demo/) · [demo report](https://jameskomo.github.io/config-drift-checker-demo/report.html)

A tool that tells your team **when** its coding-agent setup — `CLAUDE.md`

, skills, hooks — stopped
doing what it should, **why**, and **what changed**: a model change behind an alias, a Claude Code
release, or a teammate's edit.

Your CLAUDE.md, skills and hooks are how your code gets written now. They are configuration that
other people change underneath you: Claude Code shipped 25 versions in the 30 days before this was
written, the model behind `sonnet`

changes server-side with no changelog, and a colleague can edit
a skill in a PR nobody tests. Today you find out when a developer notices "it stopped running the
tests before committing" — often weeks later.

config-drift-checker turns *what your setup must do* into test cases, runs the real agent against
them, and keeps score over time:

**Cases** in Anthropic's own`claude plugin eval`

format — a prompt, graders (regex, tool-use, file, LLM rubric), an optional scaffold that sets up a scratch repo or copies your real source.`/config-drift-checker:setup`

writes the first ones*from your actual setup*, so you don't start from a blank page.**Runs** on every Claude Code release (a watcher polls npm), on every PR that touches the setup, and on demand — in a throwaway workspace, with your plugin loaded, several times per case.**Scores and reasons**: every grader's verdict and the judge's explanation, the tool calls, the full response — not just a number.** Diff against your baseline**→ red or green check, PR comment, Slack alert, HTML report, and a dashboard of every case over every Claude Code version, served from your results branch by GitHub Pages.

Two things ride on top of that core:

**Ablation**— the same cases run*with*and*without*your plugin. The delta tells you what each skill or hook is actually worth: in our demo the guard hook is the only thing that reliably stops a destructive command, and a conventions skill turned out to add nothing the codebase and CLAUDE.md didn't already carry.**Generated cases**— the setup skill reads your configuration and writes real-code cases, negative-trigger cases and hook cases for it, then repairs its own graders until the smoke run passes.

**What it is not:** a linter for CLAUDE.md (it runs the real agent), a test of the model's general
quality (it tests *your* configuration on *your* tasks), or a hosted service (it runs on your
machine and your CI with your key; nothing is sent anywhere).

Works with any codebase — it tests the agent's behaviour, not your app. Uses the official runner
automatically where `claude plugin eval`

is enabled, a bundled runner otherwise.

```
claude plugin marketplace add jameskomo/config-drift-checker
claude plugin install config-drift-checker@jameskomo
```

Then, in the repo whose setup you want protected:

```
claude
> /config-drift-checker:setup
```

It finds your CLAUDE.md, skills and hooks, writes starter eval cases from them, smoke-runs them,
and writes the GitHub workflow. You add one secret (`ANTHROPIC_API_KEY`

) and push. From then on
the suite runs on every Claude Code release and every PR that touches your setup: red check, PR
comment, Slack alert, and an HTML report with every grader's reason.

Already have a suite, or just want the stage? One step:

```
- uses: jameskomo/config-drift-checker/action@v0
  with: { plugin-dir: . }
```

See it end to end on a small Spring Boot service: ** config-drift-checker-demo**.

```
config-drift-checker/     the plugin: skills (setup · run · write-case) and the tools they use
  tools/eval-shim.mjs       runs a suite via `claude -p` when the official runner is gated
  tools/eval-diff.mjs       baseline vs current → table, exit 1 on regression
  tools/eval-report.mjs     aggregate-result.json → self-contained HTML report
  tools/eval-dashboard.mjs  results history → dashboard (score per case over versions, run list)
  tools/release-watch.mjs   "did Claude Code publish a new version?"
  tools/safety-net.mjs      PreToolUse hook injected into every eval run
action/                   composite GitHub Action: install → run → diff → store → report → alert
examples/komo-stack/      an example plugin with a full eval suite and baseline results
docs/                     user guide · architecture · eval format & runner · runbook · security
```

Start with the [user guide](/jameskomo/config-drift-checker/blob/main/docs/user-guide.md). The full index is in [docs/](/jameskomo/config-drift-checker/blob/main/docs/README.md).

[FSL-1.1-Apache-2.0](/jameskomo/config-drift-checker/blob/main/LICENSE): free to use, modify and self-host; not to be offered as a competing
commercial service; each release becomes Apache-2.0 two years after publication.