{"slug": "your-claude-md-has-zero-tests-here-s-what-happened-when-i-broke-mine-on-purpose", "title": "Your CLAUDE.md has zero tests. Here's what happened when I broke mine on purpose.", "summary": "Developer James Komo built config-drift-checker, a CI tool that turns CLAUDE.md files, skills, and hooks into eval cases to test them against every Claude Code release. He deliberately sabotaged his own setup to validate the tool, discovering that Claude Code auto-discovers the skills directory and that a broken trigger description caused a tripwire case to score zero across three runs, distinguishing genuine rule breakage from model randomness.", "body_md": "Claude Code shipped roughly 25 releases last month. My setup, 400 lines of `CLAUDE.md`, four skills and a guard hook, had exactly zero tests against any of them.\n\nThat bothered me, because this configuration has behavior, and behavior breaks. A new model version ships and suddenly a skill stops triggering. Nothing errors. Nothing goes red. The agent just quietly stops doing the thing you taught it, and you notice three weeks later when the code review comes back weird.\n\nSo I built CI for it: [config-drift-checker](https://github.com/jameskomo/config-drift-checker) turns your `CLAUDE.md`, skills and hooks into eval cases and re-runs them on every Claude Code release and every PR that touches the setup.\n\nBut a tester you've never seen fail is worthless. So I sabotaged my own setup to test the tester.\n\nI deleted the `skills` entry from my plugin manifest, expecting everything to break.\n\nNothing broke.\n\nClaude Code auto-discovers the skills directory, so the manifest key does nothing. My first sabotage was a no-op, which is exactly the kind of thing you only learn by trying to break your own system.\n\nThis time I rewrote one skill's trigger description the way a careless PR would. The skill still existed, still had all its content, but its description now talked about Terraform instead of Spring services.\n\n|  | Before | After sabotage | \n|---|---|---|\n| Suite score | 1.00 | **0.36** | \n| Tripwire case (did the skill fire?) | 1.00 | **0.00** | \n| Content cases | 1.00 | 0.33 to 0.60 | \n\nThe content cases dropped because the agent no longer followed conventions it used to follow. And one case went to exactly zero: the tripwire.\n\nA tripwire case has a single grader:\n\nWas the skill actually invoked?\n\nNot \"did the output look right\". Just \"did the trigger fire\". When a trigger breaks, that case reads 0 out of 3 runs, every run, every day, and no amount of model randomness produces that pattern.\n\nOne failed run doesn't mean your rule broke. Models are stochastic; sometimes they just ignore an instruction once. Telling \"rule stopped firing\" apart from \"model ignored it run\" turned out to be the real engineering problem.\n\n                                                                                                                                                                               Three things made the signal trustworthy:\n\n                                                                                                                                                                               1. **Three runs per case.** A genuine flake usually recovers within the same three runs. A broken trigger never does.\n\nLikely refusals (zero tool calls, one turn, short reply) get labeled separately, so they don't masquerade as broken rules.\n\nThe whole broken run is published, unedited:\n\n[The red report from the sabotage run](https://jameskomo.github.io/config-drift-checker/example-break/report.html)\n\nIt runs as a GitHub Action on your runner with your key:\n\n```\n- uses: jameskomo/config-drift-checker/action@v0\n  with:\n    plugin-dir: .\n```\n\nCosts $0 on a Claude Pro or Max plan via a subscription token.\n\nRepo: [https://github.com/jameskomo/config-drift-checker](https://github.com/jameskomo/config-drift-checker)\n\nIf you've built your own version of this in-house (I keep meeting people who have), I'd genuinely like to compare notes, especially on how you separate \"rule stopped firing\" from \"model ignored it this run\".", "url": "https://wpnews.pro/news/your-claude-md-has-zero-tests-here-s-what-happened-when-i-broke-mine-on-purpose", "canonical_source": "https://dev.to/smartword_world_0b5b38003/your-claudemd-has-zero-tests-heres-what-happened-when-i-broke-mine-on-purpose-58mp", "published_at": "2026-09-08 10:03:18+00:00", "updated_at": "2026-09-08 10:33:09.116248+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "mlops"], "entities": ["James Komo", "config-drift-checker", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/your-claude-md-has-zero-tests-here-s-what-happened-when-i-broke-mine-on-purpose", "markdown": "https://wpnews.pro/news/your-claude-md-has-zero-tests-here-s-what-happened-when-i-broke-mine-on-purpose.md", "text": "https://wpnews.pro/news/your-claude-md-has-zero-tests-here-s-what-happened-when-i-broke-mine-on-purpose.txt", "jsonld": "https://wpnews.pro/news/your-claude-md-has-zero-tests-here-s-what-happened-when-i-broke-mine-on-purpose.jsonld"}}