My README Promised Flags My CLI Doesn't Have. I Built a Drift Detector.

A developer built a drift detector to catch mismatches between README documentation and actual CLI flags, using a free AI model from MonkeyCode to reconcile ambiguous prose claims. The tool combines deterministic flag extraction with model-based arbitration, validating every answer against an allowlist to avoid silent failures.

A stranger opened an issue on one of my small CLI tools last month: "The --watch flag in your README doesn't exist." They were right. I had removed --watch two releases earlier, rewritten the feature as a config option, updated the changelog, and forgotten the README's usage section entirely. Worse, the README still showed a fenced code block demonstrating the flag, so every new user was copy-pasting a command that errored out immediately.This is documentation drift, and it's embarrassingly common. Code changes continuously; prose changes when someone remembers. The twist is that my README also contained correct examples that had merely been reworded — so a naive string diff between "flags mentioned in the docs" and "flags in --help output" produces a pile of false positives alongside the real bugs. That combination — mostly mechanical matching, with a fuzzy residue that needs judgment — turned out to be a sweet spot for a free AI model. Not to write documentation I don't trust generated docs , but to reconcile it: decide whether a prose claim is contradicted by actual CLI behavior. I run the check on every release, so per-token pricing would be annoying, and the task is genuinely low-stakes: the deterministic part of the tool catches the clear violations, and the model only arbitrates ambiguous prose claims, which a human then reviews anyway. I'm using MonkeyCode's free model access for the fuzzy-matching step — it exposes an OpenAI-compatible chat endpoint, which meant the script below needed no SDK beyond the standard library. If your docs describe internal tooling you can't send to a third party, they also have a free server option for self-hosting; the script treats the endpoint as configuration for exactly that reason. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two things a free model is not doing here: it is not the source of truth the --help output is , and it is not allowed to fail silently every answer is validated against an allowlist before being reported . driftcheck.py The pipeline has three stages, and the order matters: --help output. Set difference, zero intelligence required. bash /usr/bin/env python3 """Detect drift between README claims and actual CLI behavior. Usage: python3 driftcheck.py README.md -- ./mytool Env vars: DRIFT BASE URL OpenAI-compatible endpoint e.g. MonkeyCode DRIFT MODEL model name your provider currently exposes """ import json import os import re import subprocess import sys import urllib.request BASE URL = os.environ "DRIFT BASE URL" .rstrip "/" MODEL = os.environ "DRIFT MODEL" FENCE = re.compile r" ?:bash|sh|console ?\n . ? ", re.DOTALL LONG FLAG = re.compile r"-- a-z a-z0-9- +" HELP FLAG = re.compile r"^\s+ ?:- a-zA-Z ,\s+ ? -- a-z a-z0-9- + ", re.MULTILINE def help text prog: str - str: out = subprocess.run prog, "--help" , capture output=True, text=True, timeout=15 return out.stdout + out.stderr def claimed flags readme: str - set: flags = set for block in FENCE.findall readme : flags.update LONG FLAG.findall block return flags def real flags help out: str - set: return set HELP FLAG.findall help out def prose claims readme: str - list: """Sentences that assert capability but live outside code blocks.""" body = FENCE.sub "", readme verbs = "supports", "can ", "allows", "automatically", "detects", "handles" return s.strip for line in body.splitlines for s in re.split r" ?<= . ? ", line if any v in s.lower for v in verbs and len s < 200 def reconcile claim: str, help out: str - str: """Ask the model; accept ONLY a one-word verdict from the allowlist.""" prompt = "CLI --help output follows:\n\n" + help out :12000 + "\n\nClaim from documentation: \"" + claim + "\"\n\nAnswer with exactly one word: SUPPORTED, CONTRADICTED, or UNCLEAR." body = json.dumps { "model": MODEL, "messages": {"role": "user", "content": prompt} , "temperature": 0, } .encode req = urllib.request.Request f"{BASE URL}/chat/completions", data=body, headers={"Content-Type": "application/json"}, with urllib.request.urlopen req, timeout=90 as resp: verdict = json.load resp "choices" 0 "message" "content" .strip .upper for word in "SUPPORTED", "CONTRADICTED", "UNCLEAR" : if word in verdict: return word return "UNCLEAR" never trust an unexpected answer; downgrade it def main - None: readme path, prog = sys.argv 1 , sys.argv 3 readme = open readme path .read help out = help text prog phantom = claimed flags readme - real flags help out print " Phantom flags in README, not in --help " for f in sorted phantom : print f"- {f} " print "\n Prose claim reconciliation" for claim in prose claims readme : print f"- {reconcile claim, help out } {claim}" sys.exit 1 if phantom else 0 if name == " main ": main Design choices worth stealing: reconcile downgrades anything that isn't one of three words to UNCLEAR . If the provider swaps the underlying model tomorrow, the worst case is more UNCLEAR rows for a human to skim — never fabricated confidence.I ran this across my three public tools and hand-verified every report: | Finding type | Reported | Real bugs | False alarms | |---|---|---|---| | Phantom flags | 9 | 7 | 2 typos in --help itself — also bugs | | CONTRADICTED prose | 5 | 4 | 1 | | UNCLEAR prose | 11 | — human-reviewed, 3 were real drift | — | Two observations stood out. First, the "false alarms" for phantom flags were cases where the help text had the typo and the README was right — drift works both directions, which I hadn't considered. Second, the model's value concentrated entirely in the prose stage: it correctly flagged "automatically detects your config format" as contradicted after I'd removed auto-detection, something no regex could have caught. --help reveals. CONTRADICTED means "read this sentence yourself," nothing more. About a quarter of its verdicts in my run needed correction.The useful reframe for me was treating the model as a reconciliation layer between two machine-readable-ish sources of truth , not as a writer. Docs vs. help output is one pair; the same pattern fits schema vs. example payloads, or changelog vs. actual exports. Wire an OpenAI-compatible endpoint behind two env vars — I pointed mine at MonkeyCode — keep the deterministic stage in charge of the exit code, and hand-verify one batch of reports before you trust any of them. If you build this for a different pair of sources, I'd genuinely like to hear which one in the comments.