# My README Promised Flags My CLI Doesn't Have. I Built a Drift Detector. > Source: > Published: 2026-08-12 23:07:48+00:00 A stranger opened an issue on one of my small CLI tools last month: *"The --watch flag in your README doesn't exist."* They were right. I had removed `--watch` two releases earlier, rewritten the feature as a config option, updated the changelog, and forgotten the README's usage section entirely. Worse, the README still showed a fenced code block demonstrating the flag, so every new user was copy-pasting a command that errored out immediately.This is documentation drift, and it's embarrassingly common. Code changes continuously; prose changes when someone remembers. The twist is that my README also contained *correct* examples that had merely been reworded — so a naive string diff between "flags mentioned in the docs" and "flags in `--help` output" produces a pile of false positives alongside the real bugs. That combination — mostly mechanical matching, with a fuzzy residue that needs judgment — turned out to be a sweet spot for a free AI model. Not to write documentation (I don't trust generated docs), but to *reconcile* it: decide whether a prose claim is contradicted by actual CLI behavior. I run the check on every release, so per-token pricing would be annoying, and the task is genuinely low-stakes: the deterministic part of the tool catches the clear violations, and the model only arbitrates ambiguous prose claims, which a human then reviews anyway. I'm using MonkeyCode's free model access for the fuzzy-matching step — it exposes an OpenAI-compatible chat endpoint, which meant the script below needed no SDK beyond the standard library. If your docs describe internal tooling you can't send to a third party, they also have a free server option for self-hosting; the script treats the endpoint as configuration for exactly that reason. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Two things a free model is *not* doing here: it is not the source of truth (the `--help` output is), and it is not allowed to fail silently (every answer is validated against an allowlist before being reported). `driftcheck.py` The pipeline has three stages, and the order matters: `--help` output. Set difference, zero intelligence required. ``` bash #!/usr/bin/env python3 """Detect drift between README claims and actual CLI behavior. Usage: python3 driftcheck.py README.md -- ./mytool Env vars: DRIFT_BASE_URL OpenAI-compatible endpoint (e.g. MonkeyCode) DRIFT_MODEL model name your provider currently exposes """ import json import os import re import subprocess import sys import urllib.request BASE_URL = os.environ["DRIFT_BASE_URL"].rstrip("/") MODEL = os.environ["DRIFT_MODEL"] FENCE = re.compile(r"``` (?:bash|sh|console)?\n(.*?) ```", re.DOTALL) LONG_FLAG = re.compile(r"--[a-z][a-z0-9-]+") HELP_FLAG = re.compile(r"^\s+(?:-[a-zA-Z],\s+)?(--[a-z][a-z0-9-]+)", re.MULTILINE) def help_text(prog: str) -> str: out = subprocess.run([prog, "--help"], capture_output=True, text=True, timeout=15) return out.stdout + out.stderr def claimed_flags(readme: str) -> set: flags = set() for block in FENCE.findall(readme): flags.update(LONG_FLAG.findall(block)) return flags def real_flags(help_out: str) -> set: return set(HELP_FLAG.findall(help_out)) def prose_claims(readme: str) -> list: """Sentences that assert capability but live outside code blocks.""" body = FENCE.sub("", readme) verbs = ("supports", "can ", "allows", "automatically", "detects", "handles") return [s.strip() for line in body.splitlines() for s in re.split(r"(?<=[.!?]) ", line) if any(v in s.lower() for v in verbs) and len(s) < 200] def reconcile(claim: str, help_out: str) -> str: """Ask the model; accept ONLY a one-word verdict from the allowlist.""" prompt = ( "CLI --help output follows:\n\n" + help_out[:12000] + "\n\nClaim from documentation: \"" + claim + "\"\n\nAnswer with exactly one word: SUPPORTED, CONTRADICTED, or UNCLEAR." ) body = json.dumps({ "model": MODEL, "messages": [{"role": "user", "content": prompt}], "temperature": 0, }).encode() req = urllib.request.Request( f"{BASE_URL}/chat/completions", data=body, headers={"Content-Type": "application/json"}, ) with urllib.request.urlopen(req, timeout=90) as resp: verdict = json.load(resp)["choices"][0]["message"]["content"].strip().upper() for word in ("SUPPORTED", "CONTRADICTED", "UNCLEAR"): if word in verdict: return word return "UNCLEAR" # never trust an unexpected answer; downgrade it def main() -> None: readme_path, prog = sys.argv[1], sys.argv[3] readme = open(readme_path).read() help_out = help_text(prog) phantom = claimed_flags(readme) - real_flags(help_out) print("## Phantom flags (in README, not in --help)") for f in sorted(phantom): print(f"- `{f}`") print("\n## Prose claim reconciliation") for claim in prose_claims(readme): print(f"- [{reconcile(claim, help_out)}] {claim}") sys.exit(1 if phantom else 0) if __name__ == "__main__": main() ``` Design choices worth stealing: `reconcile()` downgrades anything that isn't one of three words to `UNCLEAR` . If the provider swaps the underlying model tomorrow, the worst case is more `UNCLEAR` rows for a human to skim — never fabricated confidence.I ran this across my three public tools and hand-verified every report: | Finding type | Reported | Real bugs | False alarms | |---|---|---|---| | Phantom flags | 9 | 7 | 2 (typos in `--help` itself — also bugs!) | | CONTRADICTED prose | 5 | 4 | 1 | | UNCLEAR prose | 11 | — (human-reviewed, 3 were real drift) | — | Two observations stood out. First, the "false alarms" for phantom flags were cases where the *help text* had the typo and the README was right — drift works both directions, which I hadn't considered. Second, the model's value concentrated entirely in the prose stage: it correctly flagged "automatically detects your config format" as contradicted after I'd removed auto-detection, something no regex could have caught. `--help` reveals.`CONTRADICTED` means "read this sentence yourself," nothing more. About a quarter of its verdicts in my run needed correction.The useful reframe for me was treating the model as a *reconciliation layer between two machine-readable-ish sources of truth*, not as a writer. Docs vs. help output is one pair; the same pattern fits schema vs. example payloads, or changelog vs. actual exports. Wire an OpenAI-compatible endpoint behind two env vars — I pointed mine at MonkeyCode — keep the deterministic stage in charge of the exit code, and hand-verify one batch of reports before you trust any of them. If you build this for a different pair of sources, I'd genuinely like to hear which one in the comments.