{"slug": "my-readme-promised-flags-my-cli-doesn-t-have-i-built-a-drift-detector", "title": "My README Promised Flags My CLI Doesn't Have. I Built a Drift Detector.", "summary": "A developer built a drift detector to catch mismatches between README documentation and actual CLI flags, using a free AI model from MonkeyCode to reconcile ambiguous prose claims. The tool combines deterministic flag extraction with model-based arbitration, validating every answer against an allowlist to avoid silent failures.", "body_md": "A stranger opened an issue on one of my small CLI tools last month: *\"The --watch flag in your README doesn't exist.\"* They were right. I had removed\n\n`--watch`\n\ntwo releases earlier, rewritten the feature as a config option, updated the changelog, and forgotten the README's usage section entirely. Worse, the README still showed a fenced code block demonstrating the flag, so every new user was copy-pasting a command that errored out immediately.This is documentation drift, and it's embarrassingly common. Code changes continuously; prose changes when someone remembers. The twist is that my README also contained *correct* examples that had merely been reworded — so a naive string diff between \"flags mentioned in the docs\" and \"flags in `--help`\n\noutput\" produces a pile of false positives alongside the real bugs.\n\nThat combination — mostly mechanical matching, with a fuzzy residue that needs judgment — turned out to be a sweet spot for a free AI model. Not to write documentation (I don't trust generated docs), but to *reconcile* it: decide whether a prose claim is contradicted by actual CLI behavior.\n\nI run the check on every release, so per-token pricing would be annoying, and the task is genuinely low-stakes: the deterministic part of the tool catches the clear violations, and the model only arbitrates ambiguous prose claims, which a human then reviews anyway.\n\nI'm using MonkeyCode's free model access for the fuzzy-matching step — it exposes an OpenAI-compatible chat endpoint, which meant the script below needed no SDK beyond the standard library. If your docs describe internal tooling you can't send to a third party, they also have a free server option for self-hosting; the script treats the endpoint as configuration for exactly that reason.\n\nDisclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nTwo things a free model is *not* doing here: it is not the source of truth (the `--help`\n\noutput is), and it is not allowed to fail silently (every answer is validated against an allowlist before being reported).\n\n`driftcheck.py`\n\nThe pipeline has three stages, and the order matters:\n\n`--help`\n\noutput. Set difference, zero intelligence required.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Detect drift between README claims and actual CLI behavior.\n\nUsage: python3 driftcheck.py README.md -- ./mytool\n\nEnv vars:\n    DRIFT_BASE_URL   OpenAI-compatible endpoint (e.g. MonkeyCode)\n    DRIFT_MODEL      model name your provider currently exposes\n\"\"\"\nimport json\nimport os\nimport re\nimport subprocess\nimport sys\nimport urllib.request\n\nBASE_URL = os.environ[\"DRIFT_BASE_URL\"].rstrip(\"/\")\nMODEL = os.environ[\"DRIFT_MODEL\"]\n\nFENCE = re.compile(r\"```\n\n(?:bash|sh|console)?\\n(.*?)\n\n```\", re.DOTALL)\nLONG_FLAG = re.compile(r\"--[a-z][a-z0-9-]+\")\nHELP_FLAG = re.compile(r\"^\\s+(?:-[a-zA-Z],\\s+)?(--[a-z][a-z0-9-]+)\", re.MULTILINE)\n\ndef help_text(prog: str) -> str:\n    out = subprocess.run([prog, \"--help\"], capture_output=True, text=True, timeout=15)\n    return out.stdout + out.stderr\n\ndef claimed_flags(readme: str) -> set:\n    flags = set()\n    for block in FENCE.findall(readme):\n        flags.update(LONG_FLAG.findall(block))\n    return flags\n\ndef real_flags(help_out: str) -> set:\n    return set(HELP_FLAG.findall(help_out))\n\ndef prose_claims(readme: str) -> list:\n    \"\"\"Sentences that assert capability but live outside code blocks.\"\"\"\n    body = FENCE.sub(\"\", readme)\n    verbs = (\"supports\", \"can \", \"allows\", \"automatically\", \"detects\", \"handles\")\n    return [s.strip() for line in body.splitlines() for s in re.split(r\"(?<=[.!?]) \", line)\n            if any(v in s.lower() for v in verbs) and len(s) < 200]\n\ndef reconcile(claim: str, help_out: str) -> str:\n    \"\"\"Ask the model; accept ONLY a one-word verdict from the allowlist.\"\"\"\n    prompt = (\n        \"CLI --help output follows:\\n\\n\" + help_out[:12000] +\n        \"\\n\\nClaim from documentation: \\\"\" + claim +\n        \"\\\"\\n\\nAnswer with exactly one word: SUPPORTED, CONTRADICTED, or UNCLEAR.\"\n    )\n    body = json.dumps({\n        \"model\": MODEL,\n        \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n        \"temperature\": 0,\n    }).encode()\n    req = urllib.request.Request(\n        f\"{BASE_URL}/chat/completions\",\n        data=body,\n        headers={\"Content-Type\": \"application/json\"},\n    )\n    with urllib.request.urlopen(req, timeout=90) as resp:\n        verdict = json.load(resp)[\"choices\"][0][\"message\"][\"content\"].strip().upper()\n    for word in (\"SUPPORTED\", \"CONTRADICTED\", \"UNCLEAR\"):\n        if word in verdict:\n            return word\n    return \"UNCLEAR\"  # never trust an unexpected answer; downgrade it\n\ndef main() -> None:\n    readme_path, prog = sys.argv[1], sys.argv[3]\n    readme = open(readme_path).read()\n    help_out = help_text(prog)\n\n    phantom = claimed_flags(readme) - real_flags(help_out)\n    print(\"## Phantom flags (in README, not in --help)\")\n    for f in sorted(phantom):\n        print(f\"- `{f}`\")\n\n    print(\"\\n## Prose claim reconciliation\")\n    for claim in prose_claims(readme):\n        print(f\"- [{reconcile(claim, help_out)}] {claim}\")\n\n    sys.exit(1 if phantom else 0)\n\nif __name__ == \"__main__\":\n    main()\n```\n\nDesign choices worth stealing:\n\n`reconcile()`\n\ndowngrades anything that isn't one of three words to `UNCLEAR`\n\n. If the provider swaps the underlying model tomorrow, the worst case is more `UNCLEAR`\n\nrows for a human to skim — never fabricated confidence.I ran this across my three public tools and hand-verified every report:\n\n| Finding type | Reported | Real bugs | False alarms |\n|---|---|---|---|\n| Phantom flags | 9 | 7 | 2 (typos in `--help` itself — also bugs!) |\n| CONTRADICTED prose | 5 | 4 | 1 |\n| UNCLEAR prose | 11 | — (human-reviewed, 3 were real drift) | — |\n\nTwo observations stood out. First, the \"false alarms\" for phantom flags were cases where the *help text* had the typo and the README was right — drift works both directions, which I hadn't considered. Second, the model's value concentrated entirely in the prose stage: it correctly flagged \"automatically detects your config format\" as contradicted after I'd removed auto-detection, something no regex could have caught.\n\n`--help`\n\nreveals.`CONTRADICTED`\n\nmeans \"read this sentence yourself,\" nothing more. About a quarter of its verdicts in my run needed correction.The useful reframe for me was treating the model as a *reconciliation layer between two machine-readable-ish sources of truth*, not as a writer. Docs vs. help output is one pair; the same pattern fits schema vs. example payloads, or changelog vs. actual exports. Wire an OpenAI-compatible endpoint behind two env vars — I pointed mine at MonkeyCode — keep the deterministic stage in charge of the exit code, and hand-verify one batch of reports before you trust any of them. If you build this for a different pair of sources, I'd genuinely like to hear which one in the comments.", "url": "https://wpnews.pro/news/my-readme-promised-flags-my-cli-doesn-t-have-i-built-a-drift-detector", "canonical_source": "https://dev.to/datacpp_8185/my-readme-promised-flags-my-cli-doesnt-have-i-built-a-drift-detector-25jl", "published_at": "2026-08-12 23:07:48+00:00", "updated_at": "2026-08-12 23:46:08.205997+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "natural-language-processing"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/my-readme-promised-flags-my-cli-doesn-t-have-i-built-a-drift-detector", "markdown": "https://wpnews.pro/news/my-readme-promised-flags-my-cli-doesn-t-have-i-built-a-drift-detector.md", "text": "https://wpnews.pro/news/my-readme-promised-flags-my-cli-doesn-t-have-i-built-a-drift-detector.txt", "jsonld": "https://wpnews.pro/news/my-readme-promised-flags-my-cli-doesn-t-have-i-built-a-drift-detector.jsonld"}}