My README Promised Flags My CLI Doesn't Have. I Built a Drift Detector.

A stranger opened an issue on one of my small CLI tools last month: "The --watch flag in your README doesn't exist." They were right. I had removed

--watch

two releases earlier, rewritten the feature as a config option, updated the changelog, and forgotten the README's usage section entirely. Worse, the README still showed a fenced code block demonstrating the flag, so every new user was copy-pasting a command that errored out immediately.This is documentation drift, and it's embarrassingly common. Code changes continuously; prose changes when someone remembers. The twist is that my README also contained correct examples that had merely been reworded — so a naive string diff between "flags mentioned in the docs" and "flags in --help

output" produces a pile of false positives alongside the real bugs.

That combination — mostly mechanical matching, with a fuzzy residue that needs judgment — turned out to be a sweet spot for a free AI model. Not to write documentation (I don't trust generated docs), but to reconcile it: decide whether a prose claim is contradicted by actual CLI behavior.

I run the check on every release, so per-token pricing would be annoying, and the task is genuinely low-stakes: the deterministic part of the tool catches the clear violations, and the model only arbitrates ambiguous prose claims, which a human then reviews anyway.

I'm using MonkeyCode's free model access for the fuzzy-matching step — it exposes an OpenAI-compatible chat endpoint, which meant the script below needed no SDK beyond the standard library. If your docs describe internal tooling you can't send to a third party, they also have a free server option for self-hosting; the script treats the endpoint as configuration for exactly that reason.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two things a free model is not doing here: it is not the source of truth (the --help

output is), and it is not allowed to fail silently (every answer is validated against an allowlist before being reported).

driftcheck.py

The pipeline has three stages, and the order matters:

--help

output. Set difference, zero intelligence required.

#!/usr/bin/env python3
"""Detect drift between README claims and actual CLI behavior.

Usage: python3 driftcheck.py README.md -- ./mytool

Env vars:
    DRIFT_BASE_URL   OpenAI-compatible endpoint (e.g. MonkeyCode)
    DRIFT_MODEL      model name your provider currently exposes
"""
import json
import os
import re
import subprocess
import sys
import urllib.request

BASE_URL = os.environ["DRIFT_BASE_URL"].rstrip("/")
MODEL = os.environ["DRIFT_MODEL"]

FENCE = re.compile(r"```

(?:bash|sh|console)?\n(.*?)

```", re.DOTALL)
LONG_FLAG = re.compile(r"--[a-z][a-z0-9-]+")
HELP_FLAG = re.compile(r"^\s+(?:-[a-zA-Z],\s+)?(--[a-z][a-z0-9-]+)", re.MULTILINE)

def help_text(prog: str) -> str:
    out = subprocess.run([prog, "--help"], capture_output=True, text=True, timeout=15)
    return out.stdout + out.stderr

def claimed_flags(readme: str) -> set:
    flags = set()
    for block in FENCE.findall(readme):
        flags.update(LONG_FLAG.findall(block))
    return flags

def real_flags(help_out: str) -> set:
    return set(HELP_FLAG.findall(help_out))

def prose_claims(readme: str) -> list:
    """Sentences that assert capability but live outside code blocks."""
    body = FENCE.sub("", readme)
    verbs = ("supports", "can ", "allows", "automatically", "detects", "handles")
    return [s.strip() for line in body.splitlines() for s in re.split(r"(?<=[.!?]) ", line)
            if any(v in s.lower() for v in verbs) and len(s) < 200]

def reconcile(claim: str, help_out: str) -> str:
    """Ask the model; accept ONLY a one-word verdict from the allowlist."""
    prompt = (
        "CLI --help output follows:\n\n" + help_out[:12000] +
        "\n\nClaim from documentation: \"" + claim +
        "\"\n\nAnswer with exactly one word: SUPPORTED, CONTRADICTED, or UNCLEAR."
    )
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=90) as resp:
        verdict = json.load(resp)["choices"][0]["message"]["content"].strip().upper()
    for word in ("SUPPORTED", "CONTRADICTED", "UNCLEAR"):
        if word in verdict:
            return word
    return "UNCLEAR"  # never trust an unexpected answer; downgrade it

def main() -> None:
    readme_path, prog = sys.argv[1], sys.argv[3]
    readme = open(readme_path).read()
    help_out = help_text(prog)

    phantom = claimed_flags(readme) - real_flags(help_out)
    print("## Phantom flags (in README, not in --help)")
    for f in sorted(phantom):
        print(f"- `{f}`")

    print("\n## Prose claim reconciliation")
    for claim in prose_claims(readme):
        print(f"- [{reconcile(claim, help_out)}] {claim}")

    sys.exit(1 if phantom else 0)

if __name__ == "__main__":
    main()

Design choices worth stealing:

reconcile()

downgrades anything that isn't one of three words to UNCLEAR

. If the provider swaps the underlying model tomorrow, the worst case is more UNCLEAR

rows for a human to skim — never fabricated confidence.I ran this across my three public tools and hand-verified every report:

Two observations stood out. First, the "false alarms" for phantom flags were cases where the help text had the typo and the README was right — drift works both directions, which I hadn't considered. Second, the model's value concentrated entirely in the prose stage: it correctly flagged "automatically detects your config format" as contradicted after I'd removed auto-detection, something no regex could have caught.

--help

reveals.CONTRADICTED

means "read this sentence yourself," nothing more. About a quarter of its verdicts in my run needed correction.The useful reframe for me was treating the model as a reconciliation layer between two machine-readable-ish sources of truth, not as a writer. Docs vs. help output is one pair; the same pattern fits schema vs. example payloads, or changelog vs. actual exports. Wire an OpenAI-compatible endpoint behind two env vars — I pointed mine at MonkeyCode — keep the deterministic stage in charge of the exit code, and hand-verify one batch of reports before you trust any of them. If you build this for a different pair of sources, I'd genuinely like to hear which one in the comments.