The Model May Draft the Prose. The Repo Must Confirm the Facts.

A doc said the retry limit defaults to 5. The code said 3. The env var in the doc had been renamed two releases earlier, so anyone who copy-pasted the snippet got a silent no-op instead of an error.

Nothing about that draft looked wrong. The sentences were clean, the structure was better than what we had, and the reviewer approved it in four minutes because reading prose is easy and checking facts is work.

That is the actual failure mode with AI-drafted documentation. Not hallucinated paragraphs — unverified specifics wearing good grammar.

The last week of DEV's front page leaned hard into AI fatigue and vibe-coding regret. Fair. But the useful response isn't refusing the tool; it's narrowing what the tool is allowed to produce and making the rest mechanically checkable.

Before any prompt runs, decide two things per section: is this prose the model may draft, and are the facts in it something a machine can confirm against the repo?

Sections where the answer is "no" on the second question are human-owned, always.

The rule that survives contact with reality: if the fact lives outside the repository, the model does not get to state it. No amount of retrieval fixes a source of truth you don't control.

Here is the piece I keep coming back to. A scope file declares which headings may be drafted, and a small standard-library script extracts every hard claim — env vars, CLI flags, repo paths, version strings — and requires each one to appear somewhere in the repo.

The code below is a template, and the output further down is illustrative. It runs on the Python standard library plus git grep; nothing else.

docs/ai-scope.json:

{
  "draftable": ["Overview", "Local setup", "Configuration reference", "API surface"],
  "human_owned": ["Security model", "Billing and quotas", "Deprecation", "SLA"],
  "fail_on_missing": ["env_var", "flag", "path"],
  "warn_on_missing": ["version"]
}

tools/claim_check.py:

#!/usr/bin/env python3
"""Fail a model-drafted doc that asserts repo facts the repo cannot confirm."""
import json, re, subprocess, sys
from pathlib import Path

REPO = Path(__file__).resolve().parents[1]
SCOPE = json.loads((REPO / "docs/ai-scope.json").read_text())

PATTERNS = {
    "env_var": re.compile(r"`([A-Z][A-Z0-9]*(?:_[A-Z0-9]+)+)`"),
    "flag":    re.compile(r"`(--[a-z][a-z0-9-]+)`"),
    "path":    re.compile(r"`((?:src|docs|scripts|config)/[\w./-]+)`"),
    "version": re.compile(r"\bv?(\d+\.\d+\.\d+)\b"),
}

def grep(needle: str):
    out = subprocess.run(["git", "grep", "-n", "-F", "--", needle],
                         cwd=REPO, capture_output=True, text=True)
    return [l for l in out.stdout.splitlines() if l.strip()]

def sections(text: str):
    """Yield (heading, body) for each H2."""
    current, buf = None, []
    for line in text.splitlines():
        if line.startswith("## "):
            if current is not None:
                yield current, "\n".join(buf)
            current, buf = line[3:].strip(), []
        elif current is not None:
            buf.append(line)
    if current is not None:
        yield current, "\n".join(buf)

def main(path: str) -> int:
    draft = Path(path).read_text()
    failures, warnings = [], []

    for heading, body in sections(draft):
        if any(h.lower() in heading.lower() for h in SCOPE["human_owned"]):
            failures.append(f"human-owned section present: {heading!r}")
            continue
        if not any(h.lower() in heading.lower() for h in SCOPE["draftable"]):
            failures.append(f"heading outside the draftable scope: {heading!r}")
            continue
        for kind, rx in PATTERNS.items():
            for claim in sorted(set(rx.findall(body))):
                hits = grep(claim)
                if hits:
                    print(f"OK   {kind:8} {claim:32} {hits[0].split(':')[0]}")
                elif kind in SCOPE["fail_on_missing"]:
                    failures.append(f"unverified {kind}: {claim} (section {heading!r})")
                else:
                    warnings.append(f"unverified {kind}: {claim} (section {heading!r})")

    for w in warnings:
        print(f"WARN {w}")
    for f in failures:
        print(f"FAIL {f}")
    return 1 if failures else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run it on the draft before a human ever reads it:

python3 tools/claim_check.py docs/drafts/configuration.md; echo "exit=$?"

Illustrative output for the retry-limit draft that started this post:

OK   env_var  LOG_LEVEL                        src/config.py
OVER env_var  RETRY_MAX_ATTEMPTS               src/retry.py
OK   flag     --dry-run                        src/cli.py
OK   path     scripts/seed.py                  scripts/seed.py
WARN version  2.4.1                            (no match)
FAIL unverified env_var: RETRY_MAX_ATTEMPTS (section 'Configuration reference')
FAIL human-owned section present: 'Billing and quotas'
exit=1

OVER is an overlay your checker will need anyway: the string exists, but not in the section the draft claims. Catching the rename is the whole point — the doc and the code disagree, and the checker refuses to let that pass.

Because the verdict has to be boring. A second model grading the first model's facts gives you two opinions and no evidence. git grep gives you a file, a line, and an exit code — something a reviewer can check in five seconds and a pipeline can gate on.

Use the model for the part it is genuinely good at: turning the diff, the config read, and the module docstrings into readable prose. Use the repo for the part prose cannot be trusted with.

docs/drafts/ only. Never generate directly into the published tree.claim_check.py; fix the draft or fix the docs, never the rule. FAIL/ WARN list, so attention lands on facts instead of adjectives.docs/ on merge; keep the draft path in the PR body for traceability. Step 2 is where a scheduled job earns its keep. A nightly run over every draft plus every recently changed config module catches drift before a reader does — and this is where free model access and a free server option become practical rather than theoretical: the generation step is a batch job you don't want to pay per-seat for, and the verification step is cheap enough to run on a small always-on box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm describing the workflow I'd use; the project advertises free model access and a free server option, and those terms are set by its operators, so check the current ones before you depend on them.

ai-scope.json deliberately. Skip it if your docs are mostly prose about why rather than how — architecture rationale, decision records, onboarding narrative. There are few hard claims to check and the overhead won't pay back.

Also skip it if your source of truth is a wiki outside the repo, or if nobody can own the scope file. An unmaintained gate becomes a skipped gate, and a skipped gate is worse than none because it looks like assurance.

If you do have a config surface, a CLI, and a repo you can grep, the split is worth the afternoon: let the model draft the setup steps and the config table, let claim_check.py argue with it, and keep the security model and the pricing page in human hands where they belong.