# The Model May Draft the Prose. The Repo Must Confirm the Facts. > Source: > Published: 2026-10-20 09:35:13+00:00 A doc said the retry limit defaults to 5. The code said 3. The env var in the doc had been renamed two releases earlier, so anyone who copy-pasted the snippet got a silent no-op instead of an error. Nothing about that draft looked wrong. The sentences were clean, the structure was better than what we had, and the reviewer approved it in four minutes because reading prose is easy and checking facts is work. That is the actual failure mode with AI-drafted documentation. Not hallucinated paragraphs — unverified specifics wearing good grammar. The last week of DEV's front page leaned hard into AI fatigue and vibe-coding regret. Fair. But the useful response isn't refusing the tool; it's narrowing what the tool is allowed to produce and making the rest mechanically checkable. Before any prompt runs, decide two things per section: is this prose the model may draft, and are the facts in it something a machine can confirm against the repo? Sections where the answer is "no" on the second question are human-owned, always. | Section | Model drafts? | Evidence source | Why | |---|---|---|---| | Overview / what it does | Yes | Repo README + module docstrings | Low risk, easy to correct | | Local setup steps | Yes | `scripts/` ,`Makefile` , CI config | Commands can be executed | | Config reference | Yes | Code that reads the env vars | Names and defaults are greppable | | API surface summary | Yes, with limits | Route definitions | Paths and methods are greppable | | Security model | No | Human security owner | Wrong text is a liability | | Billing, quotas, limits | No | Product owner, dated source | Numbers change without code | | Deprecation / sunset dates | No | Release manager | A guess here breaks customers | | SLA and support tiers | No | Contracts | Not derivable from code | The rule that survives contact with reality: **if the fact lives outside the repository, the model does not get to state it.** No amount of retrieval fixes a source of truth you don't control. Here is the piece I keep coming back to. A scope file declares which headings may be drafted, and a small standard-library script extracts every hard claim — env vars, CLI flags, repo paths, version strings — and requires each one to appear somewhere in the repo. The code below is a template, and the output further down is illustrative. It runs on the Python standard library plus `git grep`; nothing else. `docs/ai-scope.json`: ``` { "draftable": ["Overview", "Local setup", "Configuration reference", "API surface"], "human_owned": ["Security model", "Billing and quotas", "Deprecation", "SLA"], "fail_on_missing": ["env_var", "flag", "path"], "warn_on_missing": ["version"] } ``` `tools/claim_check.py`: ``` bash #!/usr/bin/env python3 """Fail a model-drafted doc that asserts repo facts the repo cannot confirm.""" import json, re, subprocess, sys from pathlib import Path REPO = Path(__file__).resolve().parents[1] SCOPE = json.loads((REPO / "docs/ai-scope.json").read_text()) PATTERNS = { "env_var": re.compile(r"`([A-Z][A-Z0-9]*(?:_[A-Z0-9]+)+)`"), "flag": re.compile(r"`(--[a-z][a-z0-9-]+)`"), "path": re.compile(r"`((?:src|docs|scripts|config)/[\w./-]+)`"), "version": re.compile(r"\bv?(\d+\.\d+\.\d+)\b"), } def grep(needle: str): out = subprocess.run(["git", "grep", "-n", "-F", "--", needle], cwd=REPO, capture_output=True, text=True) return [l for l in out.stdout.splitlines() if l.strip()] def sections(text: str): """Yield (heading, body) for each H2.""" current, buf = None, [] for line in text.splitlines(): if line.startswith("## "): if current is not None: yield current, "\n".join(buf) current, buf = line[3:].strip(), [] elif current is not None: buf.append(line) if current is not None: yield current, "\n".join(buf) def main(path: str) -> int: draft = Path(path).read_text() failures, warnings = [], [] for heading, body in sections(draft): if any(h.lower() in heading.lower() for h in SCOPE["human_owned"]): failures.append(f"human-owned section present: {heading!r}") continue if not any(h.lower() in heading.lower() for h in SCOPE["draftable"]): failures.append(f"heading outside the draftable scope: {heading!r}") continue for kind, rx in PATTERNS.items(): for claim in sorted(set(rx.findall(body))): hits = grep(claim) if hits: print(f"OK {kind:8} {claim:32} {hits[0].split(':')[0]}") elif kind in SCOPE["fail_on_missing"]: failures.append(f"unverified {kind}: {claim} (section {heading!r})") else: warnings.append(f"unverified {kind}: {claim} (section {heading!r})") for w in warnings: print(f"WARN {w}") for f in failures: print(f"FAIL {f}") return 1 if failures else 0 if __name__ == "__main__": sys.exit(main(sys.argv[1])) ``` Run it on the draft before a human ever reads it: ``` python3 tools/claim_check.py docs/drafts/configuration.md; echo "exit=$?" ``` Illustrative output for the retry-limit draft that started this post: ``` OK env_var LOG_LEVEL src/config.py OVER env_var RETRY_MAX_ATTEMPTS src/retry.py OK flag --dry-run src/cli.py OK path scripts/seed.py scripts/seed.py WARN version 2.4.1 (no match) FAIL unverified env_var: RETRY_MAX_ATTEMPTS (section 'Configuration reference') FAIL human-owned section present: 'Billing and quotas' exit=1 ``` `OVER` is an overlay your checker will need anyway: the string exists, but not in the section the draft claims. Catching the rename is the whole point — the doc and the code disagree, and the checker refuses to let that pass. Because the verdict has to be boring. A second model grading the first model's facts gives you two opinions and no evidence. `git grep` gives you a file, a line, and an exit code — something a reviewer can check in five seconds and a pipeline can gate on. Use the model for the part it is genuinely good at: turning the diff, the config read, and the module docstrings into readable prose. Use the repo for the part prose cannot be trusted with. `docs/drafts/` only. Never generate directly into the published tree.`claim_check.py`; fix the draft or fix the docs, never the rule.` FAIL`/` WARN` list, so attention lands on facts instead of adjectives.`docs/` on merge; keep the draft path in the PR body for traceability. Step 2 is where a scheduled job earns its keep. A nightly run over every draft plus every recently changed config module catches drift before a reader does — and this is where free model access and a free server option become practical rather than theoretical: the generation step is a batch job you don't want to pay per-seat for, and the verification step is cheap enough to run on a small always-on box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm describing the workflow I'd use; the project advertises free model access and a free server option, and those terms are set by its operators, so check the current ones before you depend on them. `ai-scope.json` deliberately. Skip it if your docs are mostly prose about *why* rather than *how* — architecture rationale, decision records, onboarding narrative. There are few hard claims to check and the overhead won't pay back. Also skip it if your source of truth is a wiki outside the repo, or if nobody can own the scope file. An unmaintained gate becomes a skipped gate, and a skipped gate is worse than none because it looks like assurance. If you do have a config surface, a CLI, and a repo you can grep, the split is worth the afternoon: let the model draft the setup steps and the config table, let `claim_check.py` argue with it, and keep the security model and the pricing page in human hands where they belong.