The Model May Draft the Prose. The Repo Must Confirm the Facts.

A developer described a lightweight standard-library Python script, claim_check.py, that uses git grep to verify every hard claim in AI-drafted documentation — environment variables, CLI flags, repo paths and version strings — against the repository itself. The approach pairs a scope file (docs/ai-scope.json) declaring which sections a model may draft with a rule that facts living outside the repo, such as billing limits, SLAs and security models, must remain human-owned. The motivation was a draft that cited a retry limit of 5 when the code defaulted to 3, with a renamed environment variable producing a silent no-op.

A doc said the retry limit defaults to 5. The code said 3. The env var in the doc had been renamed two releases earlier, so anyone who copy-pasted the snippet got a silent no-op instead of an error. Nothing about that draft looked wrong. The sentences were clean, the structure was better than what we had, and the reviewer approved it in four minutes because reading prose is easy and checking facts is work. That is the actual failure mode with AI-drafted documentation. Not hallucinated paragraphs — unverified specifics wearing good grammar. The last week of DEV's front page leaned hard into AI fatigue and vibe-coding regret. Fair. But the useful response isn't refusing the tool; it's narrowing what the tool is allowed to produce and making the rest mechanically checkable. Before any prompt runs, decide two things per section: is this prose the model may draft, and are the facts in it something a machine can confirm against the repo? Sections where the answer is "no" on the second question are human-owned, always. | Section | Model drafts? | Evidence source | Why | |---|---|---|---| | Overview / what it does | Yes | Repo README + module docstrings | Low risk, easy to correct | | Local setup steps | Yes | scripts/ , Makefile , CI config | Commands can be executed | | Config reference | Yes | Code that reads the env vars | Names and defaults are greppable | | API surface summary | Yes, with limits | Route definitions | Paths and methods are greppable | | Security model | No | Human security owner | Wrong text is a liability | | Billing, quotas, limits | No | Product owner, dated source | Numbers change without code | | Deprecation / sunset dates | No | Release manager | A guess here breaks customers | | SLA and support tiers | No | Contracts | Not derivable from code | The rule that survives contact with reality: if the fact lives outside the repository, the model does not get to state it. No amount of retrieval fixes a source of truth you don't control. Here is the piece I keep coming back to. A scope file declares which headings may be drafted, and a small standard-library script extracts every hard claim — env vars, CLI flags, repo paths, version strings — and requires each one to appear somewhere in the repo. The code below is a template, and the output further down is illustrative. It runs on the Python standard library plus git grep ; nothing else. docs/ai-scope.json : { "draftable": "Overview", "Local setup", "Configuration reference", "API surface" , "human owned": "Security model", "Billing and quotas", "Deprecation", "SLA" , "fail on missing": "env var", "flag", "path" , "warn on missing": "version" } tools/claim check.py : bash /usr/bin/env python3 """Fail a model-drafted doc that asserts repo facts the repo cannot confirm.""" import json, re, subprocess, sys from pathlib import Path REPO = Path file .resolve .parents 1 SCOPE = json.loads REPO / "docs/ai-scope.json" .read text PATTERNS = { "env var": re.compile r" A-Z A-Z0-9 ?: A-Z0-9 + + " , "flag": re.compile r" -- a-z a-z0-9- + " , "path": re.compile r" ?:src|docs|scripts|config / \w./- + " , "version": re.compile r"\bv? \d+\.\d+\.\d+ \b" , } def grep needle: str : out = subprocess.run "git", "grep", "-n", "-F", "--", needle , cwd=REPO, capture output=True, text=True return l for l in out.stdout.splitlines if l.strip def sections text: str : """Yield heading, body for each H2.""" current, buf = None, for line in text.splitlines : if line.startswith " " : if current is not None: yield current, "\n".join buf current, buf = line 3: .strip , elif current is not None: buf.append line if current is not None: yield current, "\n".join buf def main path: str - int: draft = Path path .read text failures, warnings = , for heading, body in sections draft : if any h.lower in heading.lower for h in SCOPE "human owned" : failures.append f"human-owned section present: {heading r}" continue if not any h.lower in heading.lower for h in SCOPE "draftable" : failures.append f"heading outside the draftable scope: {heading r}" continue for kind, rx in PATTERNS.items : for claim in sorted set rx.findall body : hits = grep claim if hits: print f"OK {kind:8} {claim:32} {hits 0 .split ':' 0 }" elif kind in SCOPE "fail on missing" : failures.append f"unverified {kind}: {claim} section {heading r} " else: warnings.append f"unverified {kind}: {claim} section {heading r} " for w in warnings: print f"WARN {w}" for f in failures: print f"FAIL {f}" return 1 if failures else 0 if name == " main ": sys.exit main sys.argv 1 Run it on the draft before a human ever reads it: python3 tools/claim check.py docs/drafts/configuration.md; echo "exit=$?" Illustrative output for the retry-limit draft that started this post: OK env var LOG LEVEL src/config.py OVER env var RETRY MAX ATTEMPTS src/retry.py OK flag --dry-run src/cli.py OK path scripts/seed.py scripts/seed.py WARN version 2.4.1 no match FAIL unverified env var: RETRY MAX ATTEMPTS section 'Configuration reference' FAIL human-owned section present: 'Billing and quotas' exit=1 OVER is an overlay your checker will need anyway: the string exists, but not in the section the draft claims. Catching the rename is the whole point — the doc and the code disagree, and the checker refuses to let that pass. Because the verdict has to be boring. A second model grading the first model's facts gives you two opinions and no evidence. git grep gives you a file, a line, and an exit code — something a reviewer can check in five seconds and a pipeline can gate on. Use the model for the part it is genuinely good at: turning the diff, the config read, and the module docstrings into readable prose. Use the repo for the part prose cannot be trusted with. docs/drafts/ only. Never generate directly into the published tree. claim check.py ; fix the draft or fix the docs, never the rule. FAIL / WARN list, so attention lands on facts instead of adjectives. docs/ on merge; keep the draft path in the PR body for traceability. Step 2 is where a scheduled job earns its keep. A nightly run over every draft plus every recently changed config module catches drift before a reader does — and this is where free model access and a free server option become practical rather than theoretical: the generation step is a batch job you don't want to pay per-seat for, and the verification step is cheap enough to run on a small always-on box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm describing the workflow I'd use; the project advertises free model access and a free server option, and those terms are set by its operators, so check the current ones before you depend on them. ai-scope.json deliberately. Skip it if your docs are mostly prose about why rather than how — architecture rationale, decision records, onboarding narrative. There are few hard claims to check and the overhead won't pay back. Also skip it if your source of truth is a wiki outside the repo, or if nobody can own the scope file. An unmaintained gate becomes a skipped gate, and a skipped gate is worse than none because it looks like assurance. If you do have a config surface, a CLI, and a repo you can grep, the split is worth the afternoon: let the model draft the setup steps and the config table, let claim check.py argue with it, and keep the security model and the pricing page in human hands where they belong.