Machine-drafted documentation fails in one predictable place: the handful of sections a human rewrites every single time. The fix is not a stronger prompt or a larger model, it is a budget that retires a section from drafting once its rewrite cost has been measured. This article builds that loop — a word-level rewrite ratio per section, a ledger committed to the repo, and a CI gate that revokes drafting rights.
Most teams try to answer "may the model write this section?" with a rubric or an LLM judge, and both are painful to defend in review. The signal you already have is the diff between the model draft and the text a human actually merged. That diff costs nothing extra to collect, and it is produced by exactly the people whose time the workflow is meant to protect.
The ratio is deliberately narrow: it answers one question, how much of this section did a human have to replace, and nothing else. It does not tell you whether the merged text is correct.
Two mechanisms get confused here, so separate them early. A pre-drafting gate decides whether the model may touch a section at all, usually from API churn or claim class. The budget described below is post-hoc: it reads accepted history and revokes rights that were already granted. Keep both if you want them, but keep them in different files and different jobs.
The metric is 1 - similarity(draft_tokens, merged_tokens), computed on lowercased words after removing fenced code, link targets, and Markdown punctuation. Pure copy-editing inflates it, so a comma-and-article pass can look like a rewrite; dropping tokens shorter than two characters limits that distortion without hiding real edits.
It is insensitive to meaning. Swapping two equivalent sentences scores badly, while a confidently wrong sentence nobody noticed scores perfectly, which is the worst failure mode of the metric. Treat it as a labour signal, never as a truth signal, and never publish it to readers as a quality statistic.
Everything downstream depends on matching the same section across runs, so the ID must be derived from heading text and must survive the heading being reworded. That tension is real in fast-moving docs. Derive the slug from the heading, freeze it in the ledger, and require an explicit alias entry whenever a heading is renamed.
The cheapest enforcement is a small reviewed file mapping IDs to current headings, edited in the same pull request as the rename. If a known slug vanishes from the page without an alias, the gate fails and a human decides whether the section was deleted or renamed.
Draft jobs run on a schedule, while humans merge whenever they get to it. Comparing a Monday draft against a Friday file measures every unrelated commit in between and produces meaningless ratios.
Write drafts to dated paths and record the base commit beside them. The gate then only scores a sample when the merged file's history contains that base commit; otherwise it skips the sample and says so in the job log, which keeps the ledger honest about coverage.
out/drafts/2026-11-06/guide.md
out/drafts/2026-11-06/base.txt # output of: git rev-parse HEAD
The script below is the whole artifact: it parses headings, normalises tokens, scores each drafted section against the merged page, and appends one sample to a JSON ledger. It uses only the standard library, so it runs anywhere Python does.
#!/usr/bin/env python3
"""doc_draft_ratio.py - per-section rewrite budget for model-drafted docs."""
import argparse, json, re, sys
from difflib import SequenceMatcher
from pathlib import Path
FENCE = re.compile(r"^```
.*?^
```", re.S | re.M)
LINK = re.compile(r"\[([^\]]+)\]\([^)]+\)")
HEADING = re.compile(r"^(#{2,4})\s+(.*)$", re.M)
def split_sections(text):
marks = list(HEADING.finditer(text))
out = {}
for i, m in enumerate(marks):
end = marks[i + 1].start() if i + 1 < len(marks) else len(text)
slug = re.sub(r"[^a-z0-9]+", "-", m.group(2).lower()).strip("-")
out[slug] = text[m.end():end]
return out
def tokens(chunk):
chunk = FENCE.sub(" ", chunk)
chunk = LINK.sub(r"\1", chunk)
chunk = re.sub(r"[`*_>#|-]", " ", chunk)
return [t for t in chunk.lower().split() if len(t) > 1]
def rewrite_ratio(draft, merged):
a, b = tokens(draft), tokens(merged)
if not a and not b:
return 0.0
return 1.0 - SequenceMatcher(None, a, b, autojunk=False).ratio()
def band(mean):
if mean < 0.15:
return "auto-draft"
if mean < 0.40:
return "draft-with-review"
return "human-owned"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--draft", required=True)
ap.add_argument("--merged", required=True)
ap.add_argument("--ledger", required=True)
ap.add_argument("--protect", default="", help="comma-separated human-owned slugs")
args = ap.parse_args()
drafts = split_sections(Path(args.draft).read_text())
merged = split_sections(Path(args.merged).read_text())
path = Path(args.ledger)
ledger = json.loads(path.read_text()) if path.exists() else {"sections": {}}
protected = {s for s in args.protect.split(",") if s}
violations = []
for slug, text in merged.items():
rec = ledger["sections"].setdefault(
slug, {"samples": 0, "mean": 0.0, "band": "auto-draft"}
)
if slug in drafts:
r = rewrite_ratio(drafts[slug], text)
rec["samples"] += 1
rec["mean"] += (r - rec["mean"]) / rec["samples"]
rec["band"] = band(rec["mean"])
if slug in protected and slug in drafts and drafts[slug].strip() != text.strip():
violations.append(slug)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(ledger, indent=2, sort_keys=True) + "\n")
for slug, rec in sorted(ledger["sections"].items()):
print(f"{rec['band']:18} {rec['mean']:.2f} n={rec['samples']:<3} {slug}")
if violations:
print("drafted protected sections: " + ", ".join(violations), file=sys.stderr)
return 1 if violations else 0
if __name__ == "__main__":
sys.exit(main())
Run it after each merge, or from CI once the draft has been accepted:
python doc_draft_ratio.py \
--draft out/drafts/2026-11-06/guide.md \
--merged docs/guide.md \
--ledger .docs/rewrite-ledger.json \
--protect install-from-source,security-model
The output is meant to be read in review, not admired as a dashboard. A ledger after six nightly runs on a 24-section page looks roughly like the block below, and these values are illustrative sample data rather than measured results from any repository:
auto-draft 0.07 n=6 install-from-source
auto-draft 0.12 n=5 configuration-reference
draft-with-review 0.26 n=6 migration-notes
human-owned 0.51 n=6 security-model
human-owned 0.63 n=4 supported-platforms
Thresholds are a policy choice, not a discovery, and they should live in the repository so nobody renegotiates them per pull request. The table below is a reasonable starting policy; change the numbers for your team, not the structure.
| Mean rewrite ratio | Band | Who may produce the text |
|---|---|---|
| 0.00 – 0.14 | auto-draft |
Model drafts; normal review applies |
| 0.15 – 0.39 | draft-with-review |
Model drafts; one named reviewer must edit |
| ≥ 0.40 | human-owned |
Nobody drafts; a human writes and freezes it |
Add one asymmetric rule, because regressions are more expensive than missed opportunities: three consecutive samples above a band boundary demote the section one band and freeze it there. Promotion back up requires a short written note in the pull request explaining what changed, which prevents threshold thrash.
The gate above only reports; the structure should prevent the mistake. Split the page into two directories, docs/draftable/ and docs/human/, and concatenate them at build time so the published page stays a single document.
./scripts/build-docs.sh # cat docs/human/*.md docs/draftable/*.md > site/guide.md
Now the drafting job can be given write access to one directory only, and --protect becomes a second line of defence rather than the only one. A protected section that changes in the draft is a bug in the job configuration, not a judgement call for the reviewer.
This loop produces drafts that are frequently thrown away, so the compute should be disposable. I run the nightly job on MonkeyCode's free server option, which is enough for a scheduled script that clones a repo, drafts a handful of sections, and commits a draft directory; the free model access keeps the drafting step from becoming a cost decision. The setup described in this article is my own, and it is deliberately unambitious: one cron entry, one draft commit, one gate. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
If you want to try the loop before wiring a scheduler, the free option is sufficient to run the script by hand and see whether your own ledger produces the same split between stable reference sections and argumentative ones.
auto-draft as approval.human-owned section off to the side, compare it, and keep the result out of the live page.
Skip this loop if your documentation carries legal or certification weight, where every sentence needs an attestation trail rather than a draft budget. Skip it if your pages have no stable heading structure, because the slug map will churn faster than the ledger can follow. Skip it if your docs are generated from source comments, where the model edits the generator rather than the prose. And skip it if you are the only maintainer of a handful of pages: you will already know which sections you rewrite, and the ledger is pure overhead.
The point of the budget is not to measure the model. It is to make the boundary between drafted prose and human-owned prose visible, testable, and slightly annoying to cross.