Retire a Draft Before It Ships: Edit-Distance Budgets for Model-Written Docs

A developer built a post-hoc rewrite-budget system for model-drafted documentation that measures a per-section edit-distance ratio between the model's draft and the text a human actually merged, then revokes drafting rights for sections that humans consistently rewrite. The approach records drafts to dated paths with their base commit, appends samples to a JSON ledger committed to the repo, and enforces the budget through a CI gate that fails when a known section slug disappears without an explicit alias. The developer stresses the metric is a labour signal, not a truth signal, and should never be published as a quality statistic.

Machine-drafted documentation fails in one predictable place: the handful of sections a human rewrites every single time. The fix is not a stronger prompt or a larger model, it is a budget that retires a section from drafting once its rewrite cost has been measured. This article builds that loop — a word-level rewrite ratio per section, a ledger committed to the repo, and a CI gate that revokes drafting rights. Most teams try to answer "may the model write this section?" with a rubric or an LLM judge, and both are painful to defend in review. The signal you already have is the diff between the model draft and the text a human actually merged. That diff costs nothing extra to collect, and it is produced by exactly the people whose time the workflow is meant to protect. The ratio is deliberately narrow: it answers one question, how much of this section did a human have to replace, and nothing else. It does not tell you whether the merged text is correct. Two mechanisms get confused here, so separate them early. A pre-drafting gate decides whether the model may touch a section at all, usually from API churn or claim class. The budget described below is post-hoc: it reads accepted history and revokes rights that were already granted. Keep both if you want them, but keep them in different files and different jobs. The metric is 1 - similarity draft tokens, merged tokens , computed on lowercased words after removing fenced code, link targets, and Markdown punctuation. Pure copy-editing inflates it, so a comma-and-article pass can look like a rewrite; dropping tokens shorter than two characters limits that distortion without hiding real edits. It is insensitive to meaning. Swapping two equivalent sentences scores badly, while a confidently wrong sentence nobody noticed scores perfectly, which is the worst failure mode of the metric. Treat it as a labour signal, never as a truth signal, and never publish it to readers as a quality statistic. Everything downstream depends on matching the same section across runs, so the ID must be derived from heading text and must survive the heading being reworded. That tension is real in fast-moving docs. Derive the slug from the heading, freeze it in the ledger, and require an explicit alias entry whenever a heading is renamed. The cheapest enforcement is a small reviewed file mapping IDs to current headings, edited in the same pull request as the rename. If a known slug vanishes from the page without an alias, the gate fails and a human decides whether the section was deleted or renamed. Draft jobs run on a schedule, while humans merge whenever they get to it. Comparing a Monday draft against a Friday file measures every unrelated commit in between and produces meaningless ratios. Write drafts to dated paths and record the base commit beside them. The gate then only scores a sample when the merged file's history contains that base commit; otherwise it skips the sample and says so in the job log, which keeps the ledger honest about coverage. nightly draft job output out/drafts/2026-11-06/guide.md out/drafts/2026-11-06/base.txt output of: git rev-parse HEAD The script below is the whole artifact: it parses headings, normalises tokens, scores each drafted section against the merged page, and appends one sample to a JSON ledger. It uses only the standard library, so it runs anywhere Python does. bash /usr/bin/env python3 """doc draft ratio.py - per-section rewrite budget for model-drafted docs.""" import argparse, json, re, sys from difflib import SequenceMatcher from pathlib import Path FENCE = re.compile r"^ . ?^ ", re.S | re.M LINK = re.compile r"\ ^\ + \ \ ^ +\ " HEADING = re.compile r"^ {2,4} \s+ . $", re.M def split sections text : marks = list HEADING.finditer text out = {} for i, m in enumerate marks : end = marks i + 1 .start if i + 1 < len marks else len text slug = re.sub r" ^a-z0-9 +", "-", m.group 2 .lower .strip "-" out slug = text m.end :end return out def tokens chunk : chunk = FENCE.sub " ", chunk chunk = LINK.sub r"\1", chunk chunk = re.sub r" |- ", " ", chunk return t for t in chunk.lower .split if len t 1 def rewrite ratio draft, merged : a, b = tokens draft , tokens merged if not a and not b: return 0.0 return 1.0 - SequenceMatcher None, a, b, autojunk=False .ratio def band mean : if mean < 0.15: return "auto-draft" if mean < 0.40: return "draft-with-review" return "human-owned" def main : ap = argparse.ArgumentParser ap.add argument "--draft", required=True ap.add argument "--merged", required=True ap.add argument "--ledger", required=True ap.add argument "--protect", default="", help="comma-separated human-owned slugs" args = ap.parse args drafts = split sections Path args.draft .read text merged = split sections Path args.merged .read text path = Path args.ledger ledger = json.loads path.read text if path.exists else {"sections": {}} protected = {s for s in args.protect.split "," if s} violations = for slug, text in merged.items : rec = ledger "sections" .setdefault slug, {"samples": 0, "mean": 0.0, "band": "auto-draft"} if slug in drafts: r = rewrite ratio drafts slug , text rec "samples" += 1 rec "mean" += r - rec "mean" / rec "samples" rec "band" = band rec "mean" if slug in protected and slug in drafts and drafts slug .strip = text.strip : violations.append slug path.parent.mkdir parents=True, exist ok=True path.write text json.dumps ledger, indent=2, sort keys=True + "\n" for slug, rec in sorted ledger "sections" .items : print f"{rec 'band' :18} {rec 'mean' :.2f} n={rec 'samples' :<3} {slug}" if violations: print "drafted protected sections: " + ", ".join violations , file=sys.stderr return 1 if violations else 0 if name == " main ": sys.exit main Run it after each merge, or from CI once the draft has been accepted: python doc draft ratio.py \ --draft out/drafts/2026-11-06/guide.md \ --merged docs/guide.md \ --ledger .docs/rewrite-ledger.json \ --protect install-from-source,security-model The output is meant to be read in review, not admired as a dashboard. A ledger after six nightly runs on a 24-section page looks roughly like the block below, and these values are illustrative sample data rather than measured results from any repository: auto-draft 0.07 n=6 install-from-source auto-draft 0.12 n=5 configuration-reference draft-with-review 0.26 n=6 migration-notes human-owned 0.51 n=6 security-model human-owned 0.63 n=4 supported-platforms Thresholds are a policy choice, not a discovery, and they should live in the repository so nobody renegotiates them per pull request. The table below is a reasonable starting policy; change the numbers for your team, not the structure. | Mean rewrite ratio | Band | Who may produce the text | |---|---|---| | 0.00 – 0.14 | auto-draft | Model drafts; normal review applies | | 0.15 – 0.39 | draft-with-review | Model drafts; one named reviewer must edit | | ≥ 0.40 | human-owned | Nobody drafts; a human writes and freezes it | Add one asymmetric rule, because regressions are more expensive than missed opportunities: three consecutive samples above a band boundary demote the section one band and freeze it there. Promotion back up requires a short written note in the pull request explaining what changed, which prevents threshold thrash. The gate above only reports; the structure should prevent the mistake. Split the page into two directories, docs/draftable/ and docs/human/ , and concatenate them at build time so the published page stays a single document. ./scripts/build-docs.sh cat docs/human/ .md docs/draftable/ .md site/guide.md Now the drafting job can be given write access to one directory only, and --protect becomes a second line of defence rather than the only one. A protected section that changes in the draft is a bug in the job configuration, not a judgement call for the reviewer. This loop produces drafts that are frequently thrown away, so the compute should be disposable. I run the nightly job on MonkeyCode's free server option, which is enough for a scheduled script that clones a repo, drafts a handful of sections, and commits a draft directory; the free model access keeps the drafting step from becoming a cost decision. The setup described in this article is my own, and it is deliberately unambitious: one cron entry, one draft commit, one gate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want to try the loop before wiring a scheduler, the free option is sufficient to run the script by hand and see whether your own ledger produces the same split between stable reference sections and argumentative ones. auto-draft as approval. human-owned section off to the side, compare it, and keep the result out of the live page. Skip this loop if your documentation carries legal or certification weight, where every sentence needs an attestation trail rather than a draft budget. Skip it if your pages have no stable heading structure, because the slug map will churn faster than the ledger can follow. Skip it if your docs are generated from source comments, where the model edits the generator rather than the prose. And skip it if you are the only maintainer of a handful of pages: you will already know which sections you rewrite, and the ledger is pure overhead. The point of the budget is not to measure the model. It is to make the boundary between drafted prose and human-owned prose visible, testable, and slightly annoying to cross.