{"slug": "retire-a-draft-before-it-ships-edit-distance-budgets-for-model-written-docs", "title": "Retire a Draft Before It Ships: Edit-Distance Budgets for Model-Written Docs", "summary": "A developer built a post-hoc rewrite-budget system for model-drafted documentation that measures a per-section edit-distance ratio between the model's draft and the text a human actually merged, then revokes drafting rights for sections that humans consistently rewrite. The approach records drafts to dated paths with their base commit, appends samples to a JSON ledger committed to the repo, and enforces the budget through a CI gate that fails when a known section slug disappears without an explicit alias. The developer stresses the metric is a labour signal, not a truth signal, and should never be published as a quality statistic.", "body_md": "Machine-drafted documentation fails in one predictable place: the handful of sections a human rewrites every single time. The fix is not a stronger prompt or a larger model, it is a budget that retires a section from drafting once its rewrite cost has been measured. This article builds that loop — a word-level rewrite ratio per section, a ledger committed to the repo, and a CI gate that revokes drafting rights.\n\nMost teams try to answer \"may the model write this section?\" with a rubric or an LLM judge, and both are painful to defend in review. The signal you already have is the diff between the model draft and the text a human actually merged. That diff costs nothing extra to collect, and it is produced by exactly the people whose time the workflow is meant to protect.\n\nThe ratio is deliberately narrow: it answers one question, how much of this section did a human have to replace, and nothing else. It does not tell you whether the merged text is correct.\n\nTwo mechanisms get confused here, so separate them early. A pre-drafting gate decides whether the model may touch a section at all, usually from API churn or claim class. The budget described below is post-hoc: it reads accepted history and revokes rights that were already granted. Keep both if you want them, but keep them in different files and different jobs.\n\nThe metric is `1 - similarity(draft_tokens, merged_tokens)`, computed on lowercased words after removing fenced code, link targets, and Markdown punctuation. Pure copy-editing inflates it, so a comma-and-article pass can look like a rewrite; dropping tokens shorter than two characters limits that distortion without hiding real edits.\n\nIt is insensitive to meaning. Swapping two equivalent sentences scores badly, while a confidently wrong sentence nobody noticed scores perfectly, which is the worst failure mode of the metric. Treat it as a labour signal, never as a truth signal, and never publish it to readers as a quality statistic.\n\nEverything downstream depends on matching the same section across runs, so the ID must be derived from heading text and must survive the heading being reworded. That tension is real in fast-moving docs. Derive the slug from the heading, freeze it in the ledger, and require an explicit alias entry whenever a heading is renamed.\n\nThe cheapest enforcement is a small reviewed file mapping IDs to current headings, edited in the same pull request as the rename. If a known slug vanishes from the page without an alias, the gate fails and a human decides whether the section was deleted or renamed.\n\nDraft jobs run on a schedule, while humans merge whenever they get to it. Comparing a Monday draft against a Friday file measures every unrelated commit in between and produces meaningless ratios.\n\nWrite drafts to dated paths and record the base commit beside them. The gate then only scores a sample when the merged file's history contains that base commit; otherwise it skips the sample and says so in the job log, which keeps the ledger honest about coverage.\n\n```\n# nightly draft job output\nout/drafts/2026-11-06/guide.md\nout/drafts/2026-11-06/base.txt   # output of: git rev-parse HEAD\n```\n\nThe script below is the whole artifact: it parses headings, normalises tokens, scores each drafted section against the merged page, and appends one sample to a JSON ledger. It uses only the standard library, so it runs anywhere Python does.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"doc_draft_ratio.py - per-section rewrite budget for model-drafted docs.\"\"\"\nimport argparse, json, re, sys\nfrom difflib import SequenceMatcher\nfrom pathlib import Path\n\nFENCE = re.compile(r\"^```\n\n.*?^\n\n```\", re.S | re.M)\nLINK = re.compile(r\"\\[([^\\]]+)\\]\\([^)]+\\)\")\nHEADING = re.compile(r\"^(#{2,4})\\s+(.*)$\", re.M)\n\ndef split_sections(text):\n    marks = list(HEADING.finditer(text))\n    out = {}\n    for i, m in enumerate(marks):\n        end = marks[i + 1].start() if i + 1 < len(marks) else len(text)\n        slug = re.sub(r\"[^a-z0-9]+\", \"-\", m.group(2).lower()).strip(\"-\")\n        out[slug] = text[m.end():end]\n    return out\n\ndef tokens(chunk):\n    chunk = FENCE.sub(\" \", chunk)\n    chunk = LINK.sub(r\"\\1\", chunk)\n    chunk = re.sub(r\"[`*_>#|-]\", \" \", chunk)\n    return [t for t in chunk.lower().split() if len(t) > 1]\n\ndef rewrite_ratio(draft, merged):\n    a, b = tokens(draft), tokens(merged)\n    if not a and not b:\n        return 0.0\n    return 1.0 - SequenceMatcher(None, a, b, autojunk=False).ratio()\n\ndef band(mean):\n    if mean < 0.15:\n        return \"auto-draft\"\n    if mean < 0.40:\n        return \"draft-with-review\"\n    return \"human-owned\"\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--draft\", required=True)\n    ap.add_argument(\"--merged\", required=True)\n    ap.add_argument(\"--ledger\", required=True)\n    ap.add_argument(\"--protect\", default=\"\", help=\"comma-separated human-owned slugs\")\n    args = ap.parse_args()\n\n    drafts = split_sections(Path(args.draft).read_text())\n    merged = split_sections(Path(args.merged).read_text())\n    path = Path(args.ledger)\n    ledger = json.loads(path.read_text()) if path.exists() else {\"sections\": {}}\n    protected = {s for s in args.protect.split(\",\") if s}\n    violations = []\n\n    for slug, text in merged.items():\n        rec = ledger[\"sections\"].setdefault(\n            slug, {\"samples\": 0, \"mean\": 0.0, \"band\": \"auto-draft\"}\n        )\n        if slug in drafts:\n            r = rewrite_ratio(drafts[slug], text)\n            rec[\"samples\"] += 1\n            rec[\"mean\"] += (r - rec[\"mean\"]) / rec[\"samples\"]\n            rec[\"band\"] = band(rec[\"mean\"])\n        if slug in protected and slug in drafts and drafts[slug].strip() != text.strip():\n            violations.append(slug)\n\n    path.parent.mkdir(parents=True, exist_ok=True)\n    path.write_text(json.dumps(ledger, indent=2, sort_keys=True) + \"\\n\")\n    for slug, rec in sorted(ledger[\"sections\"].items()):\n        print(f\"{rec['band']:18} {rec['mean']:.2f} n={rec['samples']:<3} {slug}\")\n    if violations:\n        print(\"drafted protected sections: \" + \", \".join(violations), file=sys.stderr)\n    return 1 if violations else 0\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\nRun it after each merge, or from CI once the draft has been accepted:\n\n```\npython doc_draft_ratio.py \\\n  --draft out/drafts/2026-11-06/guide.md \\\n  --merged docs/guide.md \\\n  --ledger .docs/rewrite-ledger.json \\\n  --protect install-from-source,security-model\n```\n\nThe output is meant to be read in review, not admired as a dashboard. A ledger after six nightly runs on a 24-section page looks roughly like the block below, and these values are illustrative sample data rather than measured results from any repository:\n\n```\nauto-draft         0.07 n=6   install-from-source\nauto-draft         0.12 n=5   configuration-reference\ndraft-with-review  0.26 n=6   migration-notes\nhuman-owned        0.51 n=6   security-model\nhuman-owned        0.63 n=4   supported-platforms\n```\n\nThresholds are a policy choice, not a discovery, and they should live in the repository so nobody renegotiates them per pull request. The table below is a reasonable starting policy; change the numbers for your team, not the structure.\n\n| Mean rewrite ratio | Band | Who may produce the text | \n|---|---|---|\n| 0.00 – 0.14 | `auto-draft` | Model drafts; normal review applies | \n| 0.15 – 0.39 | `draft-with-review` | Model drafts; one named reviewer must edit | \n| ≥ 0.40 | `human-owned` | Nobody drafts; a human writes and freezes it | \n\nAdd one asymmetric rule, because regressions are more expensive than missed opportunities: three consecutive samples above a band boundary demote the section one band and freeze it there. Promotion back up requires a short written note in the pull request explaining what changed, which prevents threshold thrash.\n\nThe gate above only reports; the structure should prevent the mistake. Split the page into two directories, `docs/draftable/` and `docs/human/`, and concatenate them at build time so the published page stays a single document.\n\n```\n./scripts/build-docs.sh   # cat docs/human/*.md docs/draftable/*.md > site/guide.md\n```\n\nNow the drafting job can be given write access to one directory only, and `--protect` becomes a second line of defence rather than the only one. A protected section that changes in the draft is a bug in the job configuration, not a judgement call for the reviewer.\n\nThis loop produces drafts that are frequently thrown away, so the compute should be disposable. I run the nightly job on MonkeyCode's free server option, which is enough for a scheduled script that clones a repo, drafts a handful of sections, and commits a draft directory; the free model access keeps the drafting step from becoming a cost decision. The setup described in this article is my own, and it is deliberately unambitious: one cron entry, one draft commit, one gate. Disclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nIf you want to try the loop before wiring a scheduler, the free option is sufficient to run the script by hand and see whether your own ledger produces the same split between stable reference sections and argumentative ones.\n\n`auto-draft` as approval.`human-owned` section off to the side, compare it, and keep the result out of the live page.\nSkip this loop if your documentation carries legal or certification weight, where every sentence needs an attestation trail rather than a draft budget. Skip it if your pages have no stable heading structure, because the slug map will churn faster than the ledger can follow. Skip it if your docs are generated from source comments, where the model edits the generator rather than the prose. And skip it if you are the only maintainer of a handful of pages: you will already know which sections you rewrite, and the ledger is pure overhead.\n\nThe point of the budget is not to measure the model. It is to make the boundary between drafted prose and human-owned prose visible, testable, and slightly annoying to cross.", "url": "https://wpnews.pro/news/retire-a-draft-before-it-ships-edit-distance-budgets-for-model-written-docs", "canonical_source": "https://dev.to/github_7727/retire-a-draft-before-it-ships-edit-distance-budgets-for-model-written-docs-1121", "published_at": "2026-11-05 20:18:53+00:00", "updated_at": "2026-11-05 20:19:26.186618+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "mlops", "large-language-models"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/retire-a-draft-before-it-ships-edit-distance-budgets-for-model-written-docs", "markdown": "https://wpnews.pro/news/retire-a-draft-before-it-ships-edit-distance-budgets-for-model-written-docs.md", "text": "https://wpnews.pro/news/retire-a-draft-before-it-ships-edit-distance-budgets-for-model-written-docs.txt", "jsonld": "https://wpnews.pro/news/retire-a-draft-before-it-ships-edit-distance-budgets-for-model-written-docs.jsonld"}}