{"slug": "using-ai-to-write-technical-documentation-what-actually-works", "title": "Using AI to Write Technical Documentation — What Actually Works", "summary": "An engineer's experiments with using LLMs to write technical documentation found that models produce fluent but factually wrong docs, inventing parameters and behaviors not present in the code. The developer recommends constraining models to ground-truth artifacts like function signatures, tests, and diffs, and using diff-scoped documentation passes instead of whole-repo rewrites.", "body_md": "An LLM asked to document a payments module — about 900 lines, four public functions, a retry wrapper — will typically produce something gorgeous. Structured headings, a parameters table, a \"Common Pitfalls\" section. It will also document a `timeout_seconds`\n\nparameter that does not exist. It will describe exponential backoff with jitter when the code does a flat `time.sleep(2)`\n\nin a `for`\n\nloop. It will include a usage example that imports a symbol from the wrong module.\n\nErrors like these survive for weeks, because nobody reads documentation that nobody trusts. The first person to try the example files a bug against the library.\n\nThat's the core problem with pointing a model at a repo and asking for docs. The model isn't reading the code so much as pattern-matching it to the average of every similar library it has seen. A retry helper *looks* like `tenacity`\n\n, so it gets `tenacity`\n\n's parameters. A webhook handler *looks* like Stripe's, so it gets Stripe's idempotency semantics. The output is fluent, structurally correct, and wrong in exactly the places a reader can't check without reading the source — which is the whole reason they came to the docs.\n\nWrong docs are worse than no docs. No docs make you read the code. Wrong docs make you skip reading the code.\n\nModels are bad at inventing facts about a system and good at reformatting facts handed to them. So stop asking for documentation and start asking for transformations of artefacts that already encode the truth.\n\nA codebase is full of these:\n\n`def charge(account_id: UUID, amount: Money, *, idempotency_key: str) -> ChargeResult`\n\nalready tells you every parameter, whether it's keyword-only, and what comes back. A model can't invent a fifth parameter if it's constrained to the signature.`test_refund_partial_amount`\n\ninto a documented example is a formatting job, not a reasoning job.`utoipa`\n\n), the spec is ground truth. Prose `ALTER TABLE charges ADD COLUMN settled_at timestamptz NULL`\n\ntells you the column exists, its type, and that it's nullable. That's three facts you don't have to trust anyone for.`git log --follow -p src/billing/retry.py`\n\nexplains The prompt shifts from \"document this module\" to \"here is the signature, here are the three tests that cover it, here is the commit that introduced it — write the docstring, and write `UNKNOWN`\n\nfor anything not present in this input.\"\n\nThat last instruction does most of the work. Models will happily emit `UNKNOWN`\n\nwhen it's an allowed output. They will not volunteer uncertainty when the only available move is prose.\n\nA whole-repo docs pass is a one-time event that produces a large, unreviewable PR. Nobody reads a 4,000-line documentation diff carefully. It gets approved on vibes and rots from day one.\n\nA diff-scoped pass is small enough to actually review, and it lands with the change that motivated it. A script along these lines does the job.\n\n``` bash\n#!/usr/bin/env bash\n# scripts/docstring-pass.sh — suggest docstrings for functions touched in this branch\nset -euo pipefail\n\nBASE=\"${1:-origin/main}\"\nOUT=\"${OUT:-.docsuggest}\"\nmkdir -p \"$OUT\"\n\n# Changed source files, excluding tests and generated code\nmapfile -t FILES < <(\n  git diff --name-only \"$BASE...HEAD\" -- '*.py' \\\n    ':!tests/*' ':!*_pb2.py' ':!**/migrations/*'\n)\n\n[[ ${#FILES[@]} -eq 0 ]] && { echo \"no source changes\"; exit 0; }\n\nfor f in \"${FILES[@]}\"; do\n  [[ -f \"$f\" ]] || continue\n\n  # Tests that mention the module path — cheap but effective\n  mod=$(basename \"$f\" .py)\n  tests=$(grep -rl \"$mod\" tests/ 2>/dev/null | head -3 | xargs -r cat)\n\n  {\n    echo \"## Current file: $f\"\n    echo '```\n\npython'; cat \"$f\"; echo '\n\n```'\n    echo \"## Diff in this branch\"\n    echo '```\n\ndiff'; git diff \"$BASE...HEAD\" -- \"$f\"; echo '\n\n```'\n    echo \"## Relevant tests (these are the only examples known to run)\"\n    echo '```\n\npython'; echo \"$tests\"; echo '\n\n```'\n    echo \"## Recent history\"\n    git log --oneline -5 -- \"$f\"\n  } | llm -m claude-sonnet-4-5 \\\n        --system \"$(cat prompts/docstring.md)\" \\\n        > \"$OUT/$(echo \"$f\" | tr '/' '_').md\"\ndone\n\necho \"suggestions in $OUT/\"\n```\n\n`llm`\n\nis Simon Willison's CLI (`pipx install llm`\n\n, then `llm keys set anthropic`\n\n). The system prompt is where the constraints live:\n\n``` php\n<!-- prompts/docstring.md -->\nWrite Google-style docstrings for functions ADDED or MODIFIED in the diff.\nDo not touch functions that are unchanged.\n\nRules:\n- Every parameter you document must appear in the function signature.\n  Copy the name and type annotation exactly.\n- Do not describe retry, caching, timeout, or concurrency behaviour\n  unless it is visible in the code shown.\n- Every code example must be derived from a test in the input.\n  Cite the test name in a comment. If there is no test, write no example.\n- If a parameter's meaning is not determinable from the code, tests,\n  or commit messages, write: \"UNKNOWN — needs author input\".\n- Output a unified diff against the current file. No commentary.\n```\n\nOutput goes to a scratch directory, not straight into the file. The engineer applies what's right and deletes the rest. The `UNKNOWN`\n\nmarkers become a to-do list of things only a human knows.\n\nThe single highest-value check: every example in the docs must execute in CI. For Python, doctest gets you there almost free.\n\n```\n# .github/workflows/docs.yml\n- name: Examples in docstrings must run\n  run: python -m pytest --doctest-modules src/ -q\n\n- name: Examples in markdown must run\n  run: python -m pytest --codeblocks docs/   # pytest-codeblocks\n\n- name: API docs must match the implementation\n  run: |\n    python -m app.export_openapi > /tmp/openapi.json\n    git diff --exit-code --no-index docs/openapi.json /tmp/openapi.json\n```\n\nThat last step is the one that catches drift. If someone adds a query parameter and doesn't regenerate the spec, CI fails with a diff showing exactly what changed. For contract-level checking, `schemathesis run docs/openapi.json --url http://localhost:8000`\n\nwill hammer a running service with requests derived from the spec and report where behaviour and documentation disagree.\n\nGenerated prose has no compiler. A docstring that describes behaviour removed six months ago will sit there forever. What's needed is a mechanical link between the doc and the thing it documents.\n\nOne approach is to stamp generated sections with a content hash of the source:\n\n``` php\n<!-- gen-from: src/billing/webhooks.py sha256:4f1a9c2e -->\n### Webhook verification\n...\n<!-- /gen-from -->\n```\n\nAnd check it:\n\n``` python\n# scripts/check_docstamps.py\nimport hashlib, pathlib, re, sys\n\nPAT = re.compile(r\"<!-- gen-from: (\\S+) sha256:([0-9a-f]+) -->\")\nstale = []\nfor md in pathlib.Path(\"docs\").rglob(\"*.md\"):\n    for src, want in PAT.findall(md.read_text()):\n        got = hashlib.sha256(pathlib.Path(src).read_bytes()).hexdigest()[:8]\n        if got != want:\n            stale.append(f\"{md}: {src} changed ({want} -> {got})\")\n\nif stale:\n    print(\"Stale generated docs:\\n  \" + \"\\n  \".join(stale))\n    sys.exit(1)\n```\n\nIt's blunt — a whitespace change trips it — but a noisy check that forces a five-second re-read beats silent rot. Add `python scripts/check_docstamps.py`\n\nto CI and require it on protected branches.\n\nThe highest-value prompt is rarely \"write docs.\" It's this, run against a doc page plus the code it describes:\n\nYou are a competent engineer who has never seen this system. Read the documentation, then the code. List every point where the docs would leave you unable to complete the task, and every claim in the docs you cannot verify in the code. Do not rewrite anything.\n\nTypical output from a run against a webhooks page:\n\n`on_conflict_do_nothing`\n\nthat the docs don't mention.\"`replay_window`\n\nis documented in seconds. In the code it's compared against a `timedelta`\n\nbuilt from minutes.\"The third kind of finding is a real bug, not a documentation nit — surfaced without the model writing a word of documentation. Critique is a much easier task than generation, because the ground truth is in the context window instead of the weights.\n\nArchitecture docs. Anything requiring \"why we chose this over that.\" Runbooks that depend on knowing which alert is a false alarm at 3am. Onboarding guides that need to know what a particular team finds confusing. The model has no access to the arguments in Slack, the outage that shaped the design, or the vendor limitation someone worked around. Generate these and you get a plausible-sounding history that never happened, which is a specific kind of poison for a new hire.\n\nAlso: models are bad at knowing what to leave out. Ask for a module overview and you'll get every private helper documented at equal weight to the one function anyone calls.\n\nProducing docs is easy to measure and worthless. These signals are better.\n\n**Search queries with zero results.** If the docs site has search (Algolia DocSearch, or MkDocs Material's built-in with the plugin's log), the null-result queries are a direct list of pages that should be written. If there are no null-result queries, nobody is searching.\n\n**Support questions that have an answer in the docs.** Tag them. If a question is answered on a page and someone asked anyway, the page exists but doesn't surface, doesn't rank, or isn't believable. That's three different fixes.\n\n**Links in code review.** Grep PR comments for the docs domain. Engineers linking each other to a page is the strongest signal that the page is load-bearing.\n\n**A canary.** Bury one specific, checkable detail in a page — a named constant, an unusual flag. When someone uses it correctly without asking, you know they read it.\n\n**Time-to-first-merged-PR for new hires.** Slow to move, but it's the number the docs exist to change.\n\nTeams that look honestly at page views over a quarter routinely find a third of their documentation can be deleted without anyone noticing. What deserves to survive is the set of pages updated in the same PR as the code, with examples that run in CI, and a model doing the first pass on the docstrings and a second pass as the confused reader. That combination is worth real engineering time. Pointing the model at the repo and pressing go is not.", "url": "https://wpnews.pro/news/using-ai-to-write-technical-documentation-what-actually-works", "canonical_source": "https://dev.to/ethan_linden_195175e739c9/using-ai-to-write-technical-documentation-what-actually-works-2fag", "published_at": "2026-08-27 10:03:57+00:00", "updated_at": "2026-08-27 10:18:24.515371+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools"], "entities": ["tenacity", "Stripe", "utoipa"], "alternates": {"html": "https://wpnews.pro/news/using-ai-to-write-technical-documentation-what-actually-works", "markdown": "https://wpnews.pro/news/using-ai-to-write-technical-documentation-what-actually-works.md", "text": "https://wpnews.pro/news/using-ai-to-write-technical-documentation-what-actually-works.txt", "jsonld": "https://wpnews.pro/news/using-ai-to-write-technical-documentation-what-actually-works.jsonld"}}