{"slug": "test-first-ai-documentation-a-workflow-that-keeps-generated-docs-honest", "title": "Test-First AI Documentation: A Workflow That Keeps Generated Docs Honest", "summary": "A developer introduced a test-first documentation workflow that treats documentation like code, using automated checks to keep AI-generated docs honest. The approach includes executable doctests, CI enforcement of docstring coverage, and a separation between model drafting and human review. The workflow was demonstrated with a Python artifact and validation scripts, and the article was prepared as part of MonkeyCode's product outreach.", "body_md": "AI code assistants have made documentation fast to produce and easy to ignore. The issue is not speed; it's trust. Models can write a polished docstring that describes a function that no longer exists, or an example that fails on the first run. The more docs we generate, the more stale those docs become if nothing checks them.\n\nThis article describes a test-first documentation workflow. It treats documentation like code: each claim passes an automated check before it is considered done. You will find a small Python artifact that extracts docstring stubs, a validation script that catches broken examples, and a decision table that separates what a model may draft from what a human must own.\n\nConsider a typical flow: you ask a language model to document a function. It returns a docstring with an example. The example contains a parameter name that was renamed in the last commit. The docstring looks plausible, so no one questions it.\n\nThe problem is the absence of a feedback loop. Code has compilers, linters, and tests. Documentation only has the cursor and the reader's patience.\n\nThe fix is to give documentation the same feedback loop. Run the examples. Check that documented names exist. Compare the documented behavior with the actual behavior.\n\nEvery public function should have a docstring. This is simple to enforce with `pydocstyle`\n\nor a tiny AST script. If a new function lands without a docstring, CI fails.\n\nDocstring examples must be executable. Python's `doctest`\n\nis the classic tool, but you can also build custom checks. For example:\n\n``` php\ndef format_bytes(size: int) -> str:\n    \"\"\"\n    Convert a size in bytes to a human-readable string.\n\n    >>> format_bytes(1024)\n    '1.0 KiB'\n    >>> format_bytes(1536)\n    '1.5 KiB'\n    \"\"\"\n    ...\n```\n\nRunning `python -m doctest module.py`\n\nturns those examples into tests. If the function changes behavior, the docs fail loudly.\n\nWhen a function signature changes, its docstring should be flagged. You can write a CI script that compares the set of public names defined in the code with the set of names mentioned in the documentation. Here is a minimal version:\n\n``` python\n# check_doc_coverage.py\nimport ast, pathlib, sys\n\ndef public_functions(path):\n    tree = ast.parse(pathlib.Path(path).read_text())\n    return {\n        node.name for node in ast.walk(tree)\n        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))\n        and not node.name.startswith(\"_\")\n    }\n\ndef documented_names(path):\n    text = pathlib.Path(path).read_text()\n    return {name for name in public_functions(path) if f\"`{name}`\" in text}\n\nif __name__ == \"__main__\":\n    if len(sys.argv) != 2:\n        raise SystemExit(\"usage: check_doc_coverage.py <file.py>\")\n    funcs = public_functions(sys.argv[1])\n    documented = documented_names(sys.argv[1])\n    missing = funcs - documented\n    if missing:\n        raise SystemExit(f\"Undocumented public functions: {missing}\")\n```\n\nThis is a heuristic, not a proof. A docstring could mention a function name and still be wrong. But it catches the most common drift.\n\nDrafting docstrings is a good job for a language model because it is repetitive and low-risk. The high-risk part is reviewing the result. That's why the workflow separates drafting from ownership.\n\nMonkeyCode is an open-source project that provides free model access (currently advertised as 10 million tokens, check the official README for the latest quota) and a free server option, which is useful for this kind of batch job. Instead of paying per request, you can run a nightly script that scans new public functions and sends their signatures to the model for a draft. Disclosure: This article was prepared as part of MonkeyCode's product outreach.\n\nA typical prompt might be:\n\n``` python\nGiven this function signature:\n    def calculate_total(items: list[float], tax_rate: float) -> float\nWrite a docstring with:\n1. One sentence describing what it does.\n2. Args and return sections.\n3. One doctest example.\nDo not invent edge cases.\n```\n\nThe model output should be treated as **draft**. It goes into the docstring, but the CI gates decide whether it survives.\n\nEven with three test gates, some information cannot be verified by executing examples. A human reviewer owns:\n\nThe table below summarizes what can be delegated to a model and what should stay with a human.\n\n| Documentation content | Model can draft? | Human must review? |\n|---|---|---|\n| Repeating parameter descriptions | Yes | No, if example passes |\n| Usage examples for stable functions | Yes | Yes, check for outdated calls |\n| Return-value units and ranges | No | Yes |\n| Exception and edge-case behavior | No | Yes |\n| Migration and deprecation notes | No | Yes |\n| \"Why\" explanations | No | Yes |\n\n`doctest`\n\nruns every example in the docstrings.This keeps the model's contribution useful but bounded. It also gives reviewers a checklist instead of a blank page.\n\nYou should not use this workflow if:\n\nDocumentation is code. Give it tests, give it CI gates, and give a model only the parts it can fail safely. The free tier of a tool like MonkeyCode makes the experimentation cheap, but the discipline comes from your pipeline, not from the model.", "url": "https://wpnews.pro/news/test-first-ai-documentation-a-workflow-that-keeps-generated-docs-honest", "canonical_source": "https://dev.to/datago_7777/test-first-ai-documentation-a-workflow-that-keeps-generated-docs-honest-31bn", "published_at": "2026-08-31 12:01:04+00:00", "updated_at": "2026-08-31 12:22:11.446181+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "large-language-models"], "entities": ["MonkeyCode", "Python", "doctest", "pydocstyle"], "alternates": {"html": "https://wpnews.pro/news/test-first-ai-documentation-a-workflow-that-keeps-generated-docs-honest", "markdown": "https://wpnews.pro/news/test-first-ai-documentation-a-workflow-that-keeps-generated-docs-honest.md", "text": "https://wpnews.pro/news/test-first-ai-documentation-a-workflow-that-keeps-generated-docs-honest.txt", "jsonld": "https://wpnews.pro/news/test-first-ai-documentation-a-workflow-that-keeps-generated-docs-honest.jsonld"}}