Test-First AI Documentation: A Workflow That Keeps Generated Docs Honest

A developer introduced a test-first documentation workflow that treats documentation like code, using automated checks to keep AI-generated docs honest. The approach includes executable doctests, CI enforcement of docstring coverage, and a separation between model drafting and human review. The workflow was demonstrated with a Python artifact and validation scripts, and the article was prepared as part of MonkeyCode's product outreach.

AI code assistants have made documentation fast to produce and easy to ignore. The issue is not speed; it's trust. Models can write a polished docstring that describes a function that no longer exists, or an example that fails on the first run. The more docs we generate, the more stale those docs become if nothing checks them. This article describes a test-first documentation workflow. It treats documentation like code: each claim passes an automated check before it is considered done. You will find a small Python artifact that extracts docstring stubs, a validation script that catches broken examples, and a decision table that separates what a model may draft from what a human must own. Consider a typical flow: you ask a language model to document a function. It returns a docstring with an example. The example contains a parameter name that was renamed in the last commit. The docstring looks plausible, so no one questions it. The problem is the absence of a feedback loop. Code has compilers, linters, and tests. Documentation only has the cursor and the reader's patience. The fix is to give documentation the same feedback loop. Run the examples. Check that documented names exist. Compare the documented behavior with the actual behavior. Every public function should have a docstring. This is simple to enforce with pydocstyle or a tiny AST script. If a new function lands without a docstring, CI fails. Docstring examples must be executable. Python's doctest is the classic tool, but you can also build custom checks. For example: php def format bytes size: int - str: """ Convert a size in bytes to a human-readable string. format bytes 1024 '1.0 KiB' format bytes 1536 '1.5 KiB' """ ... Running python -m doctest module.py turns those examples into tests. If the function changes behavior, the docs fail loudly. When a function signature changes, its docstring should be flagged. You can write a CI script that compares the set of public names defined in the code with the set of names mentioned in the documentation. Here is a minimal version: python check doc coverage.py import ast, pathlib, sys def public functions path : tree = ast.parse pathlib.Path path .read text return { node.name for node in ast.walk tree if isinstance node, ast.FunctionDef, ast.AsyncFunctionDef and not node.name.startswith " " } def documented names path : text = pathlib.Path path .read text return {name for name in public functions path if f" {name} " in text} if name == " main ": if len sys.argv = 2: raise SystemExit "usage: check doc coverage.py <file.py " funcs = public functions sys.argv 1 documented = documented names sys.argv 1 missing = funcs - documented if missing: raise SystemExit f"Undocumented public functions: {missing}" This is a heuristic, not a proof. A docstring could mention a function name and still be wrong. But it catches the most common drift. Drafting docstrings is a good job for a language model because it is repetitive and low-risk. The high-risk part is reviewing the result. That's why the workflow separates drafting from ownership. MonkeyCode is an open-source project that provides free model access currently advertised as 10 million tokens, check the official README for the latest quota and a free server option, which is useful for this kind of batch job. Instead of paying per request, you can run a nightly script that scans new public functions and sends their signatures to the model for a draft. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A typical prompt might be: python Given this function signature: def calculate total items: list float , tax rate: float - float Write a docstring with: 1. One sentence describing what it does. 2. Args and return sections. 3. One doctest example. Do not invent edge cases. The model output should be treated as draft . It goes into the docstring, but the CI gates decide whether it survives. Even with three test gates, some information cannot be verified by executing examples. A human reviewer owns: The table below summarizes what can be delegated to a model and what should stay with a human. | Documentation content | Model can draft? | Human must review? | |---|---|---| | Repeating parameter descriptions | Yes | No, if example passes | | Usage examples for stable functions | Yes | Yes, check for outdated calls | | Return-value units and ranges | No | Yes | | Exception and edge-case behavior | No | Yes | | Migration and deprecation notes | No | Yes | | "Why" explanations | No | Yes | doctest runs every example in the docstrings.This keeps the model's contribution useful but bounded. It also gives reviewers a checklist instead of a blank page. You should not use this workflow if: Documentation is code. Give it tests, give it CI gates, and give a model only the parts it can fail safely. The free tier of a tool like MonkeyCode makes the experimentation cheap, but the discipline comes from your pipeline, not from the model.