Test-First AI Documentation: A Workflow That Keeps Generated Docs Honest

AI code assistants have made documentation fast to produce and easy to ignore. The issue is not speed; it's trust. Models can write a polished docstring that describes a function that no longer exists, or an example that fails on the first run. The more docs we generate, the more stale those docs become if nothing checks them.

This article describes a test-first documentation workflow. It treats documentation like code: each claim passes an automated check before it is considered done. You will find a small Python artifact that extracts docstring stubs, a validation script that catches broken examples, and a decision table that separates what a model may draft from what a human must own.

Consider a typical flow: you ask a language model to document a function. It returns a docstring with an example. The example contains a parameter name that was renamed in the last commit. The docstring looks plausible, so no one questions it.

The problem is the absence of a feedback loop. Code has compilers, linters, and tests. Documentation only has the cursor and the reader's patience.

The fix is to give documentation the same feedback loop. Run the examples. Check that documented names exist. Compare the documented behavior with the actual behavior.

Every public function should have a docstring. This is simple to enforce with pydocstyle

or a tiny AST script. If a new function lands without a docstring, CI fails.

Docstring examples must be executable. Python's doctest

is the classic tool, but you can also build custom checks. For example:

def format_bytes(size: int) -> str:
    """
    Convert a size in bytes to a human-readable string.

    >>> format_bytes(1024)
    '1.0 KiB'
    >>> format_bytes(1536)
    '1.5 KiB'
    """
    ...

Running python -m doctest module.py

turns those examples into tests. If the function changes behavior, the docs fail loudly.

When a function signature changes, its docstring should be flagged. You can write a CI script that compares the set of public names defined in the code with the set of names mentioned in the documentation. Here is a minimal version:

import ast, pathlib, sys

def public_functions(path):
    tree = ast.parse(pathlib.Path(path).read_text())
    return {
        node.name for node in ast.walk(tree)
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
        and not node.name.startswith("_")
    }

def documented_names(path):
    text = pathlib.Path(path).read_text()
    return {name for name in public_functions(path) if f"`{name}`" in text}

if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("usage: check_doc_coverage.py <file.py>")
    funcs = public_functions(sys.argv[1])
    documented = documented_names(sys.argv[1])
    missing = funcs - documented
    if missing:
        raise SystemExit(f"Undocumented public functions: {missing}")

This is a heuristic, not a proof. A docstring could mention a function name and still be wrong. But it catches the most common drift.

Drafting docstrings is a good job for a language model because it is repetitive and low-risk. The high-risk part is reviewing the result. That's why the workflow separates drafting from ownership.

MonkeyCode is an open-source project that provides free model access (currently advertised as 10 million tokens, check the official README for the latest quota) and a free server option, which is useful for this kind of batch job. Instead of paying per request, you can run a nightly script that scans new public functions and sends their signatures to the model for a draft. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A typical prompt might be:

Given this function signature:
    def calculate_total(items: list[float], tax_rate: float) -> float
Write a docstring with:
1. One sentence describing what it does.
2. Args and return sections.
3. One doctest example.
Do not invent edge cases.

The model output should be treated as draft. It goes into the docstring, but the CI gates decide whether it survives.

Even with three test gates, some information cannot be verified by executing examples. A human reviewer owns:

The table below summarizes what can be delegated to a model and what should stay with a human.

doctest

runs every example in the docstrings.This keeps the model's contribution useful but bounded. It also gives reviewers a checklist instead of a blank page.

You should not use this workflow if:

Documentation is code. Give it tests, give it CI gates, and give a model only the parts it can fail safely. The free tier of a tool like MonkeyCode makes the experimentation cheap, but the discipline comes from your pipeline, not from the model.