Measure Documentation Coverage for AI Agents With This Scorecard

A developer created a 100-point scorecard to measure documentation coverage for AI agents, applied to a MonkeyCode issue and pull request. The scorecard rates five dimensions—problem, reproduction, expected behavior, verification, and limitations—to determine if a work item contains enough documented evidence for an agent to act and a human to review. The developer argues that AI-generated documentation cannot replace missing operational knowledge and that the scorecard helps keep that distinction visible.

The argument that AI makes documentation obsolete confuses two different activities: producing prose and preserving operational knowledge. An agent can generate prose cheaply. It cannot recover a missing deployment condition, a forgotten modifier-click behavior, or a test nobody recorded. When evidence is absent, fluent text can make the gap harder to see. Ben Halpern's current DEV discussion https://dev.to/ben/the-myth-of-the-post-documentation-era-39al frames this as the myth of a “post-documentation era.” I want to make the product decision measurable: does the work item contain enough documented evidence for an agent to act and for a human to review it? Use a 100-point scorecard: | Dimension | Weight | What must be recoverable? | |---|---|---| | Problem | 20 | The observed failure or user need | | Reproduction | 20 | Ordered state and actions that expose it | | Expected behavior | 20 | The contract, including secondary interactions | | Verification | 20 | Checks run, environment, and result ownership | | Limitations | 20 | Unknowns, excluded cases, and evidence boundaries | Each dimension is documented 100% of its weight , partial 50% , or missing 0% . This is not a universal quality score. It is a coverage alarm for a declared unit of work. I applied the scorecard only to MonkeyCode https://github.com/chaitin/MonkeyCode issue 824 https://github.com/chaitin/MonkeyCode/issues/824 , pull request 859 https://github.com/chaitin/MonkeyCode/pull/859 , and the relevant public code at commit c58bcd4 https://github.com/chaitin/MonkeyCode/tree/c58bcd4dd4b7031f469a1271f276d22550b8f523 . This is not a score for the project or its documentation as a whole. The issue identifies a /workspace/... Markdown link returning the app home page. The PR distinguishes a normal click, which should open task file preview, from new-tab and copy-link actions, which should preserve a file-manager deep link. It also reports lint, online build, and manual Markdown-link checks. My evidence file rates that narrow case: { "name": "reproduction", "weight": 20, "status": "partial", "evidence": "Path and click are described; browser and deployment are absent" } The result is 80/100, with reproduction and limitations marked partial. The number does not declare the fix good or bad. It tells a product team where another question or test record would reduce agent guesswork. The companion zero-dependency script validates that weights total 100, statuses use the defined scale, every dimension cites evidence, and the revision is pinned. node score-docs.mjs doc-coverage.json node test-score.mjs Expected output: coverage=80/100 gaps=reproduction,limitations PASS score=80; rejected invalid weights and missing evidence The test deliberately breaks the weights and removes evidence. A spreadsheet can calculate the same number, but validation prevents a polished dashboard from hiding an invalid rubric. Different work deserves different thresholds: The useful product behavior is not “block anything below 90.” It is to change what the agent may do. Low coverage can permit investigation and test creation while forbidding an automatic merge. Run it as a product experiment, not a doctrine. Track: If teams game the status labels, require evidence URLs and sample scored packets in calibration sessions. If the score does not predict rework or review quality, change the dimensions. The broader point is simple: generated documentation can lower the cost of formatting knowledge, but it does not create missing observations. A coverage scorecard keeps that distinction visible to the agent, the reviewer, and the product manager deciding how much autonomy to grant. Disclosure: I contribute to the MonkeyCode project. The score above is a limited analysis of the linked issue, PR, and pinned code—not a project-wide documentation rating. The scoring script and fixtures were tested locally.