{"slug": "stop-your-docs-drifting-from-your-code-catch-it-on-every-change", "title": "Stop your docs drifting from your code: catch it on every change", "summary": "Tessl published the first post in a three-part series on building a \"docs-sync loop\" that keeps a command-line tool's README in sync with its code, pairing a deterministic Node.js test that runs the CLI and compares its commands against the README with a Tessl verifier, a plain-English rule an LLM checks each change against. Tessl defines a loop as an agent process that takes a defined input, runs an agent in a container with only the access and secrets that job needs, and produces a defined output, and argues that checking the output is the real work that makes a loop worth running rather than \"just another cron job.\" The post covers only Part 1, the check itself; the loop that opens a pull request to fix drift is slated for Part 2.", "body_md": "ARTICLE\n\n# Stop your docs drifting from your code: catch it on every change\n\nChandler Mayo\n\n**TL;DR:** A loop is an agent process that does one job, with a defined input and output, in an environment you control. A docs-sync loop, for example, compares a CLI's README with its code and opens a pull request when they've drifted. But a loop is only as good as your ability to trust it, so before you build one, you need a check you can trust. In this first post of a three-part series, you'll build that check: a test and a Tessl verifier that check every change to a CLI's docs. You'll watch it fail when it should and pass when it should, and by the end you'll have a check that's useful on its own, before there's a loop to put it in.\n\n**The problem: agents ship code faster than anyone can keep the chores around it in shape.**\n\nYour coding agent can ship a feature in minutes. The chores around it are a different story: docs drift, tests get flaky, and agent instructions go stale, all week long. Who's keeping up with all of that? If you're not sure, you've already met the problem this series solves.\n\nDoing it all by hand doesn't scale. You could ask an agent to \"tidy up\" every so often, but without a way to check its work, all you know is that it ran, not whether it fixed anything. There's a better way. It's called a loop, and this series walks you through building one.\n\n**What you'll get out of this:** in this post, you'll build the check for a loop that keeps a CLI's docs in sync with its code. It's two checks on every change: a deterministic test, a standard Node.js test that runs the CLI and compares its commands with the README, for what a script can check; and a [Tessl verifier](https://docs.tessl.io/codifying-and-enforcing/verifiers-overview), a rule written in plain English that an LLM checks each change against, for what takes judgment. That means changes that forget the docs get caught while you're still on the branch, including the ones your tests can't see. Better still, every loop needs a check like this, so once you've built one, the loop in Part 2 has something to stand on.\n\n## What is a loop?\n\nAt Tessl, a loop is an agent process that does one job. It takes a defined input, runs an agent in a container with only the access and secrets that job needs, and produces a defined output. An implementor loop, for example, takes the spec from a Linear ticket, runs an agent to build the code, and opens a pull request. A reviewer loop might then check that pull request. Chain loops like these together and you get a workflow. Here, you'll build one loop: a loop that keeps a CLI's README in sync with its code. This series is all hands-on, so if you want the bigger picture first, [Feedback loop engineering](https://tessl.io/blog/feedback-loop-engineering-making-your-project-agent-ready) explains the different kinds of loops and why they matter, and [Context-Driven Factories](https://tessl.io/blog/context-driven-factories) shows how they come together into a factory.\n\nChecking the output is what matters most. Scheduling an agent is easy. Trusting what it does is the real work, and it's what makes a loop worth running, rather than just another cron job.\n\nDocs drift makes a great first loop, because the job is small, drift is easy to spot, and you can check the result. It's also exactly where we suggest starting: pick one narrow problem, and run the loop by hand before you trust it on a schedule. Once you've built this one, Tessl's docs have [more example loops](https://docs.tessl.io/automations/overview/overview/example-schedules) to try next.\n\n### What you'll build\n\nAcross the series, you'll build a loop that keeps a CLI's README in sync with its code: it blocks changes that forget the docs, and opens a pull request to fix any drift that slips through. It comes together in three parts:\n\n- **Part 1: A check for every change (this post).** Two checks every change has to pass: a test that compares the CLI's commands with the README, and a Tessl verifier that checks the README is updated whenever a command changes.\n- **Part 2: A weekly sweep.** A loop you write yourself, scheduled to compare the docs with the code and open a pull request when they've drifted.\n- **Part 3: Close the loop.** Turning the check into a gate: running it on every pull request and blocking changes that fail, then using what the sweep finds to make it stronger.\n\nOnce it's enforced as a gate, it stops new drift. The sweep catches whatever gets past it, and each catch shows you where to tighten the gate. This post builds the check, boxes 1 and 2 in the diagram.\n\n## Prerequisites\n\nTo run everything smoothly, here's what you need before you start.\n\n- **Node.js and git.** The example CLI has no dependencies.\n- **The Tessl CLI and a free Tessl account.** Install the CLI with`curl -fsSL https://get.tessl.io | sh` , then run`tessl login` . If you don't have an account yet, you can sign up for a free one when you log in.\n\nIf anything below doesn't behave, the [verifier docs](https://docs.tessl.io/codifying-and-enforcing/verifiers-overview) have more details to help you along.\n\n## Step 1: Set up a sample app\n\nYou need something to check, so this series uses a tiny sample CLI called `tally`, which you'll build first. There's nothing special about it: two commands and a README, just enough for drift to happen. Once you've seen the check work, you can (and should) point the same setup at a real project.\n\nSet up a working directory and follow along. Create a folder with three files in it. First up, the CLI itself, which lives in `bin/tally.js`:\n\n```\n1#!/usr/bin/env node\n2\n3// Every command the CLI supports, with the description shown in --help\n4const commands = {\n5  add: 'Add a number to the running total',\n6  reset: 'Set the total back to zero',\n7};\n8\n9const [cmd] = process.argv.slice(2);\n10\n11if (!cmd || cmd === '--help') {\n12  // Print usage and the command list\n13  console.log('Usage: tally <command>\\n\\nCommands:');\n14  for (const [name, desc] of Object.entries(commands)) console.log(`  ${name.padEnd(8)}${desc}`);\n15} else if (!commands[cmd]) {\n16  // Reject anything that isn't a known command\n17  console.error(`Unknown command: ${cmd}. Run tally --help to see commands.`);\n18  process.exit(1);\n19}\n```\n\nNext, the docs that describe it, in `README.md`. Keep an eye on that last line, because it's about to become a problem:\n\n```\n1# tally\n2A tiny CLI that keeps a running total.\n3\n4## Commands\n5- `tally add` adds a number to the running total\n6- `tally reset` sets the total back to zero\n```\n\nAnd finally, a `package.json`, so `npm test` has something to run:\n\n```\n1{ \"name\": \"tally\", \"type\": \"module\", \"bin\": { \"tally\": \"bin/tally.js\" }, \"scripts\": { \"test\": \"node --test\" } }\n```\n\nCommit it all on a branch called `main`, because that's what you'll compare changes against later on:\n\n```\n1# Create the repo and commit the three files\n2git init && git add -A && git commit -m \"start\"\n3\n4# Make sure the branch is called main\n5git branch -M main\n```\n\n## Step 2: Set up Tessl\n\nThe verifier you'll write in Step 4 runs through the Tessl CLI, so set up Tessl in the repo:\n\n```\n1# Set up Tessl for your coding agents\n2tessl init\n3\n4# Commit the new files\n5git add -A && git commit -m \"Set up Tessl\"\n```\n\n`tessl init` sets things up for whichever coding agents you use, so don't be surprised to see `AGENTS.md`, `CLAUDE.md` and a few MCP config files appear.\n\n## Step 3: Start with a deterministic check\n\nBefore you reach for an agent, check whatever a plain old script can check. A script is faster and cheaper than an LLM, and it gives the same answer every time. \"Does every documented command actually exist?\" is a yes-or-no question, so it belongs in a test. Tessl's own [verifier guidance](https://docs.tessl.io/codifying-and-enforcing/verifiers-overview) has a simple rule of thumb: if a script can check it, write a test. If it takes judgment, like whether a command's description still matches what it does, write a verifier.\n\nAdd `test/docs.test.js`. It runs the CLI's `--help`, then compares the commands it lists with the ones in the README, in both directions, so it catches commands that vanished as well as ones nobody got round to documenting:\n\n``` js\n1import { test } from 'node:test';\n2import assert from 'node:assert/strict';\n3import { readFileSync } from 'node:fs';\n4import { execFileSync } from 'node:child_process';\n5\n6// Commands the README mentions, written as `tally <command>`\n7const readme = readFileSync('README.md', 'utf8');\n8const documented = [...new Set([...readme.matchAll(/`tally ([a-z-]+)/g)].map((m) => m[1]))];\n9\n10// Commands the CLI actually lists when you run --help\n11const help = execFileSync('node', ['bin/tally.js', '--help'], { encoding: 'utf8' });\n12const actual = [...help.matchAll(/^ {2}([a-z-]+)\\s/gm)].map((m) => m[1]);\n13\n14// Sanity check: the README parsing found something\n15test('README documents at least one command', () => assert.ok(documented.length > 0));\n16\n17// No stale docs: nothing in the README that the CLI lacks\n18test('every documented command exists in the CLI', () =>\n19  assert.deepEqual(documented.filter((c) => !actual.includes(c)), []));\n20\n21// No missing docs: nothing in the CLI that the README lacks\n22test('every CLI command is documented', () =>\n23  assert.deepEqual(actual.filter((c) => !documented.includes(c)), []));\n```\n\nRun `npm test` and you should see all three tests pass. Notice that the test runs the CLI and reads its real `--help` output, rather than trusting another description of the commands. That way the README is checked against what users actually see, not against something that could be just as out of date.\n\n### What a test can't catch\n\nThis test can't judge behavior. It knows `reset` exists, but it has no idea whether \"sets the total back to zero\" is still what `reset` actually does. For that, you need something that can read and reason, like an LLM, and that's where Tessl verifiers come in.\n\n## Step 4: Add a judgment check\n\nA verifier is a rule written in plain text that lives in your repo. Every time it runs, an LLM judge checks the change against it and returns a pass or a fail, so it behaves like a test for the rules a script can't express. The [verifiers launch post](https://tessl.io/blog/verifiers-enforce-your-repos-rules-on-every-agent-change) covers the background if you want it; here, you'll write one. Together with the test from Step 3, it makes up the two-part check.\n\nYour rule is a simple one: if a change touches a command, the same change has to update the README. Add `verifiers/cli-docs-sync.json`:\n\n```\n1{\n2  \"name\": \"CLI changes update the README\",\n3  \"instruction\": \"When a change adds, removes, renames or changes the behavior of a tally command, the same change updates README.md so it still describes what the command actually does.\",\n4  \"relevant_when\": \"The diff changes a command name, its description, or what a command does in bin/tally.js.\",\n5  \"context\": \"README.md is the only user documentation for the tally CLI; users and agents rely on it.\",\n6  \"scope\": \"diff\",\n7  \"scope_reason\": \"Synchronization rule: the README update must ship in the same change as the CLI change.\",\n8  \"checklist\": [\n9    { \"name\": \"readme-updated\", \"rule\": \"If the diff changes a command's name or behavior in bin/tally.js, the diff also changes README.md to describe it. Not applicable if no command changed.\" },\n10    { \"name\": \"readme-matches-behavior\", \"rule\": \"For each command the diff changes, README.md's description says what the code in bin/tally.js now does. A README edit that mentions the command but still describes the old behavior fails. Not applicable if no command changed.\" }\n11  ]\n12}\n```\n\nThe important bits:\n\n- `relevant_when` - when the verifier applies. Keep it specific, so the judge only checks changes that touch commands. \"The diff changes a command in bin/tally.js\" is specific enough; \"The diff changes JavaScript\" would wake the judge for every refactor and typo fix.\n- `scope` - what the judge looks at. With`file` , it judges files one at a time, which suits a rule like \"every API route uses the shared response helper.\" With`diff` , it judges the change as a whole. This rule is about two files changing together, so it needs`diff` . The[verifier docs](https://docs.tessl.io/codifying-and-enforcing/verifiers-overview) cover the other options.\n\nThe second checklist item is where judgment comes in: a script can tell that the README changed, but only something that reads both files can tell whether it now says the right thing.\n\nThen tell Tessl which files the verifier covers, by adding a `verify` block to `tessl.json`:\n\n```\n1\"verify\": {\n2  \"groups\": {\n3    \"docs-sync\": {\n4      \"path\": \"verifiers/cli-docs-sync.json\",\n5      \"level\": \"warn\",\n6      \"include\": [\"bin/**/*.js\", \"README.md\"]\n7    }\n8  }\n9}\n```\n\nThe important bits:\n\n- `level` -`warn` reports findings without blocking anything. You'll switch it to`error` once you trust it.\n- `include` - the files this verifier covers. It needs both the CLI and the README, so the judge sees both sides of a change.\n\nLint the verifier to make sure it's valid, then commit both files:\n\n```\n1# Check the verifier file is valid\n2tessl change verify lint verifiers/cli-docs-sync.json\n3\n4# Commit the verifier and the tessl.json change\n5git add -A && git commit -m \"Add docs-sync verifier\"\n```\n\n### Watch it fail\n\nA check you've only ever seen pass hasn't proved anything, so let's break something the test can't see. On a new branch, change what `reset` does: instead of going back to zero, it now restores the last saved total. Update its description in `bin/tally.js`, and leave the README well alone:\n\n```\n1# Start a throwaway branch for the experiment\n2git checkout -b drift-test\n3\n4# Change what reset does, without touching the README\n5sed -i.bak 's/Set the total back to zero/Restore the last saved total/' bin/tally.js && rm bin/tally.js.bak\n6git commit -am \"reset restores the last saved total\"\n7\n8# The test still passes: every command name still matches\n9npm test\n10\n11# Preview what the verifier will judge, then run it for real\n12tessl change verify --base main --dry-run --show-files\n13tessl change verify --base main\n```\n\n`npm test` still passes, because every command name still matches. The verifier is the only thing that notices.\n\nThe dry run is a handy habit: it tells you exactly what will be judged and how many judge calls it'll make, before you spend anything. Here it plans just one. The real run then fails, and the judge tells you exactly why. You'll see your `docs-sync#cli-docs-sync` verifier marked as a fail, with a finding for each checklist item the change broke. An LLM writes those findings, so the wording varies from run to run, but they'll say something along these lines: the description of `reset` changed in `bin/tally.js`, and the README still says it sets the total back to zero.\n\nA couple of things in the output can look a little odd the first time. The summary reports a warning rather than an error, because the group's `level` is `warn`, and a failing `warn` check won't block a pull request. It also says \"0 matching files\", which sounds alarming but isn't: a `diff` verifier judges the change as a whole, rather than file by file.\n\n### Watch it pass\n\nNow update the README on the same branch, and run it again:\n\n```\n1# Bring the README in line with the new behavior\n2sed -i.bak 's/sets the total back to zero/restores the last saved total/' README.md && rm README.md.bak\n3git commit -am \"docs: update reset description\"\n4\n5# Both checks should pass now\n6tessl change verify --base main\n7npm test\n```\n\nThis time the verifier passes, and so do the tests. Now you know the check fails when it should and passes when it should, which is the only way to really trust it. Tidy up before moving on:\n\n```\n1git checkout main && git branch -D drift-test\n```\n\nThis check has a blind spot, and it's there by design. It checks changes, so it can't see drift that's already sitting in your repo, and it can't stop a change that never goes through it. That's exactly what the loop is for.\n\n## What's next\n\nYou've built a check: a deterministic test for what a script can check, and a verifier for what takes judgment. You've seen it catch drift your tests miss, and stay quiet once the README is fixed, so you know it's worth trusting. That's worth keeping even before there's a loop: copy the test and the verifier into a project you care about, and point the `include` paths at your own code and docs.\n\nIn Part 2, you'll cover the check's blind spot by writing your first loop: a weekly sweep that compares the docs with the code across the whole repo, and opens a pull request when they've drifted.\n\nCOPY & SHARE\n\nChandler Mayo\n\nChandler Mayo is a Developer Advocate at Tessl. He works with developer communities to make Tessl easier to understand, adopt, and build with, and brings what he hears from developers back to the product team.\n\nREADING\n\n·\n\n0%\n\nCOPY & SHARE\n\nChandler Mayo\n\nChandler Mayo is a Developer Advocate at Tessl. He works with developer communities to make Tessl easier to understand, adopt, and build with, and brings what he hears from developers back to the product team.", "url": "https://wpnews.pro/news/stop-your-docs-drifting-from-your-code-catch-it-on-every-change", "canonical_source": "https://tessl.io/blog/stop-your-docs-drifting-from-your-code-catch-it-on-every-change", "published_at": "2026-11-19 14:52:16+00:00", "updated_at": "2026-11-19 15:46:25.836116+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["Tessl", "Chandler Mayo", "Node.js", "Linear"], "also_reported_by": [], "takeaway": "Pair a deterministic Node.js test that runs the CLI and diffs its commands against the README with a plain-English Tessl verifier an LLM checks each change against, so docs drift is caught on the branch before merge.", "recommendation": {"advice": "Add a Node.js test that runs the CLI and compares its commands to the README, plus an LLM-checked verifier rule, so every change that forgets the docs fails before merge.", "playbook": "coding-agents", "playbook_url": "https://wpnews.pro/playbooks/coding-agents"}, "alternates": {"html": "https://wpnews.pro/news/stop-your-docs-drifting-from-your-code-catch-it-on-every-change", "markdown": "https://wpnews.pro/news/stop-your-docs-drifting-from-your-code-catch-it-on-every-change.md", "text": "https://wpnews.pro/news/stop-your-docs-drifting-from-your-code-catch-it-on-every-change.txt", "jsonld": "https://wpnews.pro/news/stop-your-docs-drifting-from-your-code-catch-it-on-every-change.jsonld"}}