cd /news/ai-agents/stop-your-docs-drifting-from-your-co… · home › topics › ai-agents › article
[ARTICLE · art-172631] src=tessl.io ↗ pub= topic=ai-agents verified=true sentiment=↑ positive

Stop your docs drifting from your code: catch it on every change

Tessl published the first post in a three-part series on building a "docs-sync loop" that keeps a command-line tool's README in sync with its code, pairing a deterministic Node.js test that runs the CLI and compares its commands against the README with a Tessl verifier, a plain-English rule an LLM checks each change against. Tessl defines a loop as an agent process that takes a defined input, runs an agent in a container with only the access and secrets that job needs, and produces a defined output, and argues that checking the output is the real work that makes a loop worth running rather than "just another cron job." The post covers only Part 1, the check itself; the loop that opens a pull request to fix drift is slated for Part 2.

recommendation

Add a Node.js test that runs the CLI and compares its commands to the README, plus an LLM-checked verifier rule, so every change that forgets the docs fails before merge.

more in playbook: Working with AI coding agents
by read14 min views3 publishedNov 19, 2026
Stop your docs drifting from your code: catch it on every change
Image: Tessl (auto-discovered)

ARTICLE

Chandler Mayo

TL;DR: A loop is an agent process that does one job, with a defined input and output, in an environment you control. A docs-sync loop, for example, compares a CLI's README with its code and opens a pull request when they've drifted. But a loop is only as good as your ability to trust it, so before you build one, you need a check you can trust. In this first post of a three-part series, you'll build that check: a test and a Tessl verifier that check every change to a CLI's docs. You'll watch it fail when it should and pass when it should, and by the end you'll have a check that's useful on its own, before there's a loop to put it in.

The problem: agents ship code faster than anyone can keep the chores around it in shape.

Your coding agent can ship a feature in minutes. The chores around it are a different story: docs drift, tests get flaky, and agent instructions go stale, all week long. Who's keeping up with all of that? If you're not sure, you've already met the problem this series solves.

Doing it all by hand doesn't scale. You could ask an agent to "tidy up" every so often, but without a way to check its work, all you know is that it ran, not whether it fixed anything. There's a better way. It's called a loop, and this series walks you through building one.

What you'll get out of this: in this post, you'll build the check for a loop that keeps a CLI's docs in sync with its code. It's two checks on every change: a deterministic test, a standard Node.js test that runs the CLI and compares its commands with the README, for what a script can check; and a Tessl verifier, a rule written in plain English that an LLM checks each change against, for what takes judgment. That means changes that forget the docs get caught while you're still on the branch, including the ones your tests can't see. Better still, every loop needs a check like this, so once you've built one, the loop in Part 2 has something to stand on.

What is a loop? #

At Tessl, a loop is an agent process that does one job. It takes a defined input, runs an agent in a container with only the access and secrets that job needs, and produces a defined output. An implementor loop, for example, takes the spec from a Linear ticket, runs an agent to build the code, and opens a pull request. A reviewer loop might then check that pull request. Chain loops like these together and you get a workflow. Here, you'll build one loop: a loop that keeps a CLI's README in sync with its code. This series is all hands-on, so if you want the bigger picture first, Feedback loop engineering explains the different kinds of loops and why they matter, and Context-Driven Factories shows how they come together into a factory.

Checking the output is what matters most. Scheduling an agent is easy. Trusting what it does is the real work, and it's what makes a loop worth running, rather than just another cron job.

Docs drift makes a great first loop, because the job is small, drift is easy to spot, and you can check the result. It's also exactly where we suggest starting: pick one narrow problem, and run the loop by hand before you trust it on a schedule. Once you've built this one, Tessl's docs have more example loops to try next.

What you'll build

Across the series, you'll build a loop that keeps a CLI's README in sync with its code: it blocks changes that forget the docs, and opens a pull request to fix any drift that slips through. It comes together in three parts:

  • Part 1: A check for every change (this post). Two checks every change has to pass: a test that compares the CLI's commands with the README, and a Tessl verifier that checks the README is updated whenever a command changes.
  • Part 2: A weekly sweep. A loop you write yourself, scheduled to compare the docs with the code and open a pull request when they've drifted.
  • Part 3: Close the loop. Turning the check into a gate: running it on every pull request and blocking changes that fail, then using what the sweep finds to make it stronger.

Once it's enforced as a gate, it stops new drift. The sweep catches whatever gets past it, and each catch shows you where to tighten the gate. This post builds the check, boxes 1 and 2 in the diagram.

Prerequisites #

To run everything smoothly, here's what you need before you start.

  • Node.js and git. The example CLI has no dependencies.
  • The Tessl CLI and a free Tessl account. Install the CLI withcurl -fsSL https://get.tessl.io | sh , then runtessl login . If you don't have an account yet, you can sign up for a free one when you log in.

If anything below doesn't behave, the verifier docs have more details to help you along.

Step 1: Set up a sample app #

You need something to check, so this series uses a tiny sample CLI called tally, which you'll build first. There's nothing special about it: two commands and a README, just enough for drift to happen. Once you've seen the check work, you can (and should) point the same setup at a real project.

Set up a working directory and follow along. Create a folder with three files in it. First up, the CLI itself, which lives in bin/tally.js:

1#!/usr/bin/env node
2
3// Every command the CLI supports, with the description shown in --help
4const commands = {
5  add: 'Add a number to the running total',
6  reset: 'Set the total back to zero',
7};
8
9const [cmd] = process.argv.slice(2);
10
11if (!cmd || cmd === '--help') {
12  // Print usage and the command list
13  console.log('Usage: tally <command>\n\nCommands:');
14  for (const [name, desc] of Object.entries(commands)) console.log(`  ${name.padEnd(8)}${desc}`);
15} else if (!commands[cmd]) {
16  // Reject anything that isn't a known command
17  console.error(`Unknown command: ${cmd}. Run tally --help to see commands.`);
18  process.exit(1);
19}

Next, the docs that describe it, in README.md. Keep an eye on that last line, because it's about to become a problem:

1# tally
2A tiny CLI that keeps a running total.
3
4## Commands
5- `tally add` adds a number to the running total
6- `tally reset` sets the total back to zero

And finally, a package.json, so npm test has something to run:

1{ "name": "tally", "type": "module", "bin": { "tally": "bin/tally.js" }, "scripts": { "test": "node --test" } }

Commit it all on a branch called main, because that's what you'll compare changes against later on:

1# Create the repo and commit the three files
2git init && git add -A && git commit -m "start"
3
4# Make sure the branch is called main
5git branch -M main

Step 2: Set up Tessl #

The verifier you'll write in Step 4 runs through the Tessl CLI, so set up Tessl in the repo:

1# Set up Tessl for your coding agents
2tessl init
3
4# Commit the new files
5git add -A && git commit -m "Set up Tessl"

tessl init sets things up for whichever coding agents you use, so don't be surprised to see AGENTS.md, CLAUDE.md and a few MCP config files appear.

Step 3: Start with a deterministic check #

Before you reach for an agent, check whatever a plain old script can check. A script is faster and cheaper than an LLM, and it gives the same answer every time. "Does every documented command actually exist?" is a yes-or-no question, so it belongs in a test. Tessl's own verifier guidance has a simple rule of thumb: if a script can check it, write a test. If it takes judgment, like whether a command's description still matches what it does, write a verifier.

Add test/docs.test.js. It runs the CLI's --help, then compares the commands it lists with the ones in the README, in both directions, so it catches commands that vanished as well as ones nobody got round to documenting:

1import { test } from 'node:test';
2import assert from 'node:assert/strict';
3import { readFileSync } from 'node:fs';
4import { execFileSync } from 'node:child_process';
5
6// Commands the README mentions, written as `tally <command>`
7const readme = readFileSync('README.md', 'utf8');
8const documented = [...new Set([...readme.matchAll(/`tally ([a-z-]+)/g)].map((m) => m[1]))];
9
10// Commands the CLI actually lists when you run --help
11const help = execFileSync('node', ['bin/tally.js', '--help'], { encoding: 'utf8' });
12const actual = [...help.matchAll(/^ {2}([a-z-]+)\s/gm)].map((m) => m[1]);
13
14// Sanity check: the README parsing found something
15test('README documents at least one command', () => assert.ok(documented.length > 0));
16
17// No stale docs: nothing in the README that the CLI lacks
18test('every documented command exists in the CLI', () =>
19  assert.deepEqual(documented.filter((c) => !actual.includes(c)), []));
20
21// No missing docs: nothing in the CLI that the README lacks
22test('every CLI command is documented', () =>
23  assert.deepEqual(actual.filter((c) => !documented.includes(c)), []));

Run npm test and you should see all three tests pass. Notice that the test runs the CLI and reads its real --help output, rather than trusting another description of the commands. That way the README is checked against what users actually see, not against something that could be just as out of date.

What a test can't catch

This test can't judge behavior. It knows reset exists, but it has no idea whether "sets the total back to zero" is still what reset actually does. For that, you need something that can read and reason, like an LLM, and that's where Tessl verifiers come in.

Step 4: Add a judgment check #

A verifier is a rule written in plain text that lives in your repo. Every time it runs, an LLM judge checks the change against it and returns a pass or a fail, so it behaves like a test for the rules a script can't express. The verifiers launch post covers the background if you want it; here, you'll write one. Together with the test from Step 3, it makes up the two-part check.

Your rule is a simple one: if a change touches a command, the same change has to update the README. Add verifiers/cli-docs-sync.json:

1{
2  "name": "CLI changes update the README",
3  "instruction": "When a change adds, removes, renames or changes the behavior of a tally command, the same change updates README.md so it still describes what the command actually does.",
4  "relevant_when": "The diff changes a command name, its description, or what a command does in bin/tally.js.",
5  "context": "README.md is the only user documentation for the tally CLI; users and agents rely on it.",
6  "scope": "diff",
7  "scope_reason": "Synchronization rule: the README update must ship in the same change as the CLI change.",
8  "checklist": [
9    { "name": "readme-updated", "rule": "If the diff changes a command's name or behavior in bin/tally.js, the diff also changes README.md to describe it. Not applicable if no command changed." },
10    { "name": "readme-matches-behavior", "rule": "For each command the diff changes, README.md's description says what the code in bin/tally.js now does. A README edit that mentions the command but still describes the old behavior fails. Not applicable if no command changed." }
11  ]
12}

The important bits:

  • relevant_when - when the verifier applies. Keep it specific, so the judge only checks changes that touch commands. "The diff changes a command in bin/tally.js" is specific enough; "The diff changes JavaScript" would wake the judge for every refactor and typo fix.
  • scope - what the judge looks at. Withfile , it judges files one at a time, which suits a rule like "every API route uses the shared response helper." Withdiff , it judges the change as a whole. This rule is about two files changing together, so it needsdiff . Theverifier docs cover the other options.

The second checklist item is where judgment comes in: a script can tell that the README changed, but only something that reads both files can tell whether it now says the right thing.

Then tell Tessl which files the verifier covers, by adding a verify block to tessl.json:

1"verify": {
2  "groups": {
3    "docs-sync": {
4      "path": "verifiers/cli-docs-sync.json",
5      "level": "warn",
6      "include": ["bin/**/*.js", "README.md"]
7    }
8  }
9}

The important bits:

  • level -warn reports findings without blocking anything. You'll switch it toerror once you trust it.
  • include - the files this verifier covers. It needs both the CLI and the README, so the judge sees both sides of a change.

Lint the verifier to make sure it's valid, then commit both files:

1# Check the verifier file is valid
2tessl change verify lint verifiers/cli-docs-sync.json
3
4# Commit the verifier and the tessl.json change
5git add -A && git commit -m "Add docs-sync verifier"

Watch it fail

A check you've only ever seen pass hasn't proved anything, so let's break something the test can't see. On a new branch, change what reset does: instead of going back to zero, it now restores the last saved total. Update its description in bin/tally.js, and leave the README well alone:

1# Start a throwaway branch for the experiment
2git checkout -b drift-test
3
4# Change what reset does, without touching the README
5sed -i.bak 's/Set the total back to zero/Restore the last saved total/' bin/tally.js && rm bin/tally.js.bak
6git commit -am "reset restores the last saved total"
7
8# The test still passes: every command name still matches
9npm test
10
11# Preview what the verifier will judge, then run it for real
12tessl change verify --base main --dry-run --show-files
13tessl change verify --base main

npm test still passes, because every command name still matches. The verifier is the only thing that notices.

The dry run is a handy habit: it tells you exactly what will be judged and how many judge calls it'll make, before you spend anything. Here it plans just one. The real run then fails, and the judge tells you exactly why. You'll see your docs-sync#cli-docs-sync verifier marked as a fail, with a finding for each checklist item the change broke. An LLM writes those findings, so the wording varies from run to run, but they'll say something along these lines: the description of reset changed in bin/tally.js, and the README still says it sets the total back to zero.

A couple of things in the output can look a little odd the first time. The summary reports a warning rather than an error, because the group's level is warn, and a failing warn check won't block a pull request. It also says "0 matching files", which sounds alarming but isn't: a diff verifier judges the change as a whole, rather than file by file.

Watch it pass

Now update the README on the same branch, and run it again:

1# Bring the README in line with the new behavior
2sed -i.bak 's/sets the total back to zero/restores the last saved total/' README.md && rm README.md.bak
3git commit -am "docs: update reset description"
4
5# Both checks should pass now
6tessl change verify --base main
7npm test

This time the verifier passes, and so do the tests. Now you know the check fails when it should and passes when it should, which is the only way to really trust it. Tidy up before moving on:

1git checkout main && git branch -D drift-test

This check has a blind spot, and it's there by design. It checks changes, so it can't see drift that's already sitting in your repo, and it can't stop a change that never goes through it. That's exactly what the loop is for.

What's next #

You've built a check: a deterministic test for what a script can check, and a verifier for what takes judgment. You've seen it catch drift your tests miss, and stay quiet once the README is fixed, so you know it's worth trusting. That's worth keeping even before there's a loop: copy the test and the verifier into a project you care about, and point the include paths at your own code and docs.

In Part 2, you'll cover the check's blind spot by writing your first loop: a weekly sweep that compares the docs with the code across the whole repo, and opens a pull request when they've drifted.

COPY & SHARE

Chandler Mayo

Chandler Mayo is a Developer Advocate at Tessl. He works with developer communities to make Tessl easier to understand, adopt, and build with, and brings what he hears from developers back to the product team.

READING

·

0%

COPY & SHARE

Chandler Mayo

Chandler Mayo is a Developer Advocate at Tessl. He works with developer communities to make Tessl easier to understand, adopt, and build with, and brings what he hears from developers back to the product team.

── more in #ai-agents 4 stories · sorted by recency
── more on @tessl 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/stop-your-docs-drift…] indexed:0 read:14min 2026-11-19 · —