cd /news/ai-tools/turn-your-edits-into-tests · home › topics › ai-tools › article
[ARTICLE · art-173232] src=bulaev.net ↗ pub= topic=ai-tools verified=true sentiment=↑ positive

Turn your edits into tests

Every Frontier No. 045 documented a workflow in which Dan Shipper turned his edits to AI-generated Context Window newsletter drafts into automated yes/no checks, expanding eight cases into 74 checks that scored a `short-idea-summary` writing skill across nine versions. On v1, graded on the first seven cases, Opus 5.5 scored 56%, GPT-6 Astra 46%, Gemini 4 Hydrogen 40% and Sonnet 5.5 34%; on v9, against all 74 checks, Gemini reached 81.1%, Opus 78.4%, Astra 78.4% and Sonnet 75.7%, though the publication cautioned that start and finish were scored on different sets and a single model, claude-sonnet-5-5, graded blind with no human panel. Serge's Telegram channel ran the same loop in production, where `post_diff.py` logged draft-versus-published diffs into `EDITS-LOG.md`, 18 numbered lessons fed the channel rules, and a November 18 backtest over 20 draft/published pairs kept a check as a blocking gate only if it failed drafts and passed published posts.

recommendation

Log every draft-versus-published diff, turn each repeated edit into a yes/no check, and backtest all checks on both sides of your pairs; keep a check blocking only if it fails drafts and passes published posts.

more in playbook: Evals and testing for LLM apps
by read5 min views2 publishedNov 20, 2026
Turn your edits into tests
Image: Bulaev (auto-discovered)

Every edit you make to an AI draft is a test case you have not written down yet.

What Every measured #

In Every Frontier No. 045, Dan Shipper saved real drafts of his Context Window newsletter in Every Checks, together with his edits and gut reactions. Each case holds the prompt, the gold case (the version they actually wanted to publish) and the feedback behind it. Every reaction became a yes/no question, for example "does the opening explain what is being corrected and why a reader would want a test?" Eight cases gave 74 checks.

A writing skill, short-idea-summary, went through nine versions. Each version ran in isolated, zero-context sandboxes on four models, and claude-sonnet-5-5 graded the outputs blind. On v1, scored on the first seven cases, Opus 5.5 got 56%, GPT-6 Astra 46%, Gemini 4 Hydrogen 40% and Sonnet 5.5 34%. On v9, against all 74 checks: Gemini 81.1%, Opus 78.4%, Astra 78.4%, Sonnet 75.7%.

Read the numbers with care. Start and finish were scored on different sets. One model did the grading, with no human panel. Eight cases carry one editor's taste. The headline says 25+ points, and for Opus the gain is about 22.

The same loop on a Telegram channel #

Serge's Telegram channel runs the same loop in production. A script, post_diff.py, compares each channel draft with the post he actually published, writes the diff to drafts/posts/edits/ and adds a row to EDITS-LOG.md. Edits that repeat become numbered lessons in the channel rules, 18 so far, and the daily routines read them before writing. On November 18 the lessons became about two dozen yes/no checks: regex checks for things code can see (placeholders, backticks, dashes, the signature) and questions for a judge model, sent in one claude -p --model sonnet call per draft.

To repeat it:

  1. Keep pairs: what the AI wrote and what you shipped. Serge's backtest used 20.
  2. Write each repeated edit as a question where "yes" is good, with a source field naming the edit it came from.
  3. Run every check on both sides of every pair. The backtest in run_checks.py counts, per check:
df = sum(not rows[(s, "draft")][k]["pass"] for s, _, _ in pairs if k in rows[(s, "draft")])
pf = sum(not rows[(s, "published")][k]["pass"] for s, _, _ in pairs if k in rows[(s, "published")])
caught = sum(k in out["per_post"][s]["caught"] for s, _, _ in pairs)
  1. A check stays blocking ( gate ) only if it fails drafts and passes what you actually published. Otherwise it becomes a warning (watch ).
  2. Every new repeated edit becomes a new check, then rerun the backtest.

On 20 pairs, drafts already passed about nine checks in ten, because the lessons were in the prompts. The useful number was elsewhere: about twenty edits, in more than half of the posts, where a check failed the draft and passed the published post. Placeholders and backticks were caught every time. "No insider references" failed published posts more often than drafts, because Serge keeps names his readers know. That check, and the one against short chopped sentences, now only warn.

The loop closed twice on the first day. The checklist stopped a new draft that told Claude's actions in Serge's voice. Then Serge edited a draft about Papertrails (more on it below): it was missing a link to the author's profile and to the tweet with the prompt. Within the hour that was lesson 18 and a gate check called x-author-links.

The visual version: a blind critic #

The same idea works where there is no text at all. Let a blind critic decide when you are finished. Shikhar (@xikhar) built Papertrails, an endless pop-up paper world in the browser, with Fable 5.2 and Opus 5.5 on three.js. The code is on GitHub under MIT, and you can play it. He posted the prompt. The prompt starts with no reference: it tells the agent to generate 4 to 6 concept frames and collect photos of real pop-up books, and those become the target. The "How to work" section, word for word:

autonomous, no questions; log assumptions. Study reference → checklist of
components with measurable targets. One sub-agent per component, each owns
its files, project stays running. Loop on every component: separate harsh
blind critic, A/B side by side vs reference in random order, says which is
better + fixes. Continue until critics can't reliably pick the original and
are wowed. Also judge fold motion/spring feel, walking feel, sound,
performance (steady 60 fps mid-range laptop). Revert anything that makes it
worse. Append-only log. User's explicit instructions override critics.

The same move for posts: the gold reference can be your own best posts. Show a critic your draft and a published post in random order, and stop when it can no longer tell which one you wrote.

The other half: the writer #

Checks catch what a draft gets wrong. On November 16, Every's head of evals Mike Taylor published the same loop pointed at the writer. He turned each of his posts into a brief with the substance and none of the style, had a judge compare two drafts against the real post (twice, with the order swapped, because judges favor whichever draft they read first), and let an optimizer rewrite his writing skill. A style guide written by Claude lost to the optimized skills on 80 to 95% of held-out posts. With its optimized skill, Opus 5.5 went from 16% to 93%. His rule for when AI writing is defensible: your input tokens outnumber your output tokens.

Try it this week #

Start with ten pairs: what the AI wrote and what you shipped. Write down the edits that repeat as yes/no questions. Run them on both sides and keep only the ones that agree with your published work. Reply with the edit you make most often: the most common ones will become checks in a future issue.

Serge builds Publora and the open-source LinkedIn skills for Claude Code.

── more in #ai-tools 4 stories · sorted by recency
── more on @every frontier 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/turn-your-edits-into…] indexed:0 read:5min 2026-11-20 · —