{"slug": "turn-your-edits-into-tests", "title": "Turn your edits into tests", "summary": "Every Frontier No. 045 documented a workflow in which Dan Shipper turned his edits to AI-generated Context Window newsletter drafts into automated yes/no checks, expanding eight cases into 74 checks that scored a `short-idea-summary` writing skill across nine versions. On v1, graded on the first seven cases, Opus 5.5 scored 56%, GPT-6 Astra 46%, Gemini 4 Hydrogen 40% and Sonnet 5.5 34%; on v9, against all 74 checks, Gemini reached 81.1%, Opus 78.4%, Astra 78.4% and Sonnet 75.7%, though the publication cautioned that start and finish were scored on different sets and a single model, claude-sonnet-5-5, graded blind with no human panel. Serge's Telegram channel ran the same loop in production, where `post_diff.py` logged draft-versus-published diffs into `EDITS-LOG.md`, 18 numbered lessons fed the channel rules, and a November 18 backtest over 20 draft/published pairs kept a check as a blocking gate only if it failed drafts and passed published posts.", "body_md": "Every edit you make to an AI draft is a test case you have not written down yet.\n\n## What Every measured\n\nIn Every Frontier [No. 045](https://frontier.every.to/experiments/turn-your-daily-newsletter-edits-into-ai-checks-and-watch-every-frontier-model-jump-25-points), Dan Shipper saved real drafts of his Context Window newsletter in Every Checks, together with his edits and gut reactions. Each case holds the prompt, the gold case (the version they actually wanted to publish) and the feedback behind it. Every reaction became a yes/no question, for example \"does the opening explain what is being corrected and why a reader would want a test?\" Eight cases gave 74 checks.\n\nA writing skill, `short-idea-summary`, went through nine versions. Each version ran in isolated, zero-context sandboxes on four models, and claude-sonnet-5-5 graded the outputs blind. On v1, scored on the first seven cases, Opus 5.5 got 56%, GPT-6 Astra 46%, Gemini 4 Hydrogen 40% and Sonnet 5.5 34%. On v9, against all 74 checks: Gemini 81.1%, Opus 78.4%, Astra 78.4%, Sonnet 75.7%.\n\nRead the numbers with care. Start and finish were scored on different sets. One model did the grading, with no human panel. Eight cases carry one editor's taste. The headline says 25+ points, and for Opus the gain is about 22.\n\n## The same loop on a Telegram channel\n\nSerge's Telegram channel runs the same loop in production. A script, `post_diff.py`, compares each channel draft with the post he actually published, writes the diff to `drafts/posts/edits/` and adds a row to `EDITS-LOG.md`. Edits that repeat become numbered lessons in the channel rules, 18 so far, and the daily routines read them before writing. On November 18 the lessons became about two dozen yes/no checks: regex checks for things code can see (placeholders, backticks, dashes, the signature) and questions for a judge model, sent in one `claude -p --model sonnet` call per draft.\n\nTo repeat it:\n\n1. Keep pairs: what the AI wrote and what you shipped. Serge's backtest used 20.\n2. Write each repeated edit as a question where \"yes\" is good, with a `source` field naming the edit it came from.\n3. Run every check on both sides of every pair. The backtest in `run_checks.py` counts, per check:\n\n```\ndf = sum(not rows[(s, \"draft\")][k][\"pass\"] for s, _, _ in pairs if k in rows[(s, \"draft\")])\npf = sum(not rows[(s, \"published\")][k][\"pass\"] for s, _, _ in pairs if k in rows[(s, \"published\")])\ncaught = sum(k in out[\"per_post\"][s][\"caught\"] for s, _, _ in pairs)\n```\n\n1. A check stays blocking ( `gate` ) only if it fails drafts and passes what you actually published. Otherwise it becomes a warning (`watch` ).\n2. Every new repeated edit becomes a new check, then rerun the backtest.\n\nOn 20 pairs, drafts already passed about nine checks in ten, because the lessons were in the prompts. The useful number was elsewhere: about twenty edits, in more than half of the posts, where a check failed the draft and passed the published post. Placeholders and backticks were caught every time. \"No insider references\" failed published posts more often than drafts, because Serge keeps names his readers know. That check, and the one against short chopped sentences, now only warn.\n\nThe loop closed twice on the first day. The checklist stopped a new draft that told Claude's actions in Serge's voice. Then Serge edited a draft about Papertrails (more on it below): it was missing a link to the author's profile and to the tweet with the prompt. Within the hour that was lesson 18 and a gate check called `x-author-links`.\n\n## The visual version: a blind critic\n\nThe same idea works where there is no text at all. **Let a blind critic decide when you are finished.** Shikhar ([@xikhar](https://x.com/xikhar)) built [Papertrails](https://x.com/xikhar/status/2122682962647122403), an endless pop-up paper world in the browser, with Fable 5.2 and Opus 5.5 on three.js. The code is on [GitHub](https://github.com/xikhar/papertrails) under MIT, and you can [play it](https://xikhar.github.io/papertrails/). He posted [the prompt](https://x.com/xikhar/status/2122682965247639606). The prompt starts with no reference: it tells the agent to generate 4 to 6 concept frames and collect photos of real pop-up books, and those become the target. The \"How to work\" section, word for word:\n\n```\nautonomous, no questions; log assumptions. Study reference → checklist of\ncomponents with measurable targets. One sub-agent per component, each owns\nits files, project stays running. Loop on every component: separate harsh\nblind critic, A/B side by side vs reference in random order, says which is\nbetter + fixes. Continue until critics can't reliably pick the original and\nare wowed. Also judge fold motion/spring feel, walking feel, sound,\nperformance (steady 60 fps mid-range laptop). Revert anything that makes it\nworse. Append-only log. User's explicit instructions override critics.\n```\n\n**The same move for posts:** the gold reference can be your own best posts. Show a critic your draft and a published post in random order, and stop when it can no longer tell which one you wrote.\n\n## The other half: the writer\n\nChecks catch what a draft gets wrong. On November 16, Every's head of evals Mike Taylor [published](https://every.to/also-true-for-humans/how-i-finally-got-ai-to-write-like-me) the same loop pointed at the writer. He turned each of his posts into a brief with the substance and none of the style, had a judge compare two drafts against the real post (twice, with the order swapped, because judges favor whichever draft they read first), and let an optimizer rewrite his writing skill. A style guide written by Claude lost to the optimized skills on 80 to 95% of held-out posts. With its optimized skill, Opus 5.5 went from 16% to 93%. His rule for when AI writing is defensible: your input tokens outnumber your output tokens.\n\n## Try it this week\n\nStart with ten pairs: what the AI wrote and what you shipped. Write down the edits that repeat as yes/no questions. Run them on both sides and keep only the ones that agree with your published work. Reply with the edit you make most often: the most common ones will become checks in a future issue.\n\n*Serge builds [Publora](https://publora.com/?utm_source=content-engineer) and the open-source [LinkedIn skills](https://github.com/sergebulaev/linkedin-skills?utm_source=content-engineer) for Claude Code.*", "url": "https://wpnews.pro/news/turn-your-edits-into-tests", "canonical_source": "https://www.bulaev.net/p/turn-your-edits-into-tests", "published_at": "2026-11-20 12:48:34+00:00", "updated_at": "2026-11-20 13:12:47.593179+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents"], "entities": ["Every Frontier", "Dan Shipper", "Context Window", "Every Checks", "claude-sonnet-5-5", "Opus 5.5", "GPT-6 Astra", "Gemini 4 Hydrogen"], "also_reported_by": [], "takeaway": "Convert each repeated edit into a yes/no check with a `source` field, then keep it as a blocking gate only if it fails AI drafts and passes the posts you actually published — on 20 pairs this surfaced about twenty edits across more than half the posts, while checks like 'no insider references' faile", "recommendation": {"advice": "Log every draft-versus-published diff, turn each repeated edit into a yes/no check, and backtest all checks on both sides of your pairs; keep a check blocking only if it fails drafts and passes published posts.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/turn-your-edits-into-tests", "markdown": "https://wpnews.pro/news/turn-your-edits-into-tests.md", "text": "https://wpnews.pro/news/turn-your-edits-into-tests.txt", "jsonld": "https://wpnews.pro/news/turn-your-edits-into-tests.jsonld"}}