“My job is to write loops.”
That’s Boris Cherny, who leads Claude Code at Anthropic. He’s said he stopped prompting Claude directly and now spends his time designing the loops that prompt it for him [1]. That line, and a couple of others like it, kicked off a wave of loop-engineering explainers this year [1][2]. I read six of them. Then I built one.
Two pieces, specifically — the two that every explainer mentions and almost none actually run. Run-until-done: feed the model its own real test failures instead of asking it to guess again. Maker/checker: don’t let the model that wrote the code decide whether the code is correct.
I built both from scratch, about 600 lines of Python, wired to claude-opus-4-8, graded against MBPP+ [3]. Total spend across every experiment in this article: under two dollars. And the second piece — the one every write-up treats as the safe half, because it "actually runs tests" instead of just trusting the model's word — did something in my own numbers that none of those explainers warned me about.
TL;DR: Loop engineering’s two core pieces are simple to wire up and easy to get quietly wrong. My “real feedback” loop looked identical to random retries until I found the bug in my own test harness. My “safe” test-running verifier had ahigherfalse-accept rate than a checker that just asked the model how confident it felt. Building the loop is the easy 20%.
Most of what gets published about loop engineering stops at the wiring diagram. Trigger, verifiable goal, tools, state, stop rules — five boxes, one arrow between each, done. The implication is that once the boxes are connected, the loop works.
That’s the same logic as installing a smoke detector and calling the house safe. The detector is on the ceiling. It’s wired in. Nobody checked whether there’s a battery in it.
I hit this exact failure with my “real feedback” loop. It was wired to the actual test output, not a generic retry prompt. On paper, it should have clearly beaten a loop fed nothing but “that was wrong, try again.” My first run said otherwise.
Before touching the loop, I built the thing everything else depends on: a scorer that runs candidate code in an isolated subprocess with a hard timeout, and grades it against hidden tests.
I didn’t trust it until it graded itself. Feed it a known-good solution — it has to pass. Feed it a known-bad one — it has to fail, with the assertion error attached. Feed it an infinite loop — it has to get killed by the timeout, not hang forever.
def scorer_selftest() -> None: tests = ["assert add(2, 3) == 5", "assert add(-1, 1) == 0"] good = run_tests("def add(a, b):\n return a + b", tests) assert good["all_pass"] bad = run_tests("def add(a, b):\n return a - b", tests) assert not bad["all_pass"] and "AssertionError" in bad["stderr"] loop = run_tests("def add(a, b):\n while True:\n pass", tests, timeout_s=3) assert loop["timed_out"]
Then I validated the whole pipeline against 75 MBPP+ reference solutions. All 75 passed. Only after that did I trust a single number the loop produced.
The loop itself is almost insultingly simple. Generate a solution, grade it, and on failure, feed the real stderr back — not “try again,” the actual error — for up to three attempts.
I also built a control arm, because I didn’t want to trust a headline number without one: run the identical loop, but replace the real error with a generic “that was wrong, write a different solution.” If real feedback doesn’t clearly beat that, something in the wiring is broken.
First run, 35 problems:
single-shot (pass@1) 32/35 91.4%loop, real feedback 32/35 91.4% ← identicalloop, generic feedback 32/35 91.4%
Identical. All three arms. That’s not a loop working, that’s a red flag wearing a loop’s clothes.
I went digging into the failures instead of the headline number, and found it: MBPP+’s hidden-test harness was failing with a bare AssertionError — no failing input, no expected value, no actual value. "Real feedback" was informationally identical to "try again," because there was nothing in it the model could act on.
I instrumented the harness to report the failing input, the expected output, and what the code actually returned. Same 35 problems, second run:
single-shot (pass@1) 32/35 91.4%loop, real feedback 33/35 94.3%loop, generic feedback 32/35 91.4%
Real feedback recovered a problem the generic arm couldn’t touch, for about 2,500 extra input tokens across the run. The loop was never broken. The signal it was wired to was empty, and only the control arm surfaced that — the headline metric never would have.
The loop needs a stop rule, and “the model says it’s done” isn’t one. So I built a checker that writes its own tests from the spec — never seeing the hidden tests, never seeing its own solution’s code — and then actually runs them. Accept only on a clean sweep. Default to reject.
I compared it against three weaker checkers on 41 candidates my loop had produced, 33 correct and 8 wrong, measuring false-accept rate: how often each checker waves through code that’s actually broken.
checker false-accept false-reject trust everything 8/8–100% 0/33–0% ask the model if it’s confident 2/8–25% 4/33–12% a second model reads the code 2/8–25% 5/33–15% writes tests and runs them 3/8–38% 1/33–3%
I expected the test-running checker to win outright on false-accepts. It didn’t. It let through a higher fraction of wrong code than either opinion-based checker.
The reason mattered more than the number. All 8 wrong candidates came from three problems with genuinely ambiguous specs. The checker and the fixer were the same model reading the same ambiguous sentence — so the checker’s self-written tests encoded the identical misreading the wrong code already had, and the wrong code swept them cleanly. The opinion-based checkers “won” that comparison mostly by being generally hesitant, which is also exactly why they falsely rejected four to five times more correct code.
Running tests isn’t a free pass on false-accepts. It’s a different kind of evidence — one that comes with a specific input, an expected value, and an actual value attached, instead of a vibe. That’s what makes its 3% false-reject rate usable as a real gate. A checker that rejects 15% of good work will bury you in retries before it ever saves you a bad merge.
The final composition runs the feedback loop first; if that fails, it samples fresh candidates and lets the verifier — not the model’s confidence — decide what gets submitted, with an explicit surrender path if nothing clears the bar.
Evaluated on a held-out slice of MBPP+ I hadn’t touched while building any of the above, graded against the hidden tests independently of whatever the verifier decided:
single-shot 29/35 82.9%loop only 34/35 97.1%loop + verifier 34/35 97.1%
The loop did essentially all the work: +14.2 points, five of six single-shot failures recovered, for about ten extra API calls. The verifier stage fired exactly once, on the one problem the loop couldn’t crack — and its first sampled candidate swept its own self-written tests and still failed the hidden tests. A live false-accept, on precisely the failure mode the earlier table predicted. It only got caught because the runner grades submissions against a source of truth the verifier never sees. If the verifier’s own judgment had been the final word, that bug would have shipped.
Total cost for this stage: 45 calls, roughly thirteen cents.
This works when the goal is genuinely testable — a function with hidden test cases, a schema that either validates or doesn’t, an oracle the loop can’t talk its way around. It does not work when the spec itself is ambiguous, because a same-model checker inherits the same misreading the generator had. If you’re hitting that, the fix isn’t a cleverer checker — it’s a clearer spec, or an independent oracle from a different model family entirely.
I also only tested this on small, standalone MBPP+ functions. Nothing here touches a large codebase with cross-file dependencies, and I didn’t build or test worktree isolation for running multiple loops in parallel — that’s a real problem, just not the one this experiment measured.
And this loop stops at “verified.” It doesn’t decide whether to auto-apply a change or escalate it to a human, which is a separate, harder problem the moment any of this touches something with real write access.
Start with the grader. Write the self-test — known-good, known-bad, infinite-loop — before you write a single line of loop logic. That’s a five-minute script and it’s the only thing standing between you and a fictional headline number.
If you’ve got a codebase with even a handful of unit tests, wire the feedback loop next, and build the generic-retry control arm alongside it from day one. Don’t trust the improvement until you’ve seen it beat “try again.”
Then build a second checker and put it next to your first one. The moment your own false-accept table disagrees with what the theory told you to expect — and it probably will — you’ll understand exactly why the loop matters more than the model underneath it.
[1] Rohan Mistry, “Prompt Engineering Is Dead. Loop Engineering Is Here.,” Towards AI, July 2026. https://pub.towardsai.net/prompt-engineering-is-dead-loop-engineering-is-here-32191dd20066
[2] Mehmet Özel, “Loop Engineering for AI Agents: Building Verifiable, Self-Correcting Coding Workflows,” Towards AI, June 2026. https://pub.towardsai.net/loop-engineering-for-ai-agents-building-verifiable-self-correcting-coding-workflows-8b32c72184a1
[3] EvalPlus, “MBPP+ Dataset,” Hugging Face Datasets. https://huggingface.co/datasets/evalplus/mbppplus
Loop Engineering — Simplified. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.