# Loop Engineering — a deep technical explanation with realistic examples > Source: > Published: 2026-07-23 16:55:36+00:00 The idea is often credited to Boris Cherny (Head of Claude Code at Anthropic), who put it bluntly: *"I don't prompt Claude anymore. I have loops running that prompt Claude and figure out what to do. My job is to write loops."* This document goes past the slogan into the mechanisms: the control-theory framing, the exact algorithms that make a loop safe, the concurrency internals, the cost math, and the epistemics of why an agent cannot be trusted to check itself. **Loop engineering is building a deterministic control loop around an AI coding agent — schedule + durable state + verification — so the agent drives the moment-to-moment cycle and you move up to designing the machine.** The single load-bearing principle: A loop is only as trustworthy as the parts of its verification the agent cannot fake. "Am I stuck?" and "is this correct?" must be decided by deterministic code or a genuinely independent checker —neverthe agent's own self-report. Everything below is a consequence of that sentence. A control system regulates a process toward a **setpoint** using feedback: ``` setpoint (goal) │ ▼ ┌──────────────┐ command ┌──────────────┐ │ controller │ ──────────► │ process │ │ (your loop) │ │(agent + repo)│ └──────────────┘ └──────────────┘ ▲ │ │ measurement │ └─────────── sensor ◄─────┘ (verifier / tests / metrics) ``` **Setpoint**= the goal ("open PRs stay green", "no failing CI on main").** Process**= the agent acting on the repo.** Sensor**= the verifier / test suite / numeric check — how you*measure*the gap between reality and setpoint.**Controller**= the loop logic that turns the measured gap into the next command. The critical distinction from control theory: | Open-loop | Closed-loop | | |---|---|---| | Definition | Act, never measure the result | Act, measure, correct | | Agent analogy | "Prompt → hope" | "Prompt → verify → decide → repeat" | | Failure mode | Drifts arbitrarily far from goal | Bounded by sensor quality | **A raw agent call is open-loop.** You prompt, it acts, and nothing measures whether the result matched intent — *you* do, manually. Loop engineering closes the loop by making the sensor (verification) and the controller (decide/escalate) part of the automated system. The corollary that governs everything: **a closed loop is only as good as its sensor.** A control loop with a broken thermometer will happily drive the room to 1000°C while reporting "72°F, all good." That broken thermometer is exactly what an LLM asked to grade its own work is (see §4). Three nested scopes. The jump from harness → loop is defined by adding **schedule + state + verification**. ``` Context engineering → what goes in ONE prompt ("one move") Harness engineering → ONE agent's environment: tools, ("one game") permissions, rules — a single session Loop engineering → harness + SCHEDULE + STATE + ("the tournament") VERIFICATION, running over TIME ``` - Miss the **schedule**→ it's a one-off script (open-loop, runs once). - Miss the **state**→ the agent is amnesiac every run and re-tries failed things forever (no integral term — no memory of accumulated error). - Miss the **verification**→ the sensor is missing; the loop is "closed" only in appearance. Three properties of language models make the naive "just let it run" approach fail. The loop machinery exists to counter each one. - **Statelessness across sessions.** A model has no memory between separate runs. Whatever isn't written to durable storage is gone. → Requires**external state**(§5A). - **Uncalibrated self-assessment.** A model's confidence is not a reliable estimate of its correctness. The same reasoning that produced a bug produces the sincere belief that the bug is correct. Confidence and correctness are drawn from*correlated*distributions, not independent ones. → Requires an**independent sensor**(§4). - **Autoregressive momentum / no natural stop.** Given "keep trying until CI is green," a model will keep generating attempts — including plausible-looking non-fixes — indefinitely, because "produce the next token" has no built-in notion of "this is hopeless, stop." → Requires an**external circuit breaker**(§5B). Loop engineering is, precisely, the set of external mechanisms that supply the memory, the sensor, and the stop condition that the model lacks. This is the load-bearing wall. Everything else is plumbing. Let `M` be the maker (produces a solution) and `V` the verifier (judges it). Verification adds information **only if V's errors are independent of M's errors.** If `V = M` (same model, same context, same prompt lineage), then any mistake `M` is blind to, `V` is *equally*blind to. The verification is a rubber stamp — it multiplies confidence without adding evidence. This is why "run the tests once and mark it done" is not verification if the same session wrote the tests and the code: a weak test written to pass weak code confirms nothing. Loops lean on a hopeful asymmetry: **checking a solution is often cheaper and more reliable than producing one** (it's easy to verify a returned sort is ordered; harder to write the sort). When this holds, an independent checker is a cheap, strong sensor. **But the asymmetry inverts when the artifact under review is itself the distortion.** The canonical case: Someone ran a hedge fund as a fully autonomous loop with an LLM verifier judging backtests. It failed because a strategy's failure mode isoverfitting— the strategy looks brilliant on historical data precisely because it was (accidentally) shaped to fit that exact history. A second LLM reading the backtestcannotcatch the curve-fit, because the backtest it's reading isalreadythe overfit artifact. The lie and the evidence-of-the-lie are the same document. The fix was to replace the LLM verifier with a **numerical, non-overridable** check that measures something the maker structurally cannot fabricate: | Test | Sharpe | Verdict | |---|---|---| | In-sample (training data) | +2.54 | looks like a winner | | Out-of-sample (held out) | −4.33 | collapses | | Numerical checker | — | REJECT — no trade | The +2.54 is exactly the number that would make a human — or an LLM verifier — ship it. The out-of-sample split is data the maker *never touched*, so it cannot have been fit to. **That untouchable measurement is the entire product.** The maker/checker split is worthless unless the checker measures something the maker cannot fake. In code, that's an independent re-derivation of correctness (a real test run, a type check, a compile, an `EXPLAIN` ); in statistics, it's out-of-sample, cost-aware, multiple-testing-penalized numbers — never a second model's vibe. | Layer | Mechanism | What it makes unfakeable | |---|---|---| Am I stuck? | Deterministic circuit breaker (§5B) | The agent can't claim "progress" — the numbers decide | Is it correct? | Separate verifier sub-agent, default REJECT | The maker can't approve its own diff | Is it real? | Non-LLM checks: tests run, EXPLAIN, compile, OOS split | The maker can't fabricate the measurement | ``` read STATE ─► TRIAGE ─► IMPLEMENT (maker sub-agent, isolated worktree) (situation?) (worth │ ▲ doing?) ▼ │ VERIFY (checker sub-agent / numeric check) │ │ persist STATE ◄─ gate/escalate ◄─┘ (write down) (safe? or human?) ``` State is a `STATE.md` (or DB row / ticket board) answering exactly three questions — and deliberately nothing more, to avoid rot: - What are we working on now? - What did we try last time, and what was the outcome? - What is waiting on a human? In control terms this is the **integral term**: it accumulates the history of error so the controller can act on *trends*, not just the latest instant. Without it, the loop is memoryless and will re-attempt the same failed fix on every run. **State rot** is the dominant failure here: the file references merged PRs, closed tickets, or dead branches, and the loop acts on ghosts. Mitigations: a prune step every run, a `Last run` timestamp, validating IDs against the live API, and one state file per loop pattern (no shared unstructured file). This is the most technically interesting component. It answers "is this agent stuck?" with **pure code — no LLM in the decision path**, so it's cheap to run every iteration and impossible for the agent to argue past. **Data model:** ``` interface Attempt { iteration: number; action: string; // short description of what was tried outcome: 'success' | 'failure' | 'noop'; error?: string; // raw error / stack trace if failed tokensUsed?: number; } interface Ledger { goal: string; attempts: Attempt[]; } // oldest → newest ``` **Step 1 — normalize errors to a stable signature.** Raw errors differ every run even when they're "the same" failure: ``` "Error at db.ts:42 connection refused (0x7ff8a2)" ← attempt 1 "Error at db.ts:88 connection refused (0x3aa10b)" ← attempt 2 ``` `errorSignature()` strips the volatile parts so the same failure maps to the same key: ``` ISO timestamps → 0x… hex addrs → absolute paths → basename only :line:col → removed any other digits → # collapse spaces ``` Both examples above collapse to `Error at db.ts connection refused` — **identical signatures.** This is the crux: without normalization the loop would think every retry is a "new" error and never notice it's stuck. **Step 2 — measure similarity with character trigrams (Jaccard).** For actions and near-miss errors, exact-match isn't enough, so it uses set overlap of overlapping 3-character substrings: ``` Jaccard(A, B) = |trigrams(A) ∩ trigrams(B)| / |trigrams(A) ∪ trigrams(B)| → 0.0 (disjoint) … 1.0 (identical) default threshold ≈ 0.85 counts as "the same" ``` Trigrams are robust to trivial rewording ("cannot connect" vs "can not connect") in a way exact string compare is not, and cheap enough to run every iteration. **Step 3 — check triggers, cheapest & most-actionable first.** It walks the **trailing run of failures** (consecutive failures since the last non-failure) and returns the *first* condition that holds, so the reported reason is the most useful one: | Order | Trigger | Fires when | Why this order | |---|---|---|---| | 1 | `stagnation` | same error signature ≥ 3× in a row | most specific, most fixable | | 2 | `frustration` | near-identical action repeated ≥ 3× in a row | semantic looping | | 3 | `no-progress` | ≥ 5 consecutive failures, no success between | broad "going nowhere" | | 4 | `token-budget` | cumulative tokens ≥ cap | cost ceiling | | 5 | `max-iterations` | ≥ 10 iterations | absolute backstop | The ordering is deliberate: stagnation ("you keep hitting the *same* wall") gives a human the most actionable escalation message; the iteration cap is a dumb backstop of last resort. **Step 4 — prune + inject.** Before the next iteration it builds a compact context block to prepend to the prompt so the agent doesn't repeat itself: ``` ## Loop Context (managed) **Goal:** **Progress:** iteration 7 · 0 ok · 7 failed · 340k tokens **Already tried (do NOT repeat):** - bump pg driver to 8.11 - add retry wrapper around connect() **Failure patterns:** - (5×) Error at db.ts connection refused **Most recent error (pruned):** ← stack trace truncated to N lines ... > Circuit breaker: OK (7/10 iterations used). ``` Pruning keeps only the last N attempts, truncates stack traces to a few lines, and **collapses repeated identical failures into one entry with a repeated count** — so a 40-line log of the same error becomes one line, keeping the next prompt cheap and focused. Separate from the breaker. The breaker knows *run history*; the gate knows *static policy* — and the two are deliberately kept apart (single responsibility; history logic lives in exactly one place). Config (`gate.yaml` ): ``` version: 1 denylist: # globs that must never be touched without a human - "**/secrets/**" - "infra/prod/**" - "**/*_key*" maxFiles: 20 # more than this in one change → escalate autoMergeAllowlist: # for auto-merge, EVERY path must match one of these - "docs/**" - "**/*.test.ts" ``` `checkGate(action, paths)` runs three checks, most-severe first: **denylist**— any changed path matches a forbidden glob → BLOCK.** file-count**— more paths than`maxFiles` → BLOCK (a huge diff is a red flag regardless of content).**allowlist**(auto-merge only) — any path*not*on the allowlist → BLOCK. **Subtle but important:** glob matching uses `dot: true` . Without it, `*` won't match a path segment starting with `.` , so `**/secrets/**` would silently miss `.secrets/prod.json` and `**/*_key*` would miss `.aws_key` — exactly the hidden files a denylist most needs to catch. This is the kind of one-character bug that turns a safety control into security theater. When multiple agents run in parallel, two editing the same file is merge hell. Each agent gets its own **git worktree** (shares history, separate working tree). On top of that sits an **advisory file-lock** system so two agents don't grab overlapping paths: - Each owner writes `.loop/locks/.json` =`{ paths, lockedAt, expiresAt? }` . **Cross-process atomicity** for the check-then-write critical section uses an exclusive-create mutex file:`open(path, 'wx')` fails if the file exists and is atomic on POSIX and Windows. Without this, two`lock` calls racing could both pass the "is it free?" check before either writes — the exact collision the feature exists to prevent.**Path overlap** is a deliberately-simple segment-by-segment glob compare (`src/**` covers`src/a/b.ts` ; a wildcard segment matches anything). Advisory, not a full glob engine.**TTL / expiry**: locks can auto-expire so a crashed agent doesn't wedge the system forever; a sweep reclaims expired locks.** Deadlock detection**: waiting agents write`.wait.json` entries recording who they're blocked on. Before waiting, the system runs**DFS cycle detection** over the wait-for graph and throws`Deadlock detected: A → B → A` instead of hanging forever. (Classic OS deadlock avoidance, applied to agents.) When the breaker trips or the gate blocks, the loop must **stop and notify** — not silently retry. The failure mode here is "escalation only writes to a state file no one reads." Real escalation pings a human through a connector (Slack, a Linear comment) and parks the item in a `Waiting on human` section, ideally with an alert if it sits there > 24h. ``` runs_per_day = 86,400,000 ms/day ÷ interval_ms tokens_per_day = runs_per_day × tokens_per_run blast_radius_per_day ∝ runs_per_day × P(bad action) × mean_damage ``` So going from a 1-day to a 5-minute cadence is **288× the runs — and 288× the cost and 288× the blast radius.** A one-off agent making a bad edit is annoying; a loop making it 288×/day unattended and merging it is an incident. Not every run costs the same. A realistic model weights three outcome types: ``` cost_per_run = P(noop) × tokens_noop (nothing to do — cheapest) + P(report)× tokens_report (found something, described it) + P(action)× tokens_action (ran maker + verifier — most expensive) ``` The mix shifts with autonomy level (more autonomy → more actions → higher cost): ``` L1 ≈ {noop .6, report .4, action 0 } L2 ≈ {noop .5, report .3, action .2} L3 ≈ {noop .4, report .35,action .25} ``` Triage cheaply first. Spawn expensive maker+verifier sub-agents onlywhen state says there's something actionable. Empty watchlist → exit in <5k tokens. A loop that runs the full sub-agent chain on every empty tick is the #1 way to turn a 5-minute cadence into a runaway bill. "Why We Killed Our CI Sweeper After Day 4": ran every 5 min on a broken branch → **8M tokens in 48h**, proposed 11 fixes, 3 were fake symptom-fixes, **1 broke prod config**. Root causes: no report-only phase, verifier in the *same session* as the maker, no budget cap, no branch allowlist. Every one of those is a missing mechanism from §5. Autonomy is earned incrementally, each rung adding a mechanism and proving it before the next: ``` L1 REPORT ONLY looks + triages, takes NO action. Run 1–2 weeks. Measure: is the triage even accurate? ↓ add: separate verifier, worktree, attempt cap, human-approved merge L2 ASSISTED FIXES proposes changes; human approves merges. ↓ add: path denylist, token budget, metrics, human gates on risk L3 UNATTENDED acts alone within hard, machine-enforced limits. ``` A readiness scorer can encode this as a **gate, not a suggestion**: it scans the project for ~35 signals (state file? verifier skill? `gate.yaml` ? budget doc? run log? evidence of real runs in git history?), sums weighted points to a 0–100 score, and maps score → level (roughly L1 ≥ 38, L2 ≥ 58, L3 ≥ 78). Crucially, an L3 score is **capped back to L2 unless** it *also* finds: a verifier, a budget file, a run log, AND git evidence you actually ran the loop. **You cannot score your way to "unattended" without proof you earned it.** | # | Primitive | Supplies | Counters (from §3) | |---|---|---|---| | 1 | Scheduling | the heartbeat (cron, `/loop 5m` , CI on a timer) | — (makes it a loop) | | 2 | Worktrees | parallel isolation + locks (§5D) | orchestration collisions | | 3 | Skills | written-down intent: conventions, build cmds | intent debt / statelessness | | 4 | Connectors (MCP) | reach real tools: GitHub, Jira, Slack, DBs | — (makes it useful) | | 5 | Sub-agents (maker/checker) | independent verification (§4) | uncalibrated self-assessment | | 6 | Memory / State | the durable spine (§5A) | statelessness | Build the **minimum viable loop first**: schedule + one triage skill + a state file, report-only. Add each further primitive *only after* the previous version proved its value and revealed its failure modes. Worktrees when you start making changes; the verifier when you start acting autonomously; write-scoped connectors last. **Goal:** keep open PRs green and reviewed without a human polling GitHub. ``` cadence: 10–15m during work hours state: pr-babysitter-state.md (row per PR: last SHA, attempts, status) loop turn: 1. TRIAGE (cheap): gh pr list --json number,headRefOid,statusCheckRollup → PRs whose checks are RED and whose SHA changed since last run 2. if none actionable → exit in <5k tokens # §6.3 early exit 3. for each actionable PR: acquire worktree lock on that PR's paths # §5D no collision MAKER sub-agent: smallest diff to fix the failing check CHECKER sub-agent (stronger model, "find reasons to REJECT"): actually RUN npm ci && npm test and report output # §4 real sensor BREAKER: 3rd attempt on same failing check → STOP, escalate (§5B/5E) GATE: checkGate('commit', changedPaths) before pushing (§5C) 4. persist STATE + append loop-run-log.md ``` Guardrails doing real work: attempt cap kills infinite-fix loops; verifier *runs* tests (kills "verifier theater"); worktree lock kills parallel collisions; gate kills over-reach into denylisted paths. Watches a text-to-SQL service and self-heals bad generated SQL — with a checker the LLM **cannot** fake (directly applies §4.2). ``` trigger: event = "generated SQL failed / returned suspicious result" MAKER: LLM regenerates SQL from the question + schema + prior errors CHECKER: NON-LLM, non-overridable: - EXPLAIN parses without error (syntax/plan valid) - runs on a READ-ONLY replica with LIMIT (bounded, safe) - row count / column types sane vs the question's expected shape - planner cost estimate < threshold (no full scan on a 2B-row fact) BREAKER: same SQL-error signature 3× → escalate to a human analyst (§5B) STATE: questions that repeatedly fail → surfaces schema / prompt gaps ``` Why it fits the root: an LLM asked "does this SQL look right?" is verifier theater. `EXPLAIN` + a bounded run on a real replica measures something the generator **cannot** fabricate — the same lesson as the quant backtest, in SQL. ``` cadence: 6h–1d scope: PATCH + low-risk CVE only for the first 30 days # blast-radius control MAKER: bump one dependency → one PR CHECKER: full npm ci && npm test IN A WORKTREE # real, not "looks fine" GATE: majors + denylisted packages → human, never auto (§5C allowlist) cadence: 1d, weekdays, via CI on a timer skill: triage-only — structured output, ONE line per item + suggested action action: NONE. Writes findings to STATE.md; human reviews weekly. cost: ~50k tokens/day (one light run) purpose: the report-only rung — prove the triage is accurate before ANY autonomy ``` | Failure | Symptom | Root cause | Prevented by | |---|---|---|---| | Infinite fix loop | same PR "fixed" 5+ times, never converges | no attempt cap; weak verifier | circuit breaker (§5B) + separate verifier | | Verifier theater | "looks good" but CI red | same model/context; vague prompt | independent checker that runs tests (§4) | | State rot | acts on merged PRs / dead branches | no prune; unread state | prune step + `Last run` + ID validation (§5A) | | Token burn | bill spikes on empty ticks | full chain on every run | cheap triage + early exit + budget (§6.3) | | Over-reach | edits unrelated / forbidden paths | permissive skill; no denylist | gate.yaml denylist + smallest-diff (§5C) | | Parallel collision | two agents edit same file | no isolation | worktree + advisory lock (§5D) | | Escalation failure | stuck loop, human never told | max-attempts unimplemented; silent state | breaker + connector ping (§5E) | | Comprehension debt | ships faster than anyone can read | human stopped reading | mandatory review + weekly digest | | Cognitive surrender | "the loop handles it," no opinions | success metric = volume | human gates + metric = quality-held | **Intent debt**— the agent's cold-start ignorance of your conventions. Fix: skills, written once, read every run.** Comprehension debt**— the loop ships code faster than you can read it, so you stop understanding your own repo. Fix: mandatory human review of non-trivial diffs; a weekly loop digest.**Cognitive surrender**— you stop having opinions on correctness. The knife's edge:*designing loops with judgment is the cure; using loops to avoid thinking is the accelerant — same action, opposite outcome.***Orchestration tax**— the human cost of coordinating N parallel agents. Worktrees remove the mechanical collisions, but** you remain the ceiling**on how many loops you can actually absorb and review. - The thing that decides "done/correct" is **not** the thing that produced the work, and measures something the producer can't fabricate. - There is a **hard, machine-enforced** cap on iterations, tokens, and repeated failures — the loop escalates instead of spinning. - Every run **reads and writes durable state**; nothing important lives only in a model's transient context. - High-risk paths are on a **denylist enforced in code**, not in a prompt's good intentions. - Parallel workers are **isolated** and cannot silently clobber each other. - Autonomy is **earned up a ladder**, each rung proven before the next. - A human is **pinged on escalation** through a channel they actually read. The harness ships every primitive natively, so you rarely need bespoke CLIs — you need the *discipline*: | Loop primitive | Claude Code equivalent | |---|---| | Scheduling | cron / scheduled tasks / `/loop` | | Worktrees | `isolation: "worktree"` on sub-agents | | Skills | Skills / `SKILL.md` | | Connectors | MCP servers | | Maker/checker | sub-agents (Agent tool), a stronger verifier | | Circuit breaker / budget | Workflow token `budget` + loop-until conditions | | Memory / state | file memory / a committed `STATE.md` | **Bottom line:** treat the agent as an unreliable actuator inside a control loop you own, where every trust boundary is decided by deterministic code or a genuinely independent checker — build it like someone who intends to stay the engineer, not just the person who presses go.