I Built Five Self-Improving Loops in One Evening. They All Had the Same Bug.

A developer built five small self-improving agent loops in a single evening using Claude Code, finding that all five shared the same flaw: their judges scored whether the right words or shapes appeared rather than whether the artifact actually worked. The developer then built a sixth repo, sia, with a frozen judge, a held-out eval set, and a real keep/rollback gate to close that hole, though all testing was done in mock mode with no live API calls.

Quick disclosure up front, because it matters for how much weight to put on anything below: this was one evening of self-directed practice, no external user or stakeholder involved, six small repos, all built with Claude Code every commit is paired with a session, per the commit trailers . The biggest eval set in any of the six repos is 8 tasks. This is not "I built a self-improving agent." It's closer to: I ran five fast tests of one idea, found they shared a scoring bug, and then built a fix for that bug that I could actually verify. In mock mode. No live API calls anywhere in this story. If you're working on anything with an optimize-against-a-judge shape prompt search, RAG tuning, agent config search, eval-driven pipelines , the bug is worth knowing about before you hit it yourself. The question I wanted an actual answer to: does mutate → judge → keep-if-better → repeat generalize as a loop shape, or does every kind of artifact need its own bespoke approach? Instead of arguing about it, I built five tiny versions, back to back, in about 7 minutes of commit timestamps total: self-improving-prompt-agent context-improving-agent 0.8 fact coverage + 0.2 1-bloat penalty graph-improving-agent harness-improving-agent agent-improving-agent Four of the five repos have exactly one commit. That's what five fast structural tests of one hypothesis actually looks like. The loop shape held up. All five ran the same structure and all five climbed over 10 rounds: | Repo | Result 10 rounds | |---|---| | prompt | 0.10 → 0.80 | | context | 0.20 → 0.28 | | graph | 0.10 → 0.70 | | harness | 0.00 → 0.90 | | agent meta | 0.45 → 0.61 stated max: 0.80 | Five for five, on paper. That wasn't actually the interesting result. Every one of these repos has a "Known Limitations" section, because writing down what's wrong with what I just built while I still remember it is a habit. Read individually, each limitation sounds like a minor, honest caveat: Read together, though, these are the same bug five times, wearing five different filenames. Every judge in this set scored whether the right words or shapes showed up, not whether the artifact actually worked: fact coverage term almost certainly measures whether facts are textually present, not whether the context is coherent once they're added. None of the judges were malfunctioning. They did exactly what I told them to measure. The problem is I told five of them, separately, to measure the wrong thing, and I only caught it by reading all five READMEs side by side instead of shipping straight to a sixth prototype. There's a second wrinkle worth calling out on harness-improving-agent specifically: its judge used loose keyword matching early on and stalled at 0.20 for several rounds, until I rewrote the judge's wording mid-run to match the mutation wording it was supposed to be scoring. The jump to 0.90 partly reflects that fix, not clean search progress, and the repo's own README says this directly. About 2.5 hours after the fifth prototype's commit, I built a sixth repo, sia self-improving-agent , whose README opens by naming the first five as prior art and states plainly why it doesn't reuse their judges: porting a structure-presence judge into something meant to be real "would have meant the keep/rollback gate approved patches for saying the right words rather than answering questions correctly." Four changes, each one closing a specific hole from the first five: 1. Frozen judge, held-out eval. sia optimizes a genome versioned JSON: prompt/context/workflow/harness layers against a fixed 8-task set, split 5 train / 3 holdout. The judge doesn't get edited mid-run, unlike what happened with harness-improving-agent . 2. A real gate, not a bare threshold: keep iff train child = train parent + 1.0 AND holdout child = holdout parent The +1.0 minimum lift on train, combined with non-regression on holdout, is there specifically to catch a patch that overfits the five training examples without generalizing. None of the first five repos had a holdout set at all: they scored and kept against the exact same signal they optimized against, every single round. 3. Integrity checks on the eval itself. sia/loop.py hashes the eval set and the judge file at the start of a run, and re-checks that hash every iteration. If either changes, the run aborts with a logged ABORT row. Watching five judges reward the appearance of a good answer taught me this directly: an optimizer that's allowed to edit what it's graded against will eventually find that editing the test is cheaper than improving the artifact. 4. Positive and negative controls, not just a demo run. Checked directly against sia 's own ledger, not a printed summary I trusted at face value: 3 keeps and 1 correct rollback show the gate distinguishing real lift from no lift. That's a stronger claim than "the score went up." I also ran the test suite live: 21/21 unit tests passing. One more scope decision worth mentioning: sia 's README states "The AI is not allowed to call itself better." The improver component can write a hypothesis about why a patch might help, but that text is documentation only. It never feeds into the score. Everything above happened in mock mode, a simulated worker standing in for a real model call. There's an API key sitting in a .env file that was never used to make a real call: no logs, no genomes, no ledger rows anywhere attributable to a live run. sia 's client hard-raises if it's ever invoked with mock=True , so there's no ambiguity about which mode produced these results. That also means I genuinely don't know whether the gate, the holdout split, or the hash-check survive contact with real, noisy model output. It might. I haven't tested it. Also unresolved: the meta-loop from agent-improving-agent the one that optimizes guardrail priority order was designed to plug into exactly this kind of frozen judge, and I never wired it in. The README calls it future work, plainly. If you're building any kind of optimization loop over LLM output, the thing worth taking from this isn't the five prototypes or sia 's specific architecture. It's the order of operations: before investing in a better mutation strategy or a smarter search, check what your judge is actually measuring, and whether that's cheap to satisfy without being correct. A keyword-presence judge gets gamed eventually, not because the search is adversarial, but because "score well against this specific signal" is always an easier target than "be right," and any sufficiently capable optimizer finds the easier target first. I built the same mistake five times before I noticed it, then spent the rest of the night on the unglamorous fix: a frozen test set, a holdout split, and a hash check that so far has never fired.