I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.

Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · The Gate Is the Product

I spent 2 releases believing the problem was model capability. I was wrong.

v0.1.0: "The 4B model is too small. A bigger model will find better edits."

v0.2.0: "Mistral 24B will find the edit. Or 30B. Surely one of them will work."

None of them did.

4 models tested. 4,150 LLM calls. 31 v0.1.0 bugs plus 26 v0.2.0 issues resolved. Zero promotable edits.

The model was never the problem. The search strategy was.

Every model produced the same outcome: zero promotions. The 4B model was fast and free but shallow. The 24B model explored more thoughtfully but stayed in a weak local neighborhood. The 30B model added latency without improvement. The 1B model could barely function as an analyzer at all.

The per-iteration results are uncomfortable to read:

Qwen 4B (local) — 5 iterations: | Iter | p-value | Mean delta | Accuracy |

|------|---------|------------|----------| | 1 | 0.73 | -0.05 | 60% |

| 2 | 0.71 | -0.025 | 60% |

| 3 | 0.46 | +0.025 | 60% |

| 4 | 0.67 | -0.025 | 60% |

| 5 | 1.0 | 0.0 | 60% |

Noise. The 4B model's edits are essentially random — sometimes better, sometimes worse, never significant.

Mistral 24B — 5 iterations:

| Iter | p-value | Mean delta | Accuracy |

|------|---------|------------|----------| | 1 | 0.55 | +0.025 | 64% |

| 2 | 0.52 | +0.025 | 64% |

| 3 | 1.0 | 0.0 | 64% |

| 4 | 0.52 | +0.025 | 64% |

| 5 | 0.77 | -0.025 | 64% |

Directional. Mistral produced positive deltas in 3 of 5 iterations. That's real signal, not hallucinated progress. But it never cleared p < 0.05. The edit was real. It just wasn't strong enough.

This is the most frustrating result in the project: a model that clearly found something useful, but not useful enough to prove it.

Every model tested produced edits in the same local region of the prompt:

Mistral 24B explored more variants than the 4B model:

active

to security breach"only if critical to business continuity

"requires immediate intervention (e.g. ...)

"But all three are the same class of edit. None jumps to a different failure family. None touches a different part of the prompt. The model varies the wording of the same concept — "urgent should override security" — and calls it exploration.

This is not a model size problem. Mistral 24B is a capable model that can reason about complex prompts. The problem is the search mechanism. The analyzer generates one proposal, sees the same failures next iteration, and proposes a slightly different version of the same failed edit. There's no mechanism to step back, try a different approach, or evaluate multiple candidates.

Imagine a doctor who diagnoses every patient with "maybe a bit more hydration" — and when that doesn't work, tries "maybe a bit more hydration, but with electrolytes." That's what our analyzer is doing.

The failures are concentrated in 3 specific areas: multi-label classification, ambiguous "other" cases, and keyword over-indexing on "billing." This is the same pattern v0.1.0 found. The same failures persisted across 2 releases and 4 models.

The analyzer knows where the pain is. The data is explicit: multi-label tasks return single labels. An edit that forces comma-separated pair output would likely move 3 tasks immediately. No analyzer proposed it. They kept tightening urgency rules instead.

Chasing model size instead of search strategy.

Here's the thought process that cost us a release:

The pattern is obvious in hindsight: the model is not the bottleneck. The search algorithm — generate one proposal, A/B test it, get rejected, repeat — cannot escape local optima. No model size fixes a search loop with no exploration step.

Even if an analyzer found a great edit, the math makes promotion hard. With 40 tasks in the A/B corpus:

The analyzer never moved more than 5 tasks. Most edits moved 2-3. The gate requires p < 0.05. The math says: with a 40-task corpus and 5 movable tasks, the statistical floor is ~0.031 one-sided. The analyzer needs to find edits that move at least 7-10 tasks.

It has never done that. Across 4 models and 2 releases.

The fix for v0.3.0 is structural: generate multiple diverse candidates, score them cheaply on the failure batch, and A/B only the best one.

This is the most frustrating result in the project. It's also the most valuable.

v0.1.0 left us with 9 bugs and no clarity. Was the system broken? Was the data fake? Was the analyzer too weak? We couldn't tell. We spent months chasing ghost problems.

v0.2.0 eliminated every ghost. The bugs are fixed. The pipeline is validated across 4 domains in Docker. The gate is proven with 0% false positives. And after all that, the bottleneck is one specific, measurable, tractable problem: the analyzer generates one proposal at a time with no exploration mechanism.

That's not a vague concern. It's a concrete engineering target. We know exactly what to build for v0.3.0 — generate multiple diverse candidates, score them cheaply, test the best one. The search problem is hard, but it's defined. And we have 4 models worth of experimental data telling us that model size won't save us.

We proved the pipeline. We proved the safety. We proved what doesn't work. That is the foundation for building what does. And this time the answer is specific enough to implement, not just complain about.