{"slug": "my-self-improving-agent-still-couldn-t-improve-that-was-the-breakthrough", "title": "My Self-Improving Agent Still Couldn't Improve. That Was the Breakthrough.", "summary": "Debashish Ghosal's open-source project AgentSelfEdit, which rewrites its own system prompt from execution feedback, failed to produce a promotable edit in its v0.3.0 release despite extensive testing and hardening. The developer reports that this negative result is the first he fully trusts, as the system's failure is now attributable to the optimizer itself rather than pipeline bugs. The release included 807 hermetic tests, Oracle Drift Guard, and adversarial edit validation.", "body_md": "**Previously:** [9 Bugs That All Looked Like a Working System](https://dev.to/debashish_ghosal/9-bugs-that-all-looked-like-a-working-system-25mg) · [I Built an AI That Rewrites Its Own Prompts](https://dev.to/debashish_ghosal/i-built-an-ai-that-rewrites-its-own-prompts-its-safety-gate-rejected-every-single-edit-220h) · [The Edit That Fixed 4 Tasks and Broke 1](https://dev.to/debashish_ghosal/the-edit-that-fixed-4-tasks-and-broke-1-5770) · [I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It.](https://dev.to/debashish_ghosal/i-gave-an-llm-the-keys-to-rewrite-its-own-prompt-then-built-a-gate-that-said-no-4150-times-1h46) · [I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.](https://dev.to/debashish_ghosal/i-tested-4-models-and-none-could-improve-their-own-prompt-the-search-strategy-is-broken-not-the-3ajf)\n\nAgentSelfEditis an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.\n\nRepo:[github.com/deghosal-2026/agent-self-edit/tree/v0.3.0]\n\nRelease notes:[docs/release/v0.3.0/release-notes.md]\n\nField test report:[docs/field-test/v0.3.0/FIELD_TEST_REPORT.md]\n\nI wanted `v0.3.0`\n\nto be the release where the system finally promoted a good edit.\n\nThat didn't happen.\n\nWhat happened instead was stranger and, honestly, more important. For the first time, the system failed in a way I actually trusted.\n\nThat sounds like a consolation prize. It isn't.\n\nEarlier versions could still fail for dumb reasons. A result might be flat because the analyzer was weak, or because the A/B path was wrong, or because the prompt never really changed, or because the field test runner wasn't measuring the thing I thought it was measuring. A bad result still had wiggle room. I could always tell myself, maybe the loop is better than it looks.\n\n`v0.3.0`\n\ntook a lot of that comfort away. We shipped 807 hermetic tests, 16/16 Docker integration tests, 94.86% coverage at release time, Oracle Drift Guard, a real gold corpus, adversarial edit validation, and enough artifacts to inspect every serious run. After all that, the strongest analyzer still did not produce a promotable edit.\n\nThat is the first negative result from this project that I believe without adding an asterisk.\n\nThis project does one thing: it watches execution traces, proposes prompt edits, A/B tests them, and promotes only the ones that are measurably better.\n\nThat sounds neat. In practice, systems like this can lie to you in boring ways.\n\nThe candidate prompt might not actually be materialized. The A/B engine might compare the wrong thing. The gate might not run in the path you think it runs in. The analyzer might be learning from distorted failures. The report might summarize something cleanly while hiding the fact that the underlying pipeline is sloppy.\n\nThat was the real problem I wanted to kill in `v0.3.0`\n\n. Not \"make the model smarter.\" First: make the failure legible.\n\n`v0.3.0`\n\nwas the release where the project got a lot less magical and a lot more accountable.\n\nThe big things we shipped:\n\n`--cov-fail-under=91`\n\n`materialize_candidate_prompt()`\n\n`str.replace()`\n\nso missing `old_text`\n\nfails loudlyNone of that makes for sexy marketing copy. It does make for a system I can stop apologizing for.\n\nThe strongest analyzer we tested in the main classification loop was `mistralai/mistral-small-3.2-24b-instruct`\n\n.\n\nIt was better than the local 4B model. That's real. It produced the first clean positive movement in the A/B data.\n\nBest iteration:\n\n`2`\n\nimproved`0`\n\nregressed`effect_size = 0.0625`\n\n`p = 0.79`\n\nThe local `Qwen3-4B-Instruct-2507-4bit`\n\nanalyzer was worse. Mostly null edits. A couple of `1 up, 1 down`\n\niterations. No real momentum.\n\nSo yes, if you only want the headline, here it is: the optimizer still didn't produce a promotable edit.\n\nBut this time that sentence means something specific. The prompts were materialized correctly. The A/B path executed. The gate ran. The artifacts were written. The Docker path matched. The tests backed the mechanics. The framework did its part.\n\nThe optimizer failed honestly.\n\nI know that sounds backwards, but this is the part I think developers will get.\n\nThere is a stage in building systems like this where a trustworthy \"no\" is much more valuable than a suspicious \"yes.\"\n\nIn `v0.3.0`\n\n, a failed run usually narrows down to one of three real explanations:\n\nThat is a huge upgrade from:\n\nI can't overstate how much saner the project feels once the failure modes get this narrow.\n\nThree things clearly worked.\n\nFirst, the gate stayed conservative. That is still the healthiest part of the system. We saw 0% false positives, 8/8 adversarial edits blocked, and no bad promotion slipping through local, cloud, or Docker-backed runs.\n\nSecond, the system now leaves behind enough evidence to debug behavior instead of arguing about it. `analysis.json`\n\n, `accuracy.json`\n\n, `ab-comparison.json`\n\n, prompt snapshots, traffic logs. If a run falls apart, I can usually tell where.\n\nThird, Docker validation finally meant what it sounded like it meant. Earlier it was too easy to say \"Docker passed\" while skipping the hardest path. In `v0.3.0`\n\n, we exercised the actual integration surface that matters.\n\nThe main problem is not hidden anymore: the analyzer is weak.\n\nNot fake. Not random. Weak.\n\nIt sees plausible failures and proposes plausible edits. But it keeps collapsing into a tiny search neighborhood. In classification that meant repeated urgency-boundary rewrites. In other corpora it showed up in different clothes, but the habit was the same: local wording tweaks, narrow corrections, not enough exploration.\n\nThe project's main question is no longer \"does the loop run?\" It is \"can the analyzer find edits with enough breadth and magnitude to satisfy a conservative gate?\"\n\nThat is a much better question. It is also a much harder one.\n\nThe big one is simple: framework correctness comes before optimizer cleverness. People skip that all the time in AI tooling. They jump straight to model quality while the harness is still blurry. Then every result turns into an argument.\n\nI also learned that negative results are only valuable when they are inspectable. \"No improvement\" is meaningless if you can't trace how the system got there. \"No improvement, and here are the artifacts, the gate reason, the A/B outputs, and the Docker evidence\" is a real engineering result.\n\nI learned something subtler too: not every rejection means the same thing. In `v0.3.0`\n\n, the gate is no longer just saying \"no.\" It is separating null edits from net-zero edits and from weak-positive-but-underpowered edits. That matters, because those are different product signals. A candidate that moves nothing is not the same as a candidate that helps a little but cannot yet clear confidence.\n\nAnd I learned, again, that safety is not the boring add-on at the end. The gate is the boundary that keeps the system from quietly wrecking its own baseline while you're still figuring out whether the optimizer deserves any trust at all.\n\nIf you're building anything that adapts from feedback, I think `v0.3.0`\n\npoints to the right order of operations.\n\nDon't ask first whether the model can improve itself.\n\nAsk whether the system can distinguish signal from noise, reject bad edits safely, explain why it rejected them, preserve artifacts, and survive the same path in CI and containerized execution.\n\nThat is less flashy than \"self-improving agent.\" It is also the part that makes the flashy thing worth shipping.\n\n`v0.3.0`\n\ndid not give me the success story I wanted.\n\nIt gave me something I needed more: a result I don't have to squint at anymore.\n\nAnd now I think the more interesting question is not \"why didn't it promote?\" It is \"what kind of negative result should make us change the analyzer, and what kind should make us change the gate?\"\n\nIf you were building this system, what would you trust more at this point: a stronger analyzer, a bigger promotion corpus, or a looser significance bar? And at what point does a self-improving system deserve to call itself self-improving if the safety layer is working but promotion still almost never happens?", "url": "https://wpnews.pro/news/my-self-improving-agent-still-couldn-t-improve-that-was-the-breakthrough", "canonical_source": "https://dev.to/debashish_ghosal/my-self-improving-agent-still-couldnt-improve-that-was-the-breakthrough-mni", "published_at": "2026-09-04 01:16:09+00:00", "updated_at": "2026-09-04 01:53:33.841491+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "developer-tools", "ai-research"], "entities": ["AgentSelfEdit", "Debashish Ghosal", "mistralai/mistral-small-3.2-24b-instruct", "Qwen3-4B-Instruct-2507-4bit"], "alternates": {"html": "https://wpnews.pro/news/my-self-improving-agent-still-couldn-t-improve-that-was-the-breakthrough", "markdown": "https://wpnews.pro/news/my-self-improving-agent-still-couldn-t-improve-that-was-the-breakthrough.md", "text": "https://wpnews.pro/news/my-self-improving-agent-still-couldn-t-improve-that-was-the-breakthrough.txt", "jsonld": "https://wpnews.pro/news/my-self-improving-agent-still-couldn-t-improve-that-was-the-breakthrough.jsonld"}}