AgentSelfEditis an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners. Code:[github.com/deghosal-2026/agent-self-edit]
15 iterations. 4,150 LLM calls. Zero promotions. The gate said no — 15 times in a row.
And it was right every time.
I spent a full session building and testing this system. I thought I'd get a promotion — an edit that the gate approves, the prompt improves, accuracy goes up. Instead, I got the most honest result possible: the gate rejected everything, and the rejections were correct.
Here's what happened, how I found a two-line bug that was letting noise through as "improvement," and why a gate that never promotes is the most valuable thing I built.
Most prompt optimization is vibes. You change a prompt, run it on a few examples, and if it "looks better," you ship it. There's no statistics. No control group. No significance test. Just intuition.
That works fine for a human tweaking a prompt by hand. But when you're building a system that rewrites its own prompts autonomously — an LLM proposing edits, the system applying them, and the loop repeating — vibes aren't enough. You need evidence.
The easy answer is "ask an LLM to judge the edit." But that's a fox guarding the henhouse. An LLM judging its own edits will optimize toward what it likes, not what actually works. The prompt drifts. The system gets worse, not better, and nobody notices because the judge keeps saying "looks good."
The harder answer is: build a deterministic, statistical gate that evaluates edits using evidence, not opinion. Code, not prompts. p-values, not vibes.
That's what I built.
AgentSelfEdit is a sidecar. It doesn't modify the agent's runtime. It observes execution traces and proposes prompt edits through a closed loop:
The gate is the most important component. It's the safety mechanism that prevents the system from optimizing itself into a worse state. And it's not an LLM — it's six deterministic checks, running in fail-fast order:
| # | Check | What it prevents |
|---|---|---|
| 1 | Sample floor | Decisions from tiny samples |
| 2 | Effect size | Trivial improvements getting promoted |
| 3 | Confidence (p < 0.05) | Random noise getting promoted |
| 4 | Frozen sections | Analyzer modifying protected content |
| 5 | Edit distance | Wholesale prompt rewrites |
| 6 | Drift detection | Divergence from baseline |
Three outcomes: promote (all 6 pass), near-miss (most pass, logged for human review), reject (a critical check failed). No LLM in the decision. No "looks good to me." Just code.
I ran the loop. I got a promotion. Accuracy jumped from 20% to 40%. I celebrated.
Then I looked at the code.
The confidence check — the one that's supposed to prevent noise from getting promoted — was checking the wrong threshold:
passed = p < confidence_level # p < 0.95
alpha = 1 - confidence_level # 0.05
passed = p < alpha # p < 0.05
The gate was checking p < 0.95
instead of p < 0.05
. Think about what that means. A p-value of 0.9 would pass. A p-value of 0.5 would pass. Even a p-value of 0.94 would pass. Almost everything would pass.
My "promotion" at p=0.1 had a 10% chance of being random noise. One in ten. I was celebrating a result that could have been a coin flip.
Standard hypothesis testing works like this: you set a significance level (alpha), typically 0.05. You compute a p-value. If p < alpha, the result is statistically significant. The confidence_level (0.95) is 1 - alpha. So alpha = 1 - 0.95 = 0.05. The gate should have been checking p < 0.05
. It was checking p < 0.95
.
Two lines of code. That's all it took to turn noise into a "success."
After the fix, the same edit produced p=0.23. The gate rejected it. Correctly. There was a 23% chance the improvement was random — well above the 5% threshold the gate requires.
I ran the corrected loop for 15 iterations against a local Qwen3.5-4B-4bit model on Apple Silicon. Every iteration produced the same result:
The edit was real. The improvement was real. The model genuinely classified 4 tasks better with the priority rules. But it also broke 1 task — a search bug that went from "technical" (correct) to "feature" (wrong) because one of the priority rules was too broad.
Net +3 on 26 tasks. p=0.23. The permutation test asked: "if this edit had no real effect, what's the probability of seeing an improvement this large by chance?" The answer: 23%. The gate requires less than 5%.
| Metric | Value |
|---|---|
| Iterations | 15 |
| LLM calls | 4,150 |
| Total tokens | 716,580 |
| p-value | 0.23 (every single iteration) |
| Gate decision | reject (every single iteration) |
| False positive rate | 0% |
| False negative rate | 0% |
| Total cost | |
| $0.00 (local 4B on Apple Silicon) | |
| Wall time | 37 minutes |
The gate was doing its job. It was protecting the system from an edit that, while helpful, hadn't proven itself enough. The evidence bar was set at p<0.05, and the edit couldn't clear it.
The confidence check wasn't the only bug. Over the course of this session, I found and fixed 31 issues. Here are the ones that mattered most:
The A/B test was comparing a prompt against itself. The code passed a fragment (the edited section) as prompt_b
instead of the full candidate prompt. Both arms used the same prompt. Every A/B test was a tie — not because the edit didn't help, but because there was no edit to test.
The failure traces were fabricated. The script hardcoded final_output: "other"
for every trace. But the model actually outputs "billing," "security," "technical." The analyzer was learning from a failure pattern that didn't exist.
The gate received the wrong prompt. check_all
got prompt_b
(the edited version) instead of prompt_a
(the original). The frozen_sections check looked for edit.old_text
in the wrong prompt — it was already replaced.
The Docker test skipped the A/B test and gate. It used --dry-run
, which skips the two most important stages. "9/9 tests passed" was a smoke test dressed up as an integration test.
Every one of these bugs produced output that looked correct. The summary said "pass." The traffic logs said something different. I caught every one by inspecting raw LLM traffic — 4,150 request/response pairs logged to a JSONL file.
The gate is the product, not the optimizer. The most valuable part of a self-improving system isn't the component that proposes changes — it's the component that decides whether to accept them. The optimizer can be an LLM, a heuristic, or random guessing. The gate must be deterministic, verifiable, and conservative. If the gate is wrong, the system drifts. If the gate is right, the system is safe — even if the optimizer is dumb.
A gate that never promotes is still valuable. If the gate rejects everything for 15 iterations, it's tempting to say "the system doesn't work." But the gate rejecting means the evidence bar is being enforced. The analyzer's proposals aren't strong enough yet — that's a different problem. The gate is working. The optimizer needs to get better.
The goal isn't to make the loop pass. I spent time tweaking thresholds to force a promotion. I raised the drift threshold from 0.3 to 0.5. I expanded the A/B task set from 5 to 26. I got a promotion and celebrated. Then I found the confidence bug and realized the promotion was noise. I had to course-correct: the goal of the field test was to prove the gate behaves honestly, not to get a green outcome. A gate that rejects underpowered improvements is the success condition, not the failure condition.
Statistical significance is not optional. The inverted confidence check (p < 0.95
instead of p < 0.05
) let noise through as "improvement." Two lines of code. That's the difference between a system that learns and a system that drifts. If you're building a self-improving system, check your statistics. Read the gate code. Verify the p-value threshold.
pip install agent-self-edit
The full 15-iteration field test — every A/B result, every gate decision, every per-task output — is open source:
The gate works. The loop is mechanically sound. The current analyzer doesn't produce edits strong enough to pass — which is a valid finding, not a bug.
But here's the question I keep thinking about: if the gate never promotes, is the system actually "self-improving"?
For v0.2.0, I'm building a rejection-aware analyzer that learns from gate decisions, cumulative evidence across iterations, and larger A/B task sets for more statistical power.
But I'm curious — what would you do differently? Would you lower the significance threshold? Add more tasks? Build a smarter analyzer? Drop your thoughts in the comments.