9 Bugs That All Looked Like a Working System

An engineer detailed nine fundamental bugs in AgentSelfEdit, an open-source sidecar that rewrites its own system prompt from execution feedback, which made the system appear functional while it was not. The most critical flaw was a statistical gate checking p < 0.95 instead of p < 0.05, allowing noise to be promoted as improvement. Other issues included A/B tests comparing a prompt against itself and fabricated failure traces.

AgentSelfEditis an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners. Code: github.com/deghosal-2026/agent-self-edit I built an AI that rewrites its own prompts. It looked like it worked. It didn't. Over a single session, I found and fixed 31 issues. Nine of them were fundamental — each one made the system look like it was working when it wasn't. The most dangerous was this: the promotion gate was letting noise through as "improvement." It checked p < 0.95 instead of p < 0.05 . Almost everything passed. A "promotion" at p=0.1 had a 10% chance of being random noise. Two lines of code separated a system that learns from a system that drifts. But that wasn't the only one. The A/B test "passed" because it compared a prompt against itself. The scoring "passed" because it accepted any non-empty response. The Docker test "passed" because it skipped the hard parts. The failure traces were fabricated. The gate received the wrong prompt. The CLI talked to a mock instead of a real LLM. The config silently ignored the endpoint. And the field test runner was measuring the wrong thing entirely. The most dangerous bugs aren't the ones that crash. They're the ones that produce output that looks correct. The system always produces something — the question is whether that something is real. Here's every bug, how it hid, how I caught it, and what it taught me about building systems on top of LLMs. This was the most insidious bug. The gate promoted an edit. Accuracy jumped from 20% to 40%. I celebrated. Then I looked at the code. What was written: passed = p < confidence level p < 0.95 What it should have been: alpha = 1 - confidence level 0.05 passed = p < alpha p < 0.05 The gate was checking p < 0.95 instead of p < 0.05 . Almost everything passed. A p-value of 0.9 would pass. Even 0.5 would pass. The "promotion" at p=0.1 had a 10% chance of being random noise. Standard hypothesis testing requires p < alpha, where alpha = 1 - confidence level. With confidence level = 0.95, alpha = 0.05. The gate should have been checking p < 0.05. It was checking p < 0.95. After the fix, the same edit produced p=0.23. The gate rejected it. Correctly. There was a 23% chance the improvement was noise — more than the 5% threshold the gate requires. Two lines of code. That's the difference between a system that learns and a system that drifts. The lesson: Check your statistics. p < 0.95 is not p < 0.05 . The confidence level is not the p-value threshold — alpha is. Read the code. This was the first one I found, and it set the tone for everything that followed. The system has an A/B test engine. It runs a candidate prompt against the current prompt on a held-out task set, scores both, and computes whether the candidate is statistically better. The output looked like this: A/B test: tie p=1.0000, n=5 Gate: reject A tie. p=1.0. That means both prompts produced identical results on all 5 tasks. The gate correctly rejected — a tie means no improvement. But a tie with p=1.0 is suspicious. It means zero variance . Not one task changed. In practice, even a bad prompt produces some difference. A perfect tie is a red flag. I dug into the traffic logs. Every single A/B test call — all 10 of them 5 tasks × 2 prompts — used the exact same prompt text. The system was comparing a prompt against itself. The root cause was in run.py . The code passed proposal.new text as prompt b : BUG: passes the edited fragment, not the full prompt ab result = run ab test registry.current prompt, proposal.new text, task set, llm, scorer, config proposal.new text is a fragment — like "You are a technical support ticket classifier." It's the edited section , not the full prompt. The A/B engine expected a complete system prompt. The fragment wasn't valid, so the engine fell back to the current prompt for both arms. The fix was one line: candidate prompt = registry.current prompt.replace proposal.old text, proposal.new text ab result = run ab test registry.current prompt, candidate prompt, task set, llm, scorer, config Construct the full candidate prompt by applying the edit to the current prompt. Then test that. The lesson: When an A/B test produces a perfect tie, check the traffic. A tie means either a the edit doesn't change behavior, or b you're not actually testing two different prompts. Inspect before you trust the result. The system had a scoring mode called label . It was designed for real traces where the "expected output" is a success label like "no hallucination, no loop, no degradation" — not an actual answer. In label mode, the scorer checked one thing: bool llm output.strip . If the LLM produced any non-empty response, the trace was marked "passed." This meant every trace — including failure traces — scored 100%. The LLM always writes something . A trace with success: false was marked "passed" because the LLM wrote a paragraph. 100% pass rate is impossible unless the scoring is broken. I saw it and thought: "That can't be right." It wasn't. I deleted the scoring script entirely. The production scoring system scorers.py was correct — it uses ExactMatchScorer , ContainsScorer , and LLMJudgeScorer . The label mode only existed in a standalone eval script that shouldn't have been part of the self-edit loop at all. The lesson: A 100% pass rate is a red flag, not a success. If your scoring system never fails, it's not testing anything. "9/9 Docker tests passed." The WBS row was marked done. Everything looked fine. The Docker integration test ran agent-self-edit run --once --dry-run . The --dry-run flag causes run.py to skip the A/B test and the promotion gate entirely. The test only verified that the system could ingest traces and run the analyzer. It never tested the A/B test or the gate — the two most important components. This was a smoke test dressed up as an integration test. The WBS acceptance criteria said "A/B test and promotion gate" — but the test skipped both. I caught it by looking at the test output. There was no "A/B test" line. No "Gate:" line. Just "Analysis complete" and "Loop stopped." The most important stages never ran. The fix was to remove --dry-run , add a task set path to the config so the A/B test could execute, and run the full loop: ingest → analyze → A/B test → gate → reject. After the fix, the Docker test took 62 seconds instead of 5 — because it was actually doing real LLM calls for the A/B test. The lesson: --dry-run is not an integration test. If your test skips the hardest part, it's a smoke test. Label it accordingly. This was the bug that explained why the A/B test always tied. The failure traces — the data fed to the analyzer — were fabricated. The seed trace store function created traces like this: store.ingest { "task input": task "input" , "final output": "other", HARDCODED "expected output": task "expected output" , "success": False, } Every trace said the model output "other" when it should have output "technical" or "urgent" or "billing." But the model doesn't output "other" — it outputs "billing," "security," "technical." The analyzer was learning from a failure pattern that didn't exist. Imagine a doctor trying to diagnose patients, but every patient's chart says "symptom: headache" regardless of what they actually have. The doctor would propose treatments for headaches. None of them would work, because the patients don't have headaches. That's what was happening. The analyzer saw 10 traces all saying the model output "other." It proposed edits aimed at fixing "other" outputs. But the model never outputs "other" — it outputs "billing" when it should output "technical," or "security" when it should output "urgent." The edit was aimed at the wrong problem. The fix: run the current prompt against the task set, capture the model's actual outputs, and seed only the real failures. After this fix, the A/B test immediately showed non-zero deltas for the first time. The analyzer started proposing relevant edits. The lesson: Your feedback loop is only as good as the data you feed it. If the failure traces don't match reality, the system optimizes against fiction. Always seed real data. After fixing the confidence check, the gate was still failing — but on a different check: frozen sections . The error message said "edit.old text not found in current prompt." I assumed the analyzer was modifying frozen content. It wasn't. The check all function takes current prompt as its third argument. The code was passing prompt b the edited version instead of prompt a the original : BUG: passes the edited prompt gate result = check all proposal, ab result, prompt b, prompt a, config FIX: pass the original prompt gate result = check all proposal, ab result, prompt a, prompt a, config The frozen sections check looks for edit.old text in current prompt . If current prompt is prompt b the edited version , the old text has already been replaced. It's not there. The check fails — not because the edit modified frozen content, but because the check was looking at the wrong prompt. This bug was hiding behind the confidence bug. While the confidence check was inverted p < 0.95 , it was always the first check to pass, and the frozen sections failure never mattered. Once I fixed the confidence check, the frozen sections bug surfaced. The lesson: Fixing one bug can reveal another. When you fix the top of the fail-fast stack, the next failure surfaces. Keep going. run.py Talked to a Mock Instead of a Real LLM The loop ran and completed. The system was "making LLM calls." The output showed "Analysis complete: 1 proposals." But run.py:37 had this: llm = MockProvider responses=" " A debugging leftover. Even with provider: openai in the config, the code hardcoded a MockProvider that returned empty strings. The analyzer was receiving as its input — no traces, no failures, nothing to analyze. It still "produced a proposal" — but the proposal was based on nothing. The loop completed in under a second. Real LLM calls take minutes. That was the red flag. The lesson: Debugging leftovers are dangerous. If you hardcode a mock during development, replace it before shipping. And if your LLM loop completes instantly, you're not calling an LLM. The config file had base url: http://localhost:8000/v1 . The system was "configured" to use the local OMLX server. But LLMConfig — the dataclass that reads the config — didn't have a base url field. The YAML's base url was silently dropped. The OpenAI client used its default endpoint api.openai.com instead of the local server. Every call went to the cloud — or failed silently. The OMLX server never logged any requests because it never received any. This was a silent config failure. No error, no warning. The field just didn't exist, so the value was ignored. The system ran, made calls, and produced output — just not to the endpoint the user configured. The lesson: Silent config failures are the worst kind. If a config field doesn't map to a dataclass field, either validate it or log a warning. Don't silently drop it. The field test produced results — accuracy, latency, token counts. The "field test" was "running." The numbers looked reasonable. But run traces.py was a generic LLM eval runner. It sent each trace's task input to the LLM as a standalone chat completion. It didn't call any agent self edit modules. It wasn't running the self-edit loop at all — it was measuring the model's raw output on individual tasks. The script didn't import anything from the package it was supposed to test. It was a standalone OpenAI client — not the self-edit loop. The "field test results" were measuring the model's baseline behavior, not the loop's ability to improve. I deleted it and built run improvement loop.py — a script that calls the internal API directly, runs the full loop analyze → A/B test → gate → promote/reject , and writes per-iteration artifacts prompt-a/b, results-a/b, ab-comparison for every iteration. The lesson: Make sure your test runner is actually testing the thing you think it's testing. If it doesn't import the package, it's not testing the package. Every single bug was caught the same way: read the raw LLM traffic, not the summary output. The summary said "pass." The traffic said "you're comparing a prompt against yourself." I used AGENT SELF EDIT LLM LOG — one environment variable that causes every LLM request/response pair to be written to a JSONL file. 4,150 entries across 15 iterations. Every bug was found by reading this file. The first red flag was always the same: suspicious speed + perfect result. When the result looks too clean, it usually is. Real LLM calls have latency. Real A/B tests have variance. Real scoring produces failures. If everything passes, check what "passing" actually means. Log raw LLM traffic. Always. Summary output lies. Request/response pairs don't. One environment variable, one JSONL file, and every bug becomes findable. "Too fast + too clean" is a red flag. Real LLM calls take time and have variance. If your A/B test completes in 54 seconds with a perfect tie, something is wrong. If your scoring never fails, something is wrong. If your loop completes instantly, something is wrong. Every bug looked like success. That's the danger of building on top of LLMs — the system always produces something . The question is whether that something is meaningful. The A/B test produced a "result." The scoring produced a "pass." The gate produced a "promotion." None of them were real. 31 issues in one session. The system went from "looks like it works" to "actually works." The difference was inspecting the data underneath the summary. Two hours of reading traffic logs. No magic, just verification. pip install agent-self-edit What's the worst "it looked like it was working" bug you've found in an AI system? I'd love to hear about it — drop it in the comment