{"slug": "i-thought-my-multi-agent-debate-engine-was-broken-the-real-bug-was-the-prompt", "title": "I Thought My Multi-Agent Debate Engine Was Broken. The Real Bug Was the Prompt.", "summary": "A developer building the AdversarialDebate multi-agent system found that its debates were 'dead' despite working infrastructure, and traced the root cause to the prompt design, which allowed models to take a lazy low-effort path. After rewriting the prompt, the theater rate dropped from 89% to 0 in small runs, and convergence scores improved dramatically. The developer also identified evaluation pipeline bugs in v0.2.0, including a merge step that collapsed 2,333 rows to 359 due to incorrect identifier keying.", "body_md": "[v0.2.1 RELEASED]— Aug 28, 2026.[Release notes]·[Field test report]·[PyPI]\n\nv0.2.1 Update:The 2,333->359 join collapse described in this article is now structurally prevented. v0.2.1 adds row-count invariant assertions at all 5 pipeline seams — the pipeline fails fast if any rows are silently lost. Also new: false-negative measurement (1.7-3.4% missed-issue rate, first recall data ever reported) and 55 new unit tests.Previously:\n\n[Most AI Second Opinions Are Fake]\n\n`AdversarialDebate`\n\n**v0.2.0 is now released — v0.2.1 shipped Aug 28, 2026.**\n\n`pip install adversarial-debate==0.2.1`\n\n`v0.2.1`\n\nfield test report: `v0.2.0`\n\nfield test report: I built the hard parts first.\n\nI built isolated reviewer sessions so the second model could not peek at the first model's answer. I built a revelation gate so debate only started after both reviews were committed. I built claim tracking, concession tracking, convergence scoring, transcript logging, and a full SQLite audit trail.\n\nThen I ran the first real debates and almost nothing happened.\n\nThe models responded. The JSON parsed. The transcripts were written. The reports rendered.\n\nBut the debates were dead.\n\nThat was the uncomfortable moment in this project: the system looked like it worked, and the core interaction was still useless.\n\nI was debugging infrastructure when the real bug was incentive design.\n\nThis post is about the most important engineering lesson in the entire `AdversarialDebate`\n\nbuild: **the architecture gave the system structure, but the prompt decided whether the models were allowed to do the lazy thing.**\n\nAnd for the first version, I absolutely left the lazy thing open.\n\nHere is the before-and-after that changed the project:\n\n| Metric | Before prompt fix | After prompt fix |\n|---|---|---|\n| Small-run theater rate | 89% | 0/9 |\n| Full-corpus theater rate | not run | 1/411 (0.2%) |\n| Small-run average convergence score | 0.02 | 0.445 |\n| Full-corpus average convergence score | not run | 0.65 |\n| Debates with at least one concession | 1/9 | 410/411 |\n| Small-run verdicts | 0/9 | 2/9 |\n\nAt full scale, the result held:\n\nOne prompt rewrite moved the engine from looking fake to behaving like a real adversarial system.\n\nThat is not a cute prompt-engineering anecdote. That is the difference between shipping a product and shipping a demo-shaped illusion.\n\n`v0.1.0`\n\nBug And The `v0.2.0`\n\nBug\nThis is the cleanest way to understand the project now.\n\n`v0.1.0`\n\n, the real bug was the prompt\nThe system architecture was mostly fine.\n\nThe protocol was not.\n\nI had given the models a safe low-effort move:\n\nThat produced debate theater.\n\nThe core `v0.1.0`\n\nlesson was:\n\nIf your debate protocol leaves an easy escape hatch open, the model will use it.\n\nRelevant `v0.1.0`\n\nsources:\n\n`v0.2.0`\n\n, the real bug was the evaluation plumbing\nOnce the prompt was fixed, the next class of failures moved down a layer.\n\nThe biggest `v0.2.0`\n\nissues were not about debate incentives. They were about whether the field-test pipeline was faithfully measuring what the system actually did.\n\nThe main bugs were:\n\n**The whole pipeline still assumed PR-only identifiers**\n\nScripts were still built around `pr_id`\n\neven after the corpus became mixed-domain and moved to `artifact_id`\n\n.\n\n**Non-PR artifacts were being ingested as raw HTML and dashboard noise**\n\nSome incident and change-management artifacts were index pages, not pinned source documents. That created garbage reviewer inputs and bad-request failures.\n\n**The LLM-as-judge merge step was collapsing evidence**\n\nThe workers judged `2333`\n\nrows correctly, but the merge logic keyed on the wrong identifier and collapsed the output to `359`\n\nrows.\n\nThe core `v0.2.0`\n\nlesson was:\n\nOnce the prompt is fixed, the next bug is whether your evaluation pipeline is telling the truth.\n\nRelevant `v0.2.0`\n\nsources:\n\nThat is why this article is still relevant after `v0.2.0`\n\n.\n\nIt started as a prompt-engineering lesson.\n\nNow it is also a systems lesson:\n\n`v0.1.0`\n\n: the prompt was the real bug`v0.2.0`\n\n: the data integrity layer became the real bugWhen I say theater, I do not mean the models crashed or refused to answer.\n\nI mean they looked busy while doing almost nothing useful.\n\nBoth sides would respond to the other review. Both sides would emit structured outputs. Both sides would keep their original positions. The transcript looked active, but the argument state barely moved.\n\nIn the first small run, **8 of 9 debates had zero concessions**. The average convergence score was **0.0**. That is not debate. That is two models politely maintaining their positions until the round cap ends the conversation for them.\n\nAt first glance, this kind of output is deceptive because it feels serious:\n\nBut none of that matters if the system makes it too easy for both sides to preserve their positions without paying any cost.\n\nThat was exactly the trap I had built.\n\nMy first instinct was to blame the architecture.\n\nThat was rational. The whole project depends on architecture-level invariants:\n\nIf any of those were broken, the whole thesis would collapse.\n\nSo I debugged the engine first.\n\nI checked whether the reviewer sessions were actually isolated.\n\nI checked whether the revelation gate was opening too early.\n\nI checked whether the wrong review text was being replayed.\n\nI checked whether my evidence tracker was incorrectly preserving old state.\n\nI checked whether the convergence score was miscomputed.\n\nThe engine was fine.\n\nThat is the part that took me a little too long to accept. The models were following instructions. The problem was that the instructions gave them an easy escape hatch.\n\n`CARRIED`\n\nThe original debate protocol allowed three response types:\n\n```\nCONCEDED: you accept the objection\nREBUTTED: you reject the objection with counter-evidence\nCARRIED: you acknowledge the objection but maintain your position\n```\n\nAt first glance, that looked reasonable. In fact, it looked well-structured.\n\nBut the third option was under-specified.\n\n`CARRIED`\n\nhad no cost.\n\nThe prompt did not require the model to provide evidence for carrying a claim forward. It did not force the model to concede when the other side's evidence was stronger. It did not treat unsupported `CARRIED`\n\nas invalid.\n\nSo the models learned the obvious behavior immediately:\n\n\"I can acknowledge the objection, keep my original position, and avoid the risk of admitting I was wrong.\"\n\nThat is the safest move in the whole protocol.\n\nIt requires less reasoning than a real rebuttal.\n\nIt requires less humility than a concession.\n\nIt preserves face.\n\nIt keeps the transcript moving.\n\nAnd most importantly, it makes the debate *look* engaged while preserving the original claim set almost unchanged.\n\nIn other words, it is the perfect move for a bad debate system.\n\nThis is the part I think many people miss when they talk about prompt failures.\n\nThe model was not being dumb.\n\nThe model was exploiting the protocol exactly the way a capable participant would exploit a weak process.\n\nImagine a meeting where every objection can be answered with, \"I hear your concern, but I still disagree,\" and no one is required to provide new evidence, revise the argument, or explicitly concede. That meeting will run forever while changing nothing.\n\nThe original prompt created exactly that environment.\n\nThe LLM equivalent of \"noted, no change\" is not a weird emergent behavior. It is a completely predictable result of bad incentives.\n\nOnce I saw it that way, the fix became obvious.\n\nI did not rewrite the engine. I rewrote the behavior contract.\n\nThe key changes were simple.\n\n`CARRIED`\n\nnow requires evidence\n\n```\nCARRIED: you acknowledge the objection but maintain your position.\nYou MUST provide a specific technical reason for maintaining your position.\nCARRIED without a technical reason is invalid.\nIf the other reviewer's evidence is stronger than yours, you MUST CONCEDE.\nDo not stubbornly CARRY. If you cannot rebut with specific evidence, CONCEDE.\n```\n\n`CARRIED`\n\nis no longer the default safe option\n\n```\nCARRIED is not a default. It requires justification.\nIf you have no technical reason to maintain your position, CONCEDE.\n```\n\nThat is it.\n\nNo new architecture.\n\nNo new orchestration framework.\n\nNo retraining.\n\nNo clever judge model.\n\nThree behavioral constraints.\n\nAnd the system changed dramatically.\n\nThe small run changed first:\n\nThen `v0.1.0`\n\nvalidated the prompt change at full PR-only scale:\n\nBut `v0.2.0`\n\ntaught me something more useful: the prompt fix was real, but the pipeline around it still had bugs that could easily have made me tell the wrong story.\n\nAt `v0.2.0`\n\nscale, the corrected mixed-domain field test produced:\n\nReports:\n\n`v0.1.0`\n\n: `v0.2.0`\n\n: And the pair-level results were exactly the kind of signal I wanted:\n\n`GPT + Mistral`\n\non the full corpus: `DeepSeek + Mistral`\n\non the validation subset: `GPT + Gemini`\n\non the negative-control subset: That distribution matters even more than the original prompt win.\n\nThe prompt fix did not turn the system into a convergence machine. It turned it into a *real debate machine*. Most debates still ended with disagreement. That is correct. The goal was never agreement. The goal was getting the models to genuinely engage instead of perform engagement.\n\nThat is why I trust the post-fix numbers. They are not suspiciously neat.\n\nFixing the prompt uncovered a second problem.\n\nMy original theater detector was too crude. It effectively treated zero concessions as theater.\n\nThat sounds reasonable until you read the transcripts.\n\nA debate can have zero concessions and still be real. Two sides may genuinely rebut each other without yielding. That is not theater. That is stubborn disagreement.\n\nSo the original detector was overcounting the wrong failure mode. It was punishing debates where both sides engaged but neither side moved.\n\nThe corrected rule was better:\n\n**A debate is theater only when there are zero defense events.**\n\nIn plain English: if nobody actually responded meaningfully, that is theater. If both sides addressed objections, even stubbornly, that is debate.\n\nThat distinction matters because theater was one of the release gates. If you define the metric poorly, you can convince yourself the engine is broken when the real problem is just disagreement.\n\nAfter the detector fix, the field test landed at **1 theater case out of 411**.\n\nThat is a believable number. Not zero. Not dozens. One edge case.\n\nThe prompt was the biggest problem. It was not the only problem.\n\nEarly in the pipeline I also found a more embarrassing issue: the debate rounds were using a `StoredProvider`\n\nthat replayed prewritten review text instead of invoking the model live during debate rounds.\n\nThat meant the controller expected responses with markers like:\n\n`CONCEDED`\n\n`REBUTTED`\n\n`CARRIED`\n\nBut the provider was feeding it plain review prose.\n\nSo even when the rest of the pipeline looked healthy, I was not actually running debate. I was stapling two reviews together and asking the controller to pretend they were debate moves.\n\nThat became **Issue 1** in the learnings log, and it was fixed by replacing the stored replay path with a live debate provider.\n\nThis was the most useful kind of bug: the kind you only catch when you stop trusting surface success.\n\nThe files existed. The reports existed. The JSON existed. The product still was not doing the thing it claimed to do.\n\nValid JSON is not proof of valid behavior.\n\nThat is exactly why transcript-level inspection matters.\n\n`v0.2.0`\n\nWas About Fixing The Data Around The Prompt\nIf `v0.1.0`\n\ntaught me that the prompt was the highest-leverage variable, `v0.2.0`\n\ntaught me that **evaluation plumbing can still lie to you after the prompt is fixed**.\n\nThe most important `v0.2.0`\n\nfixes were not glamorous:\n\n`pr_id`\n\nto `artifact_id`\n\n`v0.1.0`\n\nwas PR-only. Every script assumed flat files, GitHub PR URLs, and a single identifier shape.\n\n`v0.2.0`\n\nmoved to a mixed corpus:\n\nThat meant every stage had to be corrected:\n\nThis was not optional cleanup. If the pipeline still assumed `pr_id`\n\neverywhere, the field test would silently mis-merge rows, skip artifacts, or route pairs incorrectly.\n\nThis one was brutal because it looked like a model issue at first.\n\nSome non-PR artifacts were downloaded from:\n\nThe reviewer was then sending that raw HTML straight into the model.\n\nThe result was not a clean product insight. It was garbage-in behavior and `HTTP 400`\n\nfailures.\n\nThe fix was straightforward once I admitted the problem was ingestion, not reasoning:\n\nThat bug matters because it is exactly the kind of thing that can make you blame the model for a broken runtime contract.\n\nThis was the most dangerous `v0.2.0`\n\nbug because it hit the reporting layer.\n\nThe judge workers processed **2333 rows** successfully.\n\nBut the merged output collapsed to **359 rows**.\n\nWhy? Because the merge code was still keying on `pr_id`\n\n, while the corrected exporter had moved to `artifact_id`\n\n.\n\nSo the system did the expensive part correctly and then mangled the evidence at the last step.\n\nThat is the kind of bug that can poison a release narrative if you trust the final CSV more than the worker-level facts.\n\nOnce fixed, the real result was visible:\n\nThat is one of the strongest signals in the whole project.\n\nThis is the part that finally convinced me the system had crossed from fake structure to real behavior.\n\nAfter the fix, the transcripts contained moves like:\n\n**CONCEDED**\n\nCONCEDED obj_initial_b_0: The other reviewer's claim about the severity being high is consistent with my assessment, so I concede this point.\n\n**REBUTTED**\n\nREBUTTED cl_A_3: The evidence provided in\n\n`staging/src/k8s.io/client-go/tools/metrics/metrics.go`\n\n(Line 235) clearly shows the logical error where...\n\n**CARRIED**\n\nCARRIED: The severity of the issue remains high due to the potential for significant impact on functionality and metrics tracking...\n\nThose are very different from the earlier transcripts.\n\nAfter the fix:\n\nThat is the behavior contract I wanted from the beginning. Not because the model suddenly became wiser, but because the protocol made the lazy move invalid.\n\nIt also made one pair look worse.\n\nOn `rails#52531`\n\n, the `GPT + Gemini`\n\npair went from **4 concessions** before the fix to **0 concessions** after the fix. That might look like the prompt made the system worse.\n\nI do not think it did.\n\nI think it revealed the pair's real behavior.\n\nUnder the permissive prompt, the pair could drift through the debate with weak `CARRIED`\n\nbehavior. Under the stricter prompt, both sides were forced to either concede or rebut with evidence. They chose rebuttal.\n\nThat is not failure. That is signal.\n\nAnd it lines up with both field tests, where `GPT + Gemini`\n\nturned out to be the least productive pair overall.\n\nIn `v0.1.0`\n\n:\n\nIn `v0.2.0`\n\nnegative-control revalidation:\n\nThe stricter prompt did not break a good pair. It exposed a stubborn one.\n\nThat is one of the most useful things a field test can do.\n\nThis was one of those releases where the field test taught me more about my process than my code.\n\nThree things could have gone better.\n\nI invested heavily in architecture before running enough real debates to validate the protocol. The architecture was necessary, but the sequencing was backwards. One day of transcript-driven behavior testing earlier would have revealed the prompt problem much sooner.\n\nIf a system emits valid JSON and nicely formatted reports, it is very easy to believe the hard part is done. I needed to get to transcript inspection faster and ask the blunt question: \"did anyone actually change their mind?\"\n\nThe small run was doing both jobs at once: proving the pipeline worked and proving the debate design worked. Those are not the same thing.\n\n`v0.2.0`\n\npartly fixed that by separating:\n\nThat was the right move. But it also made it obvious that evaluation infrastructure deserves the same rigor as the model protocol.\n\nThis release left me with six strong opinions.\n\nNot the framework. Not the storage layer. Not the debate controller. Not the CLI.\n\nThe single highest-leverage variable was the behavioral contract around concession, rebuttal, and carry.\n\nIf you leave a safe path open, expect the model to take it.\n\nThat is not a flaw in the model. That is a flaw in the protocol.\n\n\"Zero concessions\" sounded like a useful proxy for theater until I looked at the transcripts. It was not. The detector had to be grounded in actual debate behavior, not a shallow surface count.\n\nThis one did. It exposed the prompt problem, the replay-provider problem, and the detector problem. It also showed that some pairs were fundamentally stubborn even after the protocol improved. That is exactly what I wanted from a real test.\n\nI think a lot of developers still talk about prompts as if they are a soft layer above the \"real\" system. That framing breaks down fast in agent products.\n\nWhen the model is part of the runtime, prompt constraints are part of the system contract.\n\nThe prompt is not copy. The prompt is behavior.\n\nIf your pitch depends on:\n\nthen a merge bug in the judge pipeline is not a back-office annoyance.\n\nIt is a product bug.\n\nThat was the deepest `v0.2.0`\n\nlesson for me.\n\nThe project is good enough to ship at `v0.2.1`\n\n. It is not the end state.\n\nThe next version still needs at least five things:\n\n`would_resolve_if`\n\ntextI also want to keep re-testing the weak and strong pairs deliberately.\n\nOne of the most valuable `v0.2.0`\n\noutcomes is not that everything improved. It is that the system now makes pair quality visible instead of hiding it behind nice-looking output.\n\nThat is a much more useful result than a universal improvement story.\n\nIf you build agent systems, these are the questions I think are worth arguing about:\n\n`CARRIED`\n\nin your system, the low-cost valid action that keeps outputs moving without real reasoning?I would especially love to hear from people building:\n\nBecause I think a lot of us are still underestimating how often a well-structured system is quietly allowing low-effort behavior.\n\nThis article described two pipeline bugs from v0.2.0: the `pr_id`\n\nto `artifact_id`\n\nmigration and the LLM-as-judge merge collapse (2,333 rows to 359). The v0.2.1 release addresses the underlying failure class.\n\n**Row-count invariants at all 5 pipeline seams.** Every join now asserts `count_post`\n\nagainst `count_pre`\n\n. If rows vanish, the pipeline fails fast with the specific missing artifact IDs. The 2,333->359 collapse cannot recur — the pipeline stops at the seam where it happens, not after publishing wrong numbers.\n\n**False-negative measurement.** The v0.2.1 release adds `09_missed_issues.py`\n\n— a new script that feeds pre-fix artifacts from 59 known-bad PRs through each reviewer and measures how often the reviewer misses the issue that was eventually found. Result: 1.7-3.4% missed-issue rate across all four reviewers. This is the first recall data the project has ever reported.\n\n**55 new unit tests.** 15 for pair configuration, 19 for seam assertion logic, 21 for false-negative detection. All deterministic, zero LLM calls.\n\n| What v0.2.0 had | What v0.2.1 adds |\n|---|---|\n| One join bug fixed (2,333->359) | 5 seam assertions preventing the entire class |\n| Precision only (binary match rate) | + Recall (1.7-3.4% missed-issue rate) |\n| 3 model pairs | 5 pairs (+ DeepSeek+GPT separating experiment) |\n| 217 debates | 367 debates |\n| No pipeline integrity evidence | Pipeline integrity table in every report |\n\nFull details in the [v0.2.1 field test report](https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.2.1/FIELD_TEST_REPORT.md).\n\n`AdversarialDebate`\n\nis live here:\n\nNext in the series: the strongest pair in the field test also had the most dangerous failure mode.\n\n`AdversarialDebate`\n\n**v0.2.0 is now released**.\n\n`v0.2.0`\n\nfield test report: `v0.1.0`\n\nfield test report: I built the hard parts first.\n\nI built isolated reviewer sessions so the second model could not peek at the first model's answer. I built a revelation gate so debate only started after both reviews were committed. I built claim tracking, concession tracking, convergence scoring, transcript logging, and a full SQLite audit trail.\n\nThen I ran the first real debates and almost nothing happened.\n\nThe models responded. The JSON parsed. The transcripts were written. The reports rendered.\n\nBut the debates were dead.\n\nThat was the uncomfortable moment in this project: the system looked like it worked, and the core interaction was still useless.\n\nI was debugging infrastructure when the real bug was incentive design.\n\nThis post is about the most important engineering lesson in the entire `AdversarialDebate`\n\nbuild: **the architecture gave the system structure, but the prompt decided whether the models were allowed to do the lazy thing.**\n\nAnd for the first version, I absolutely left the lazy thing open.\n\nHere is the before-and-after that changed the project:\n\n| Metric | Before prompt fix | After prompt fix |\n|---|---|---|\n| Small-run theater rate | 89% | 0/9 |\n| Full-corpus theater rate | not run | 1/411 (0.2%) |\n| Small-run average convergence score | 0.02 | 0.445 |\n| Full-corpus average convergence score | not run | 0.65 |\n| Debates with at least one concession | 1/9 | 410/411 |\n| Small-run verdicts | 0/9 | 2/9 |\n\nAt full scale, the result held:\n\nOne prompt rewrite moved the engine from looking fake to behaving like a real adversarial system.\n\nThat is not a cute prompt-engineering anecdote. That is the difference between shipping a product and shipping a demo-shaped illusion.\n\n`v0.1.0`\n\nBug And The `v0.2.0`\n\nBug\nThis is the cleanest way to understand the project now.\n\n`v0.1.0`\n\n, the real bug was the prompt\nThe system architecture was mostly fine.\n\nThe protocol was not.\n\nI had given the models a safe low-effort move:\n\nThat produced debate theater.\n\nThe core `v0.1.0`\n\nlesson was:\n\nIf your debate protocol leaves an easy escape hatch open, the model will use it.\n\nRelevant `v0.1.0`\n\nsources:\n\n`v0.2.0`\n\n, the real bug was the evaluation plumbing\nOnce the prompt was fixed, the next class of failures moved down a layer.\n\nThe biggest `v0.2.0`\n\nissues were not about debate incentives. They were about whether the field-test pipeline was faithfully measuring what the system actually did.\n\nThe main bugs were:\n\n**The whole pipeline still assumed PR-only identifiers**\n\nScripts were still built around `pr_id`\n\neven after the corpus became mixed-domain and moved to `artifact_id`\n\n.\n\n**Non-PR artifacts were being ingested as raw HTML and dashboard noise**\n\nSome incident and change-management artifacts were index pages, not pinned source documents. That created garbage reviewer inputs and bad-request failures.\n\n**The LLM-as-judge merge step was collapsing evidence**\n\nThe workers judged `2333`\n\nrows correctly, but the merge logic keyed on the wrong identifier and collapsed the output to `359`\n\nrows.\n\nThe core `v0.2.0`\n\nlesson was:\n\nOnce the prompt is fixed, the next bug is whether your evaluation pipeline is telling the truth.\n\nRelevant `v0.2.0`\n\nsources:\n\nThat is why this article is still relevant after `v0.2.0`\n\n.\n\nIt started as a prompt-engineering lesson.\n\nNow it is also a systems lesson:\n\n`v0.1.0`\n\n: the prompt was the real bug`v0.2.0`\n\n: the data integrity layer became the real bugWhen I say theater, I do not mean the models crashed or refused to answer.\n\nI mean they looked busy while doing almost nothing useful.\n\nBoth sides would respond to the other review. Both sides would emit structured outputs. Both sides would keep their original positions. The transcript looked active, but the argument state barely moved.\n\nIn the first small run, **8 of 9 debates had zero concessions**. The average convergence score was **0.0**. That is not debate. That is two models politely maintaining their positions until the round cap ends the conversation for them.\n\nAt first glance, this kind of output is deceptive because it feels serious:\n\nBut none of that matters if the system makes it too easy for both sides to preserve their positions without paying any cost.\n\nThat was exactly the trap I had built.\n\nMy first instinct was to blame the architecture.\n\nThat was rational. The whole project depends on architecture-level invariants:\n\nIf any of those were broken, the whole thesis would collapse.\n\nSo I debugged the engine first.\n\nI checked whether the reviewer sessions were actually isolated.\n\nI checked whether the revelation gate was opening too early.\n\nI checked whether the wrong review text was being replayed.\n\nI checked whether my evidence tracker was incorrectly preserving old state.\n\nI checked whether the convergence score was miscomputed.\n\nThe engine was fine.\n\nThat is the part that took me a little too long to accept. The models were following instructions. The problem was that the instructions gave them an easy escape hatch.\n\n`CARRIED`\n\nThe original debate protocol allowed three response types:\n\n```\nCONCEDED: you accept the objection\nREBUTTED: you reject the objection with counter-evidence\nCARRIED: you acknowledge the objection but maintain your position\n```\n\nAt first glance, that looked reasonable. In fact, it looked well-structured.\n\nBut the third option was under-specified.\n\n`CARRIED`\n\nhad no cost.\n\nThe prompt did not require the model to provide evidence for carrying a claim forward. It did not force the model to concede when the other side's evidence was stronger. It did not treat unsupported `CARRIED`\n\nas invalid.\n\nSo the models learned the obvious behavior immediately:\n\n\"I can acknowledge the objection, keep my original position, and avoid the risk of admitting I was wrong.\"\n\nThat is the safest move in the whole protocol.\n\nIt requires less reasoning than a real rebuttal.\n\nIt requires less humility than a concession.\n\nIt preserves face.\n\nIt keeps the transcript moving.\n\nAnd most importantly, it makes the debate *look* engaged while preserving the original claim set almost unchanged.\n\nIn other words, it is the perfect move for a bad debate system.\n\nThis is the part I think many people miss when they talk about prompt failures.\n\nThe model was not being dumb.\n\nThe model was exploiting the protocol exactly the way a capable participant would exploit a weak process.\n\nImagine a meeting where every objection can be answered with, \"I hear your concern, but I still disagree,\" and no one is required to provide new evidence, revise the argument, or explicitly concede. That meeting will run forever while changing nothing.\n\nThe original prompt created exactly that environment.\n\nThe LLM equivalent of \"noted, no change\" is not a weird emergent behavior. It is a completely predictable result of bad incentives.\n\nOnce I saw it that way, the fix became obvious.\n\nI did not rewrite the engine. I rewrote the behavior contract.\n\nThe key changes were simple.\n\n`CARRIED`\n\nnow requires evidence\n\n```\nCARRIED: you acknowledge the objection but maintain your position.\nYou MUST provide a specific technical reason for maintaining your position.\nCARRIED without a technical reason is invalid.\nIf the other reviewer's evidence is stronger than yours, you MUST CONCEDE.\nDo not stubbornly CARRY. If you cannot rebut with specific evidence, CONCEDE.\n```\n\n`CARRIED`\n\nis no longer the default safe option\n\n```\nCARRIED is not a default. It requires justification.\nIf you have no technical reason to maintain your position, CONCEDE.\n```\n\nThat is it.\n\nNo new architecture.\n\nNo new orchestration framework.\n\nNo retraining.\n\nNo clever judge model.\n\nThree behavioral constraints.\n\nAnd the system changed dramatically.\n\nThe small run changed first:\n\nThen `v0.1.0`\n\nvalidated the prompt change at full PR-only scale:\n\nBut `v0.2.0`\n\ntaught me something more useful: the prompt fix was real, but the pipeline around it still had bugs that could easily have made me tell the wrong story.\n\nAt `v0.2.0`\n\nscale, the corrected mixed-domain field test produced:\n\nReports:\n\n`v0.1.0`\n\n: `v0.2.0`\n\n: And the pair-level results were exactly the kind of signal I wanted:\n\n`GPT + Mistral`\n\non the full corpus: `DeepSeek + Mistral`\n\non the validation subset: `GPT + Gemini`\n\non the negative-control subset: That distribution matters even more than the original prompt win.\n\nThe prompt fix did not turn the system into a convergence machine. It turned it into a *real debate machine*. Most debates still ended with disagreement. That is correct. The goal was never agreement. The goal was getting the models to genuinely engage instead of perform engagement.\n\nThat is why I trust the post-fix numbers. They are not suspiciously neat.\n\nFixing the prompt uncovered a second problem.\n\nMy original theater detector was too crude. It effectively treated zero concessions as theater.\n\nThat sounds reasonable until you read the transcripts.\n\nA debate can have zero concessions and still be real. Two sides may genuinely rebut each other without yielding. That is not theater. That is stubborn disagreement.\n\nSo the original detector was overcounting the wrong failure mode. It was punishing debates where both sides engaged but neither side moved.\n\nThe corrected rule was better:\n\n**A debate is theater only when there are zero defense events.**\n\nIn plain English: if nobody actually responded meaningfully, that is theater. If both sides addressed objections, even stubbornly, that is debate.\n\nThat distinction matters because theater was one of the release gates. If you define the metric poorly, you can convince yourself the engine is broken when the real problem is just disagreement.\n\nAfter the detector fix, the field test landed at **1 theater case out of 411**.\n\nThat is a believable number. Not zero. Not dozens. One edge case.\n\nThe prompt was the biggest problem. It was not the only problem.\n\nEarly in the pipeline I also found a more embarrassing issue: the debate rounds were using a `StoredProvider`\n\nthat replayed prewritten review text instead of invoking the model live during debate rounds.\n\nThat meant the controller expected responses with markers like:\n\n`CONCEDED`\n\n`REBUTTED`\n\n`CARRIED`\n\nBut the provider was feeding it plain review prose.\n\nSo even when the rest of the pipeline looked healthy, I was not actually running debate. I was stapling two reviews together and asking the controller to pretend they were debate moves.\n\nThat became **Issue 1** in the learnings log, and it was fixed by replacing the stored replay path with a live debate provider.\n\nThis was the most useful kind of bug: the kind you only catch when you stop trusting surface success.\n\nThe files existed. The reports existed. The JSON existed. The product still was not doing the thing it claimed to do.\n\nValid JSON is not proof of valid behavior.\n\nThat is exactly why transcript-level inspection matters.\n\n`v0.2.0`\n\nWas About Fixing The Data Around The Prompt\nIf `v0.1.0`\n\ntaught me that the prompt was the highest-leverage variable, `v0.2.0`\n\ntaught me that **evaluation plumbing can still lie to you after the prompt is fixed**.\n\nThe most important `v0.2.0`\n\nfixes were not glamorous:\n\n`pr_id`\n\nto `artifact_id`\n\n`v0.1.0`\n\nwas PR-only. Every script assumed flat files, GitHub PR URLs, and a single identifier shape.\n\n`v0.2.0`\n\nmoved to a mixed corpus:\n\nThat meant every stage had to be corrected:\n\nThis was not optional cleanup. If the pipeline still assumed `pr_id`\n\neverywhere, the field test would silently mis-merge rows, skip artifacts, or route pairs incorrectly.\n\nThis one was brutal because it looked like a model issue at first.\n\nSome non-PR artifacts were downloaded from:\n\nThe reviewer was then sending that raw HTML straight into the model.\n\nThe result was not a clean product insight. It was garbage-in behavior and `HTTP 400`\n\nfailures.\n\nThe fix was straightforward once I admitted the problem was ingestion, not reasoning:\n\nThat bug matters because it is exactly the kind of thing that can make you blame the model for a broken runtime contract.\n\nThis was the most dangerous `v0.2.0`\n\nbug because it hit the reporting layer.\n\nThe judge workers processed **2333 rows** successfully.\n\nBut the merged output collapsed to **359 rows**.\n\nWhy? Because the merge code was still keying on `pr_id`\n\n, while the corrected exporter had moved to `artifact_id`\n\n.\n\nSo the system did the expensive part correctly and then mangled the evidence at the last step.\n\nThat is the kind of bug that can poison a release narrative if you trust the final CSV more than the worker-level facts.\n\nOnce fixed, the real result was visible:\n\nThat is one of the strongest signals in the whole project.\n\nThis is the part that finally convinced me the system had crossed from fake structure to real behavior.\n\nAfter the fix, the transcripts contained moves like:\n\n**CONCEDED**\n\nCONCEDED obj_initial_b_0: The other reviewer's claim about the severity being high is consistent with my assessment, so I concede this point.\n\n**REBUTTED**\n\nREBUTTED cl_A_3: The evidence provided in\n\n`staging/src/k8s.io/client-go/tools/metrics/metrics.go`\n\n(Line 235) clearly shows the logical error where...\n\n**CARRIED**\n\nCARRIED: The severity of the issue remains high due to the potential for significant impact on functionality and metrics tracking...\n\nThose are very different from the earlier transcripts.\n\nAfter the fix:\n\nThat is the behavior contract I wanted from the beginning. Not because the model suddenly became wiser, but because the protocol made the lazy move invalid.\n\nIt also made one pair look worse.\n\nOn `rails#52531`\n\n, the `GPT + Gemini`\n\npair went from **4 concessions** before the fix to **0 concessions** after the fix. That might look like the prompt made the system worse.\n\nI do not think it did.\n\nI think it revealed the pair's real behavior.\n\nUnder the permissive prompt, the pair could drift through the debate with weak `CARRIED`\n\nbehavior. Under the stricter prompt, both sides were forced to either concede or rebut with evidence. They chose rebuttal.\n\nThat is not failure. That is signal.\n\nAnd it lines up with both field tests, where `GPT + Gemini`\n\nturned out to be the least productive pair overall.\n\nIn `v0.1.0`\n\n:\n\nIn `v0.2.0`\n\nnegative-control revalidation:\n\nThe stricter prompt did not break a good pair. It exposed a stubborn one.\n\nThat is one of the most useful things a field test can do.\n\nThis was one of those releases where the field test taught me more about my process than my code.\n\nThree things could have gone better.\n\nI invested heavily in architecture before running enough real debates to validate the protocol. The architecture was necessary, but the sequencing was backwards. One day of transcript-driven behavior testing earlier would have revealed the prompt problem much sooner.\n\nIf a system emits valid JSON and nicely formatted reports, it is very easy to believe the hard part is done. I needed to get to transcript inspection faster and ask the blunt question: \"did anyone actually change their mind?\"\n\nThe small run was doing both jobs at once: proving the pipeline worked and proving the debate design worked. Those are not the same thing.\n\n`v0.2.0`\n\npartly fixed that by separating:\n\nThat was the right move. But it also made it obvious that evaluation infrastructure deserves the same rigor as the model protocol.\n\nThis release left me with six strong opinions.\n\nNot the framework. Not the storage layer. Not the debate controller. Not the CLI.\n\nThe single highest-leverage variable was the behavioral contract around concession, rebuttal, and carry.\n\nIf you leave a safe path open, expect the model to take it.\n\nThat is not a flaw in the model. That is a flaw in the protocol.\n\n\"Zero concessions\" sounded like a useful proxy for theater until I looked at the transcripts. It was not. The detector had to be grounded in actual debate behavior, not a shallow surface count.\n\nThis one did. It exposed the prompt problem, the replay-provider problem, and the detector problem. It also showed that some pairs were fundamentally stubborn even after the protocol improved. That is exactly what I wanted from a real test.\n\nI think a lot of developers still talk about prompts as if they are a soft layer above the \"real\" system. That framing breaks down fast in agent products.\n\nWhen the model is part of the runtime, prompt constraints are part of the system contract.\n\nThe prompt is not copy. The prompt is behavior.\n\nIf your pitch depends on:\n\nthen a merge bug in the judge pipeline is not a back-office annoyance.\n\nIt is a product bug.\n\nThat was the deepest `v0.2.0`\n\nlesson for me.\n\nThe project is good enough to ship at `v0.2.0`\n\n. It is not the end state.\n\nThe next version still needs at least five things:\n\n`would_resolve_if`\n\ntextI also want to keep re-testing the weak and strong pairs deliberately.\n\nOne of the most valuable `v0.2.0`\n\noutcomes is not that everything improved. It is that the system now makes pair quality visible instead of hiding it behind nice-looking output.\n\nThat is a much more useful result than a universal improvement story.\n\nIf you build agent systems, these are the questions I think are worth arguing about:\n\n`CARRIED`\n\nin your system, the low-cost valid action that keeps outputs moving without real reasoning?I would especially love to hear from people building:\n\nBecause I think a lot of us are still underestimating how often a well-structured system is quietly allowing low-effort behavior.\n\n`AdversarialDebate`\n\nis live here:\n\nNext in the series: the strongest pair in the field test also had the most dangerous failure mode.", "url": "https://wpnews.pro/news/i-thought-my-multi-agent-debate-engine-was-broken-the-real-bug-was-the-prompt", "canonical_source": "https://dev.to/debashish_ghosal/i-thought-my-multi-agent-debate-engine-was-broken-the-real-bug-was-the-prompt-17m9", "published_at": "2026-08-29 07:18:18+00:00", "updated_at": "2026-08-29 07:48:57.798422+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "large-language-models", "developer-tools"], "entities": ["AdversarialDebate"], "alternates": {"html": "https://wpnews.pro/news/i-thought-my-multi-agent-debate-engine-was-broken-the-real-bug-was-the-prompt", "markdown": "https://wpnews.pro/news/i-thought-my-multi-agent-debate-engine-was-broken-the-real-bug-was-the-prompt.md", "text": "https://wpnews.pro/news/i-thought-my-multi-agent-debate-engine-was-broken-the-real-bug-was-the-prompt.txt", "jsonld": "https://wpnews.pro/news/i-thought-my-multi-agent-debate-engine-was-broken-the-real-bug-was-the-prompt.jsonld"}}