[v0.2.1 RELEASED]— Aug 28, 2026.[Release notes]·[Field test report]·[PyPI]
v0.2.1 Update:The 2,333->359 join collapse described in this article is now structurally prevented. v0.2.1 adds row-count invariant assertions at all 5 pipeline seams — the pipeline fails fast if any rows are silently lost. Also new: false-negative measurement (1.7-3.4% missed-issue rate, first recall data ever reported) and 55 new unit tests.Previously:
[Most AI Second Opinions Are Fake]
AdversarialDebate
v0.2.0 is now released — v0.2.1 shipped Aug 28, 2026.
pip install adversarial-debate==0.2.1
v0.2.1
field test report: v0.2.0
field test report: I built the hard parts first.
I built isolated reviewer sessions so the second model could not peek at the first model's answer. I built a revelation gate so debate only started after both reviews were committed. I built claim tracking, concession tracking, convergence scoring, transcript logging, and a full SQLite audit trail.
Then I ran the first real debates and almost nothing happened.
The models responded. The JSON parsed. The transcripts were written. The reports rendered.
But the debates were dead.
That was the uncomfortable moment in this project: the system looked like it worked, and the core interaction was still useless.
I was debugging infrastructure when the real bug was incentive design.
This post is about the most important engineering lesson in the entire AdversarialDebate
build: the architecture gave the system structure, but the prompt decided whether the models were allowed to do the lazy thing.
And for the first version, I absolutely left the lazy thing open.
Here is the before-and-after that changed the project:
| Metric | Before prompt fix | After prompt fix |
|---|---|---|
| Small-run theater rate | 89% | 0/9 |
| Full-corpus theater rate | not run | 1/411 (0.2%) |
| Small-run average convergence score | 0.02 | 0.445 |
| Full-corpus average convergence score | not run | 0.65 |
| Debates with at least one concession | 1/9 | 410/411 |
| Small-run verdicts | 0/9 | 2/9 |
At full scale, the result held:
One prompt rewrite moved the engine from looking fake to behaving like a real adversarial system.
That is not a cute prompt-engineering anecdote. That is the difference between shipping a product and shipping a demo-shaped illusion.
v0.1.0
Bug And The v0.2.0
Bug This is the cleanest way to understand the project now.
v0.1.0
, the real bug was the prompt The system architecture was mostly fine.
The protocol was not.
I had given the models a safe low-effort move:
That produced debate theater.
The core v0.1.0
lesson was:
If your debate protocol leaves an easy escape hatch open, the model will use it.
Relevant v0.1.0
sources:
v0.2.0
, the real bug was the evaluation plumbing Once the prompt was fixed, the next class of failures moved down a layer.
The biggest v0.2.0
issues were not about debate incentives. They were about whether the field-test pipeline was faithfully measuring what the system actually did.
The main bugs were:
The whole pipeline still assumed PR-only identifiers
Scripts were still built around pr_id
even after the corpus became mixed-domain and moved to artifact_id
.
Non-PR artifacts were being ingested as raw HTML and dashboard noise
Some incident and change-management artifacts were index pages, not pinned source documents. That created garbage reviewer inputs and bad-request failures.
The LLM-as-judge merge step was collapsing evidence
The workers judged 2333
rows correctly, but the merge logic keyed on the wrong identifier and collapsed the output to 359
rows.
The core v0.2.0
lesson was:
Once the prompt is fixed, the next bug is whether your evaluation pipeline is telling the truth.
Relevant v0.2.0
sources:
That is why this article is still relevant after v0.2.0
.
It started as a prompt-engineering lesson.
Now it is also a systems lesson:
v0.1.0
: the prompt was the real bugv0.2.0
: the data integrity layer became the real bugWhen I say theater, I do not mean the models crashed or refused to answer.
I mean they looked busy while doing almost nothing useful.
Both sides would respond to the other review. Both sides would emit structured outputs. Both sides would keep their original positions. The transcript looked active, but the argument state barely moved.
In the first small run, 8 of 9 debates had zero concessions. The average convergence score was 0.0. That is not debate. That is two models politely maintaining their positions until the round cap ends the conversation for them.
At first glance, this kind of output is deceptive because it feels serious:
But none of that matters if the system makes it too easy for both sides to preserve their positions without paying any cost.
That was exactly the trap I had built.
My first instinct was to blame the architecture.
That was rational. The whole project depends on architecture-level invariants:
If any of those were broken, the whole thesis would collapse.
So I debugged the engine first.
I checked whether the reviewer sessions were actually isolated.
I checked whether the revelation gate was opening too early.
I checked whether the wrong review text was being replayed.
I checked whether my evidence tracker was incorrectly preserving old state.
I checked whether the convergence score was miscomputed.
The engine was fine.
That is the part that took me a little too long to accept. The models were following instructions. The problem was that the instructions gave them an easy escape hatch.
CARRIED
The original debate protocol allowed three response types:
CONCEDED: you accept the objection
REBUTTED: you reject the objection with counter-evidence
CARRIED: you acknowledge the objection but maintain your position
At first glance, that looked reasonable. In fact, it looked well-structured.
But the third option was under-specified.
CARRIED
had no cost.
The prompt did not require the model to provide evidence for carrying a claim forward. It did not force the model to concede when the other side's evidence was stronger. It did not treat unsupported CARRIED
as invalid.
So the models learned the obvious behavior immediately:
"I can acknowledge the objection, keep my original position, and avoid the risk of admitting I was wrong."
That is the safest move in the whole protocol.
It requires less reasoning than a real rebuttal.
It requires less humility than a concession.
It preserves face.
It keeps the transcript moving.
And most importantly, it makes the debate look engaged while preserving the original claim set almost unchanged.
In other words, it is the perfect move for a bad debate system.
This is the part I think many people miss when they talk about prompt failures.
The model was not being dumb.
The model was exploiting the protocol exactly the way a capable participant would exploit a weak process.
Imagine a meeting where every objection can be answered with, "I hear your concern, but I still disagree," and no one is required to provide new evidence, revise the argument, or explicitly concede. That meeting will run forever while changing nothing.
The original prompt created exactly that environment.
The LLM equivalent of "noted, no change" is not a weird emergent behavior. It is a completely predictable result of bad incentives.
Once I saw it that way, the fix became obvious.
I did not rewrite the engine. I rewrote the behavior contract.
The key changes were simple.
CARRIED
now requires evidence
CARRIED: you acknowledge the objection but maintain your position.
You MUST provide a specific technical reason for maintaining your position.
CARRIED without a technical reason is invalid.
If the other reviewer's evidence is stronger than yours, you MUST CONCEDE.
Do not stubbornly CARRY. If you cannot rebut with specific evidence, CONCEDE.
CARRIED
is no longer the default safe option
CARRIED is not a default. It requires justification.
If you have no technical reason to maintain your position, CONCEDE.
That is it.
No new architecture.
No new orchestration framework.
No retraining.
No clever judge model.
Three behavioral constraints.
And the system changed dramatically.
The small run changed first:
Then v0.1.0
validated the prompt change at full PR-only scale:
But v0.2.0
taught me something more useful: the prompt fix was real, but the pipeline around it still had bugs that could easily have made me tell the wrong story.
At v0.2.0
scale, the corrected mixed-domain field test produced:
Reports:
v0.1.0
: v0.2.0
: And the pair-level results were exactly the kind of signal I wanted:
GPT + Mistral
on the full corpus: DeepSeek + Mistral
on the validation subset: GPT + Gemini
on the negative-control subset: That distribution matters even more than the original prompt win.
The prompt fix did not turn the system into a convergence machine. It turned it into a real debate machine. Most debates still ended with disagreement. That is correct. The goal was never agreement. The goal was getting the models to genuinely engage instead of perform engagement.
That is why I trust the post-fix numbers. They are not suspiciously neat.
Fixing the prompt uncovered a second problem.
My original theater detector was too crude. It effectively treated zero concessions as theater.
That sounds reasonable until you read the transcripts.
A debate can have zero concessions and still be real. Two sides may genuinely rebut each other without yielding. That is not theater. That is stubborn disagreement.
So the original detector was overcounting the wrong failure mode. It was punishing debates where both sides engaged but neither side moved.
The corrected rule was better:
A debate is theater only when there are zero defense events.
In plain English: if nobody actually responded meaningfully, that is theater. If both sides addressed objections, even stubbornly, that is debate.
That distinction matters because theater was one of the release gates. If you define the metric poorly, you can convince yourself the engine is broken when the real problem is just disagreement.
After the detector fix, the field test landed at 1 theater case out of 411.
That is a believable number. Not zero. Not dozens. One edge case.
The prompt was the biggest problem. It was not the only problem.
Early in the pipeline I also found a more embarrassing issue: the debate rounds were using a StoredProvider
that replayed prewritten review text instead of invoking the model live during debate rounds.
That meant the controller expected responses with markers like:
CONCEDED
REBUTTED
CARRIED
But the provider was feeding it plain review prose.
So even when the rest of the pipeline looked healthy, I was not actually running debate. I was stapling two reviews together and asking the controller to pretend they were debate moves.
That became Issue 1 in the learnings log, and it was fixed by replacing the stored replay path with a live debate provider.
This was the most useful kind of bug: the kind you only catch when you stop trusting surface success.
The files existed. The reports existed. The JSON existed. The product still was not doing the thing it claimed to do.
Valid JSON is not proof of valid behavior.
That is exactly why transcript-level inspection matters.
v0.2.0
Was About Fixing The Data Around The Prompt
If v0.1.0
taught me that the prompt was the highest-leverage variable, v0.2.0
taught me that evaluation plumbing can still lie to you after the prompt is fixed.
The most important v0.2.0
fixes were not glamorous:
pr_id
to artifact_id
v0.1.0
was PR-only. Every script assumed flat files, GitHub PR URLs, and a single identifier shape.
v0.2.0
moved to a mixed corpus:
That meant every stage had to be corrected:
This was not optional cleanup. If the pipeline still assumed pr_id
everywhere, the field test would silently mis-merge rows, skip artifacts, or route pairs incorrectly.
This one was brutal because it looked like a model issue at first.
Some non-PR artifacts were downloaded from:
The reviewer was then sending that raw HTML straight into the model.
The result was not a clean product insight. It was garbage-in behavior and HTTP 400
failures.
The fix was straightforward once I admitted the problem was ingestion, not reasoning:
That bug matters because it is exactly the kind of thing that can make you blame the model for a broken runtime contract.
This was the most dangerous v0.2.0
bug because it hit the reporting layer.
The judge workers processed 2333 rows successfully.
But the merged output collapsed to 359 rows.
Why? Because the merge code was still keying on pr_id
, while the corrected exporter had moved to artifact_id
.
So the system did the expensive part correctly and then mangled the evidence at the last step.
That is the kind of bug that can poison a release narrative if you trust the final CSV more than the worker-level facts.
Once fixed, the real result was visible:
That is one of the strongest signals in the whole project.
This is the part that finally convinced me the system had crossed from fake structure to real behavior.
After the fix, the transcripts contained moves like:
CONCEDED
CONCEDED obj_initial_b_0: The other reviewer's claim about the severity being high is consistent with my assessment, so I concede this point.
REBUTTED
REBUTTED cl_A_3: The evidence provided in
staging/src/k8s.io/client-go/tools/metrics/metrics.go
(Line 235) clearly shows the logical error where...
CARRIED
CARRIED: The severity of the issue remains high due to the potential for significant impact on functionality and metrics tracking...
Those are very different from the earlier transcripts.
After the fix:
That is the behavior contract I wanted from the beginning. Not because the model suddenly became wiser, but because the protocol made the lazy move invalid.
It also made one pair look worse.
On rails#52531
, the GPT + Gemini
pair went from 4 concessions before the fix to 0 concessions after the fix. That might look like the prompt made the system worse.
I do not think it did.
I think it revealed the pair's real behavior.
Under the permissive prompt, the pair could drift through the debate with weak CARRIED
behavior. Under the stricter prompt, both sides were forced to either concede or rebut with evidence. They chose rebuttal.
That is not failure. That is signal.
And it lines up with both field tests, where GPT + Gemini
turned out to be the least productive pair overall.
In v0.1.0
:
In v0.2.0
negative-control revalidation:
The stricter prompt did not break a good pair. It exposed a stubborn one.
That is one of the most useful things a field test can do.
This was one of those releases where the field test taught me more about my process than my code.
Three things could have gone better.
I invested heavily in architecture before running enough real debates to validate the protocol. The architecture was necessary, but the sequencing was backwards. One day of transcript-driven behavior testing earlier would have revealed the prompt problem much sooner.
If a system emits valid JSON and nicely formatted reports, it is very easy to believe the hard part is done. I needed to get to transcript inspection faster and ask the blunt question: "did anyone actually change their mind?"
The small run was doing both jobs at once: proving the pipeline worked and proving the debate design worked. Those are not the same thing.
v0.2.0
partly fixed that by separating:
That was the right move. But it also made it obvious that evaluation infrastructure deserves the same rigor as the model protocol.
This release left me with six strong opinions.
Not the framework. Not the storage layer. Not the debate controller. Not the CLI.
The single highest-leverage variable was the behavioral contract around concession, rebuttal, and carry.
If you leave a safe path open, expect the model to take it.
That is not a flaw in the model. That is a flaw in the protocol.
"Zero concessions" sounded like a useful proxy for theater until I looked at the transcripts. It was not. The detector had to be grounded in actual debate behavior, not a shallow surface count.
This one did. It exposed the prompt problem, the replay-provider problem, and the detector problem. It also showed that some pairs were fundamentally stubborn even after the protocol improved. That is exactly what I wanted from a real test.
I think a lot of developers still talk about prompts as if they are a soft layer above the "real" system. That framing breaks down fast in agent products.
When the model is part of the runtime, prompt constraints are part of the system contract.
The prompt is not copy. The prompt is behavior.
If your pitch depends on:
then a merge bug in the judge pipeline is not a back-office annoyance.
It is a product bug.
That was the deepest v0.2.0
lesson for me.
The project is good enough to ship at v0.2.1
. It is not the end state.
The next version still needs at least five things:
would_resolve_if
textI also want to keep re-testing the weak and strong pairs deliberately.
One of the most valuable v0.2.0
outcomes is not that everything improved. It is that the system now makes pair quality visible instead of hiding it behind nice-looking output.
That is a much more useful result than a universal improvement story.
If you build agent systems, these are the questions I think are worth arguing about:
CARRIED
in your system, the low-cost valid action that keeps outputs moving without real reasoning?I would especially love to hear from people building:
Because I think a lot of us are still underestimating how often a well-structured system is quietly allowing low-effort behavior.
This article described two pipeline bugs from v0.2.0: the pr_id
to artifact_id
migration and the LLM-as-judge merge collapse (2,333 rows to 359). The v0.2.1 release addresses the underlying failure class.
Row-count invariants at all 5 pipeline seams. Every join now asserts count_post
against count_pre
. If rows vanish, the pipeline fails fast with the specific missing artifact IDs. The 2,333->359 collapse cannot recur — the pipeline stops at the seam where it happens, not after publishing wrong numbers.
False-negative measurement. The v0.2.1 release adds 09_missed_issues.py
— a new script that feeds pre-fix artifacts from 59 known-bad PRs through each reviewer and measures how often the reviewer misses the issue that was eventually found. Result: 1.7-3.4% missed-issue rate across all four reviewers. This is the first recall data the project has ever reported.
55 new unit tests. 15 for pair configuration, 19 for seam assertion logic, 21 for false-negative detection. All deterministic, zero LLM calls.
| What v0.2.0 had | What v0.2.1 adds |
|---|---|
| One join bug fixed (2,333->359) | 5 seam assertions preventing the entire class |
| Precision only (binary match rate) | + Recall (1.7-3.4% missed-issue rate) |
| 3 model pairs | 5 pairs (+ DeepSeek+GPT separating experiment) |
| 217 debates | 367 debates |
| No pipeline integrity evidence | Pipeline integrity table in every report |
Full details in the v0.2.1 field test report.
AdversarialDebate
is live here:
Next in the series: the strongest pair in the field test also had the most dangerous failure mode.
AdversarialDebate
v0.2.0 is now released.
v0.2.0
field test report: v0.1.0
field test report: I built the hard parts first.
I built isolated reviewer sessions so the second model could not peek at the first model's answer. I built a revelation gate so debate only started after both reviews were committed. I built claim tracking, concession tracking, convergence scoring, transcript logging, and a full SQLite audit trail.
Then I ran the first real debates and almost nothing happened.
The models responded. The JSON parsed. The transcripts were written. The reports rendered.
But the debates were dead.
That was the uncomfortable moment in this project: the system looked like it worked, and the core interaction was still useless.
I was debugging infrastructure when the real bug was incentive design.
This post is about the most important engineering lesson in the entire AdversarialDebate
build: the architecture gave the system structure, but the prompt decided whether the models were allowed to do the lazy thing.
And for the first version, I absolutely left the lazy thing open.
Here is the before-and-after that changed the project:
| Metric | Before prompt fix | After prompt fix |
|---|---|---|
| Small-run theater rate | 89% | 0/9 |
| Full-corpus theater rate | not run | 1/411 (0.2%) |
| Small-run average convergence score | 0.02 | 0.445 |
| Full-corpus average convergence score | not run | 0.65 |
| Debates with at least one concession | 1/9 | 410/411 |
| Small-run verdicts | 0/9 | 2/9 |
At full scale, the result held:
One prompt rewrite moved the engine from looking fake to behaving like a real adversarial system.
That is not a cute prompt-engineering anecdote. That is the difference between shipping a product and shipping a demo-shaped illusion.
v0.1.0
Bug And The v0.2.0
Bug This is the cleanest way to understand the project now.
v0.1.0
, the real bug was the prompt The system architecture was mostly fine.
The protocol was not.
I had given the models a safe low-effort move:
That produced debate theater.
The core v0.1.0
lesson was:
If your debate protocol leaves an easy escape hatch open, the model will use it.
Relevant v0.1.0
sources:
v0.2.0
, the real bug was the evaluation plumbing Once the prompt was fixed, the next class of failures moved down a layer.
The biggest v0.2.0
issues were not about debate incentives. They were about whether the field-test pipeline was faithfully measuring what the system actually did.
The main bugs were:
The whole pipeline still assumed PR-only identifiers
Scripts were still built around pr_id
even after the corpus became mixed-domain and moved to artifact_id
.
Non-PR artifacts were being ingested as raw HTML and dashboard noise
Some incident and change-management artifacts were index pages, not pinned source documents. That created garbage reviewer inputs and bad-request failures.
The LLM-as-judge merge step was collapsing evidence
The workers judged 2333
rows correctly, but the merge logic keyed on the wrong identifier and collapsed the output to 359
rows.
The core v0.2.0
lesson was:
Once the prompt is fixed, the next bug is whether your evaluation pipeline is telling the truth.
Relevant v0.2.0
sources:
That is why this article is still relevant after v0.2.0
.
It started as a prompt-engineering lesson.
Now it is also a systems lesson:
v0.1.0
: the prompt was the real bugv0.2.0
: the data integrity layer became the real bugWhen I say theater, I do not mean the models crashed or refused to answer.
I mean they looked busy while doing almost nothing useful.
Both sides would respond to the other review. Both sides would emit structured outputs. Both sides would keep their original positions. The transcript looked active, but the argument state barely moved.
In the first small run, 8 of 9 debates had zero concessions. The average convergence score was 0.0. That is not debate. That is two models politely maintaining their positions until the round cap ends the conversation for them.
At first glance, this kind of output is deceptive because it feels serious:
But none of that matters if the system makes it too easy for both sides to preserve their positions without paying any cost.
That was exactly the trap I had built.
My first instinct was to blame the architecture.
That was rational. The whole project depends on architecture-level invariants:
If any of those were broken, the whole thesis would collapse.
So I debugged the engine first.
I checked whether the reviewer sessions were actually isolated.
I checked whether the revelation gate was opening too early.
I checked whether the wrong review text was being replayed.
I checked whether my evidence tracker was incorrectly preserving old state.
I checked whether the convergence score was miscomputed.
The engine was fine.
That is the part that took me a little too long to accept. The models were following instructions. The problem was that the instructions gave them an easy escape hatch.
CARRIED
The original debate protocol allowed three response types:
CONCEDED: you accept the objection
REBUTTED: you reject the objection with counter-evidence
CARRIED: you acknowledge the objection but maintain your position
At first glance, that looked reasonable. In fact, it looked well-structured.
But the third option was under-specified.
CARRIED
had no cost.
The prompt did not require the model to provide evidence for carrying a claim forward. It did not force the model to concede when the other side's evidence was stronger. It did not treat unsupported CARRIED
as invalid.
So the models learned the obvious behavior immediately:
"I can acknowledge the objection, keep my original position, and avoid the risk of admitting I was wrong."
That is the safest move in the whole protocol.
It requires less reasoning than a real rebuttal.
It requires less humility than a concession.
It preserves face.
It keeps the transcript moving.
And most importantly, it makes the debate look engaged while preserving the original claim set almost unchanged.
In other words, it is the perfect move for a bad debate system.
This is the part I think many people miss when they talk about prompt failures.
The model was not being dumb.
The model was exploiting the protocol exactly the way a capable participant would exploit a weak process.
Imagine a meeting where every objection can be answered with, "I hear your concern, but I still disagree," and no one is required to provide new evidence, revise the argument, or explicitly concede. That meeting will run forever while changing nothing.
The original prompt created exactly that environment.
The LLM equivalent of "noted, no change" is not a weird emergent behavior. It is a completely predictable result of bad incentives.
Once I saw it that way, the fix became obvious.
I did not rewrite the engine. I rewrote the behavior contract.
The key changes were simple.
CARRIED
now requires evidence
CARRIED: you acknowledge the objection but maintain your position.
You MUST provide a specific technical reason for maintaining your position.
CARRIED without a technical reason is invalid.
If the other reviewer's evidence is stronger than yours, you MUST CONCEDE.
Do not stubbornly CARRY. If you cannot rebut with specific evidence, CONCEDE.
CARRIED
is no longer the default safe option
CARRIED is not a default. It requires justification.
If you have no technical reason to maintain your position, CONCEDE.
That is it.
No new architecture.
No new orchestration framework.
No retraining.
No clever judge model.
Three behavioral constraints.
And the system changed dramatically.
The small run changed first:
Then v0.1.0
validated the prompt change at full PR-only scale:
But v0.2.0
taught me something more useful: the prompt fix was real, but the pipeline around it still had bugs that could easily have made me tell the wrong story.
At v0.2.0
scale, the corrected mixed-domain field test produced:
Reports:
v0.1.0
: v0.2.0
: And the pair-level results were exactly the kind of signal I wanted:
GPT + Mistral
on the full corpus: DeepSeek + Mistral
on the validation subset: GPT + Gemini
on the negative-control subset: That distribution matters even more than the original prompt win.
The prompt fix did not turn the system into a convergence machine. It turned it into a real debate machine. Most debates still ended with disagreement. That is correct. The goal was never agreement. The goal was getting the models to genuinely engage instead of perform engagement.
That is why I trust the post-fix numbers. They are not suspiciously neat.
Fixing the prompt uncovered a second problem.
My original theater detector was too crude. It effectively treated zero concessions as theater.
That sounds reasonable until you read the transcripts.
A debate can have zero concessions and still be real. Two sides may genuinely rebut each other without yielding. That is not theater. That is stubborn disagreement.
So the original detector was overcounting the wrong failure mode. It was punishing debates where both sides engaged but neither side moved.
The corrected rule was better:
A debate is theater only when there are zero defense events.
In plain English: if nobody actually responded meaningfully, that is theater. If both sides addressed objections, even stubbornly, that is debate.
That distinction matters because theater was one of the release gates. If you define the metric poorly, you can convince yourself the engine is broken when the real problem is just disagreement.
After the detector fix, the field test landed at 1 theater case out of 411.
That is a believable number. Not zero. Not dozens. One edge case.
The prompt was the biggest problem. It was not the only problem.
Early in the pipeline I also found a more embarrassing issue: the debate rounds were using a StoredProvider
that replayed prewritten review text instead of invoking the model live during debate rounds.
That meant the controller expected responses with markers like:
CONCEDED
REBUTTED
CARRIED
But the provider was feeding it plain review prose.
So even when the rest of the pipeline looked healthy, I was not actually running debate. I was stapling two reviews together and asking the controller to pretend they were debate moves.
That became Issue 1 in the learnings log, and it was fixed by replacing the stored replay path with a live debate provider.
This was the most useful kind of bug: the kind you only catch when you stop trusting surface success.
The files existed. The reports existed. The JSON existed. The product still was not doing the thing it claimed to do.
Valid JSON is not proof of valid behavior.
That is exactly why transcript-level inspection matters.
v0.2.0
Was About Fixing The Data Around The Prompt
If v0.1.0
taught me that the prompt was the highest-leverage variable, v0.2.0
taught me that evaluation plumbing can still lie to you after the prompt is fixed.
The most important v0.2.0
fixes were not glamorous:
pr_id
to artifact_id
v0.1.0
was PR-only. Every script assumed flat files, GitHub PR URLs, and a single identifier shape.
v0.2.0
moved to a mixed corpus:
That meant every stage had to be corrected:
This was not optional cleanup. If the pipeline still assumed pr_id
everywhere, the field test would silently mis-merge rows, skip artifacts, or route pairs incorrectly.
This one was brutal because it looked like a model issue at first.
Some non-PR artifacts were downloaded from:
The reviewer was then sending that raw HTML straight into the model.
The result was not a clean product insight. It was garbage-in behavior and HTTP 400
failures.
The fix was straightforward once I admitted the problem was ingestion, not reasoning:
That bug matters because it is exactly the kind of thing that can make you blame the model for a broken runtime contract.
This was the most dangerous v0.2.0
bug because it hit the reporting layer.
The judge workers processed 2333 rows successfully.
But the merged output collapsed to 359 rows.
Why? Because the merge code was still keying on pr_id
, while the corrected exporter had moved to artifact_id
.
So the system did the expensive part correctly and then mangled the evidence at the last step.
That is the kind of bug that can poison a release narrative if you trust the final CSV more than the worker-level facts.
Once fixed, the real result was visible:
That is one of the strongest signals in the whole project.
This is the part that finally convinced me the system had crossed from fake structure to real behavior.
After the fix, the transcripts contained moves like:
CONCEDED
CONCEDED obj_initial_b_0: The other reviewer's claim about the severity being high is consistent with my assessment, so I concede this point.
REBUTTED
REBUTTED cl_A_3: The evidence provided in
staging/src/k8s.io/client-go/tools/metrics/metrics.go
(Line 235) clearly shows the logical error where...
CARRIED
CARRIED: The severity of the issue remains high due to the potential for significant impact on functionality and metrics tracking...
Those are very different from the earlier transcripts.
After the fix:
That is the behavior contract I wanted from the beginning. Not because the model suddenly became wiser, but because the protocol made the lazy move invalid.
It also made one pair look worse.
On rails#52531
, the GPT + Gemini
pair went from 4 concessions before the fix to 0 concessions after the fix. That might look like the prompt made the system worse.
I do not think it did.
I think it revealed the pair's real behavior.
Under the permissive prompt, the pair could drift through the debate with weak CARRIED
behavior. Under the stricter prompt, both sides were forced to either concede or rebut with evidence. They chose rebuttal.
That is not failure. That is signal.
And it lines up with both field tests, where GPT + Gemini
turned out to be the least productive pair overall.
In v0.1.0
:
In v0.2.0
negative-control revalidation:
The stricter prompt did not break a good pair. It exposed a stubborn one.
That is one of the most useful things a field test can do.
This was one of those releases where the field test taught me more about my process than my code.
Three things could have gone better.
I invested heavily in architecture before running enough real debates to validate the protocol. The architecture was necessary, but the sequencing was backwards. One day of transcript-driven behavior testing earlier would have revealed the prompt problem much sooner.
If a system emits valid JSON and nicely formatted reports, it is very easy to believe the hard part is done. I needed to get to transcript inspection faster and ask the blunt question: "did anyone actually change their mind?"
The small run was doing both jobs at once: proving the pipeline worked and proving the debate design worked. Those are not the same thing.
v0.2.0
partly fixed that by separating:
That was the right move. But it also made it obvious that evaluation infrastructure deserves the same rigor as the model protocol.
This release left me with six strong opinions.
Not the framework. Not the storage layer. Not the debate controller. Not the CLI.
The single highest-leverage variable was the behavioral contract around concession, rebuttal, and carry.
If you leave a safe path open, expect the model to take it.
That is not a flaw in the model. That is a flaw in the protocol.
"Zero concessions" sounded like a useful proxy for theater until I looked at the transcripts. It was not. The detector had to be grounded in actual debate behavior, not a shallow surface count.
This one did. It exposed the prompt problem, the replay-provider problem, and the detector problem. It also showed that some pairs were fundamentally stubborn even after the protocol improved. That is exactly what I wanted from a real test.
I think a lot of developers still talk about prompts as if they are a soft layer above the "real" system. That framing breaks down fast in agent products.
When the model is part of the runtime, prompt constraints are part of the system contract.
The prompt is not copy. The prompt is behavior.
If your pitch depends on:
then a merge bug in the judge pipeline is not a back-office annoyance.
It is a product bug.
That was the deepest v0.2.0
lesson for me.
The project is good enough to ship at v0.2.0
. It is not the end state.
The next version still needs at least five things:
would_resolve_if
textI also want to keep re-testing the weak and strong pairs deliberately.
One of the most valuable v0.2.0
outcomes is not that everything improved. It is that the system now makes pair quality visible instead of hiding it behind nice-looking output.
That is a much more useful result than a universal improvement story.
If you build agent systems, these are the questions I think are worth arguing about:
CARRIED
in your system, the low-cost valid action that keeps outputs moving without real reasoning?I would especially love to hear from people building:
Because I think a lot of us are still underestimating how often a well-structured system is quietly allowing low-effort behavior.
AdversarialDebate
is live here:
Next in the series: the strongest pair in the field test also had the most dangerous failure mode.