I've been building AI code review tools since 2024. Every one I've tried, and every one on the market, has the same structural flaw: it's a single AI voice making a single judgment with no cost to being wrong.
Even the newer "multi-agent" tools like GitHub Copilot's parallel review agents don't really solve this. They divide labor: one agent for security, one for linting, one for testing. But when two of those agents disagree about how serious an issue is, nothing resolves it. The reviewer's job just gets pushed onto the developer.
That's the gap I built ShiftLeft Society to fill, for the Qwen Cloud Global AI Hackathon 2026.
The idea: two AI reviewers who actually negotiate with each other when they disagree, with a cost structure that keeps the whole thing auditable.
The result: a live, deployed multi-agent tribunal that scored 95% on a 40-case benchmark, versus 82.5% for a single-agent baseline. A 12.5-point improvement, primarily by reducing false positives on safe code.
Here's how I built it and what I learned.
Most "multi-agent negotiation" tutorials I found online let the LLM decide everything, including the math. They ask the model to self-report a confidence number (0-100), then multiply it against something, and out pops a "verdict." That produces different outputs on every run. Unauditable. Uncalibrated. Basically vibes.
I inverted that:
DEFEND
, PARTIAL
, or CONCEDE
.Here's the actual cost table:
| Position | Cost |
|---|---|
| DEFEND (dig in on your severity) | gap_tiers × 30 |
| PARTIAL (compromise halfway) | 15 |
| CONCEDE (yield fully) | 0 |
Each agent starts a negotiation with a 100-point budget. If Security wants to DEFEND against Performance and they're 2 severity tiers apart (say, Security = CRITICAL vs Performance = MEDIUM), DEFEND costs 60 points. If they're 3 tiers apart, DEFEND costs 90. Meaning: the stronger your disagreement, the more it costs you to hold your ground. That forces the LLM to actually think about whether it wants to spend the budget.
The LLM never touches a number. It just picks a category. The math is Python.
Three properties this gives you that pure-LLM agent demos rarely have:
The negotiation isn't stateless. Every time the tribunal produces a verdict, each agent's Round 2 judgment gets scored: did it defend correctly, or did it correctly defer to a peer flagging a more serious issue?
That outcome is:
The next time that agent negotiates, on a completely different PR days later, it starts from 100 + track_record_adjustment
, not a fresh 100 every time.
effective_budget = 100 + clamp(-15, +15, round((smoothed_win_rate - 0.5) * 30))
smoothed_win_rate = (wins + 2.5) / (total_negotiations + 5)
This turns the tribunal from a system that negotiates well once into a system that gets better at knowing which of its own voices to trust over time, without any human manually re-weighting anything. It shows up right in the PR comment:
"track record: 95% upheld over 42 past negotiations → budget +13"
dashscope-intl.aliyuncs.com/compatible-mode/v1
) powers all agent reasoning.scan_vulnerabilities
, detect_secrets
, check_yaml_pinning
, analyze_complexity
. If the MCP server is unreachable, everything falls back to local regex-based detection so the tribunal degrades gracefully.This is a hackathon post so I want to be honest about the developer experience, both the good and the friction.
What worked well:
The OpenAI-compatible endpoint (compatible-mode/v1
) is a genuine developer experience win. My tribunal uses LangChain's ChatOpenAI
wrapper unchanged, just pointed at Alibaba's Base URL:
llm = ChatOpenAI(
model="qwen-max",
api_key=os.getenv("QWEN_API_KEY"),
base_url="https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
temperature=0.1,
max_tokens=4096,
)
That's it. No custom client, no bespoke JSON handling. Everything from LangGraph to structured output parsing just worked.
Qwen-Max's structured output is reliable enough for production. I use LangChain's .with_structured_output(Pydantic model)
on almost every agent call, and Qwen-Max produces valid JSON on ~99% of calls. The rare failures are caught by a 4-layer fallback chain (structured output → raw parse → regex scrape → deterministic inference), which means the tribunal never crashes on a malformed model response.
Where I hit friction:
Model catalog navigation was harder than expected. Alibaba's catalog lists several Qwen models, but not all are accessible on every API tier. I tested qwen3.7-max
(which is a reasoning model) and found it works but adds ~5x latency because of the reasoning tokens. For a system that needs to respond within GitHub's 10-second webhook timeout, that's disqualifying. So I use qwen-max
(the stable current alias) which is fast enough and just as accurate for my use case.
Real production engineering decision, not a compromise: latency budget matters more than "newest model" for webhook-driven agents.
I ran a 40-case benchmark: 22 vulnerable, 18 safe code samples covering categories from SQL injection to unpinned dependencies to secure random tokens. Same model, same prompts for both systems. The only difference: single-agent verdict vs multi-agent tribunal.
| System | Correct verdicts | Notes |
|---|---|---|
| Single-agent baseline | 33 / 40 (82.5%) | Mostly false positives, flagged safe code as vulnerable |
| Multi-agent tribunal | 38 / 40 (95.0%) | |
| One miss on an insecure-random case |
The interesting story is where the tribunal improved. On six cases, the single agent wrongly flagged safe code as dangerous:
On all six, the negotiation cleared the false positive. Security's Round 1 caution got tempered by Performance's correctly identifying "no actual issue here." That's the tribunal's biggest practical win: less noise, not just more accuracy.
Results committed to the repo as benchmark_results.json
.
1. Non-determinism in LLM-driven negotiation is a real production problem. Letting the model self-report confidence produces unreproducible verdicts. Constraining the LLM to categorical choices and computing consequences in code fixed this cleanly.
2. Blocking calls silently break async systems. Early builds had requests.get
for the GitHub diff, unwrapped sqlite3.execute
in a FastAPI endpoint, and asyncio.create_task
without holding references. Every one of those is a latent bug that only shows under real load. Wrapping in asyncio.to_thread
and holding tasks in a module-level set turned "works on my laptop once" into "survives real webhook traffic."
3. Error handling has to be visible, not just present. I built 4-layer fallbacks for the mediator, MCP tool calls, and structured output parsing. But that engineering is invisible unless you tell people. Documenting them in the README is how a judge sees the depth.
4. Persistence matters more than you think. My credibility trust data got wiped twice by container restarts before I put SQLite on a Docker volume. Non-persistent state is invisible until it disappears.
1. Ground the credibility signal in real merge outcomes.
Right now the credibility metric asks: "did this agent's severity match the final verdict?" That's a proxy. The real signal is what happened to the PR after the tribunal reviewed it. Did the developer merge it as-is, rewrite the flagged code, or ignore the review entirely? That's not a bigger benchmark. It's a fundamentally different feedback loop. The system stops guessing whether it was correct and starts learning from what humans actually did with its verdicts.
2. Multi-language through the MCP layer, not just prompting harder.
The obvious "extend to JavaScript" answer is to prompt the LLM more aggressively. That works okay. What's more interesting: add language-specific detectors (ESLint for JS, go vet
for Go, cargo-audit
for Rust) as MCP tools. The deterministic layer grows, the LLM guesses less, and the whole thing gets more auditable as it gets more general.
3. A third specialist agent to stress-test the negotiation mechanic.
With only two agents, negotiation is 1-v-1. Adding a Maintainability voice tests whether confidence-budget generalizes cleanly to N-way disputes, or whether it needs redesigning at higher agent counts.
4. Team-level credibility, not just global agent-level.
Different teams have different risk tolerances. A payments team should weight security's judgment more heavily. A hackathon team can tolerate more noise. Track credibility per-repo or per-team, and the system tunes itself to each team's actual priorities over time.
The whole system is open source. Comments and PRs welcome.