{"slug": "when-ai-reviewers-disagree-building-a-multi-agent-devsecops-tribunal-with-qwen", "title": "When AI Reviewers Disagree: Building a Multi-Agent DevSecOps Tribunal with Qwen-Max", "summary": "A developer built ShiftLeft Society, a multi-agent DevSecOps tribunal using Qwen-Max, for the Qwen Cloud Global AI Hackathon 2026. The system features two AI reviewers that negotiate disagreements with a cost-based mechanism, achieving 95% accuracy on a 40-case benchmark versus 82.5% for a single-agent baseline. The project uses an OpenAI-compatible endpoint for seamless integration with LangChain.", "body_md": "I've been building AI code review tools since 2024. Every one I've tried, and every one on the market, has the same structural flaw: it's a single AI voice making a single judgment with no cost to being wrong.\n\nEven the newer \"multi-agent\" tools like GitHub Copilot's parallel review agents don't really solve this. They divide *labor*: one agent for security, one for linting, one for testing. But when two of those agents disagree about how serious an issue is, nothing resolves it. The reviewer's job just gets pushed onto the developer.\n\nThat's the gap I built ShiftLeft Society to fill, for the [Qwen Cloud Global AI Hackathon 2026](https://qwencloud-hackathon.devpost.com/).\n\n**The idea:** two AI reviewers who actually negotiate with each other when they disagree, with a cost structure that keeps the whole thing auditable.\n\n**The result:** a live, deployed multi-agent tribunal that scored 95% on a 40-case benchmark, versus 82.5% for a single-agent baseline. A 12.5-point improvement, primarily by reducing false positives on safe code.\n\nHere's how I built it and what I learned.\n\nMost \"multi-agent negotiation\" tutorials I found online let the LLM decide everything, including the math. They ask the model to self-report a confidence number (0-100), then multiply it against something, and out pops a \"verdict.\" That produces different outputs on every run. Unauditable. Uncalibrated. Basically vibes.\n\nI inverted that:\n\n`DEFEND`\n\n, `PARTIAL`\n\n, or `CONCEDE`\n\n.Here's the actual cost table:\n\n| Position | Cost |\n|---|---|\n| DEFEND (dig in on your severity) | `gap_tiers × 30` |\n| PARTIAL (compromise halfway) | 15 |\n| CONCEDE (yield fully) | 0 |\n\nEach agent starts a negotiation with a 100-point budget. If Security wants to DEFEND against Performance and they're 2 severity tiers apart (say, Security = CRITICAL vs Performance = MEDIUM), DEFEND costs 60 points. If they're 3 tiers apart, DEFEND costs 90. Meaning: **the stronger your disagreement, the more it costs you to hold your ground.** That forces the LLM to actually think about whether it wants to spend the budget.\n\nThe LLM never touches a number. It just picks a category. The math is Python.\n\nThree properties this gives you that pure-LLM agent demos rarely have:\n\nThe negotiation isn't stateless. Every time the tribunal produces a verdict, each agent's Round 2 judgment gets scored: did it defend correctly, or did it correctly defer to a peer flagging a more serious issue?\n\nThat outcome is:\n\nThe next time that agent negotiates, on a completely different PR days later, it starts from `100 + track_record_adjustment`\n\n, not a fresh 100 every time.\n\n```\neffective_budget = 100 + clamp(-15, +15, round((smoothed_win_rate - 0.5) * 30))\nsmoothed_win_rate = (wins + 2.5) / (total_negotiations + 5)\n```\n\nThis turns the tribunal from a system that negotiates well **once** into a system that gets better at knowing which of its own voices to trust **over time**, without any human manually re-weighting anything. It shows up right in the PR comment:\n\n\"track record: 95% upheld over 42 past negotiations → budget +13\"\n\n`dashscope-intl.aliyuncs.com/compatible-mode/v1`\n\n) powers all agent reasoning.`scan_vulnerabilities`\n\n, `detect_secrets`\n\n, `check_yaml_pinning`\n\n, `analyze_complexity`\n\n. If the MCP server is unreachable, everything falls back to local regex-based detection so the tribunal degrades gracefully.This is a hackathon post so I want to be honest about the developer experience, both the good and the friction.\n\n**What worked well:**\n\nThe OpenAI-compatible endpoint (`compatible-mode/v1`\n\n) is a genuine developer experience win. My tribunal uses LangChain's `ChatOpenAI`\n\nwrapper unchanged, just pointed at Alibaba's Base URL:\n\n```\nllm = ChatOpenAI(\n    model=\"qwen-max\",\n    api_key=os.getenv(\"QWEN_API_KEY\"),\n    base_url=\"https://dashscope-intl.aliyuncs.com/compatible-mode/v1\",\n    temperature=0.1,\n    max_tokens=4096,\n)\n```\n\nThat's it. No custom client, no bespoke JSON handling. Everything from LangGraph to structured output parsing just worked.\n\nQwen-Max's structured output is reliable enough for production. I use LangChain's `.with_structured_output(Pydantic model)`\n\non almost every agent call, and Qwen-Max produces valid JSON on ~99% of calls. The rare failures are caught by a 4-layer fallback chain (structured output → raw parse → regex scrape → deterministic inference), which means the tribunal never crashes on a malformed model response.\n\n**Where I hit friction:**\n\nModel catalog navigation was harder than expected. Alibaba's catalog lists several Qwen models, but not all are accessible on every API tier. I tested `qwen3.7-max`\n\n(which is a reasoning model) and found it works but adds ~5x latency because of the reasoning tokens. For a system that needs to respond within GitHub's 10-second webhook timeout, that's disqualifying. So I use `qwen-max`\n\n(the stable current alias) which is fast enough and just as accurate for my use case.\n\nReal production engineering decision, not a compromise: **latency budget matters more than \"newest model\" for webhook-driven agents.**\n\nI ran a 40-case benchmark: 22 vulnerable, 18 safe code samples covering categories from SQL injection to unpinned dependencies to secure random tokens. Same model, same prompts for both systems. The only difference: single-agent verdict vs multi-agent tribunal.\n\n| System | Correct verdicts | Notes |\n|---|---|---|\n| Single-agent baseline | 33 / 40 (82.5%) | Mostly false positives, flagged safe code as vulnerable |\n| Multi-agent tribunal | 38 / 40 (95.0%) |\nOne miss on an insecure-random case |\n\nThe interesting story is *where* the tribunal improved. On six cases, the single agent wrongly flagged **safe code** as dangerous:\n\nOn all six, the negotiation cleared the false positive. Security's Round 1 caution got tempered by Performance's correctly identifying \"no actual issue here.\" That's the tribunal's biggest practical win: **less noise, not just more accuracy.**\n\nResults committed to the repo as `benchmark_results.json`\n\n.\n\n**1. Non-determinism in LLM-driven negotiation is a real production problem.** Letting the model self-report confidence produces unreproducible verdicts. Constraining the LLM to categorical choices and computing consequences in code fixed this cleanly.\n\n**2. Blocking calls silently break async systems.** Early builds had `requests.get`\n\nfor the GitHub diff, unwrapped `sqlite3.execute`\n\nin a FastAPI endpoint, and `asyncio.create_task`\n\nwithout holding references. Every one of those is a latent bug that only shows under real load. Wrapping in `asyncio.to_thread`\n\nand holding tasks in a module-level set turned \"works on my laptop once\" into \"survives real webhook traffic.\"\n\n**3. Error handling has to be visible, not just present.** I built 4-layer fallbacks for the mediator, MCP tool calls, and structured output parsing. But that engineering is invisible unless you tell people. Documenting them in the README is how a judge sees the depth.\n\n**4. Persistence matters more than you think.** My credibility trust data got wiped twice by container restarts before I put SQLite on a Docker volume. Non-persistent state is invisible until it disappears.\n\n**1. Ground the credibility signal in real merge outcomes.**\n\nRight now the credibility metric asks: \"did this agent's severity match the final verdict?\" That's a proxy. The real signal is what happened to the PR after the tribunal reviewed it. Did the developer merge it as-is, rewrite the flagged code, or ignore the review entirely? That's not a bigger benchmark. It's a fundamentally different feedback loop. The system stops guessing whether it was correct and starts learning from what humans actually did with its verdicts.\n\n**2. Multi-language through the MCP layer, not just prompting harder.**\n\nThe obvious \"extend to JavaScript\" answer is to prompt the LLM more aggressively. That works okay. What's more interesting: add language-specific detectors (ESLint for JS, `go vet`\n\nfor Go, `cargo-audit`\n\nfor Rust) as MCP tools. The deterministic layer grows, the LLM guesses less, and the whole thing gets more auditable as it gets more general.\n\n**3. A third specialist agent to stress-test the negotiation mechanic.**\n\nWith only two agents, negotiation is 1-v-1. Adding a Maintainability voice tests whether confidence-budget generalizes cleanly to N-way disputes, or whether it needs redesigning at higher agent counts.\n\n**4. Team-level credibility, not just global agent-level.**\n\nDifferent teams have different risk tolerances. A payments team should weight security's judgment more heavily. A hackathon team can tolerate more noise. Track credibility per-repo or per-team, and the system tunes itself to each team's actual priorities over time.\n\nThe whole system is open source. Comments and PRs welcome.", "url": "https://wpnews.pro/news/when-ai-reviewers-disagree-building-a-multi-agent-devsecops-tribunal-with-qwen", "canonical_source": "https://dev.to/yoges_mohan_511bda5afbe7d/when-ai-reviewers-disagree-building-a-multi-agent-devsecops-tribunal-with-qwen-max-bj3", "published_at": "2026-07-11 02:22:47+00:00", "updated_at": "2026-07-11 02:41:29.712433+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "developer-tools", "ai-safety"], "entities": ["ShiftLeft Society", "Qwen Cloud Global AI Hackathon 2026", "Qwen-Max", "LangChain", "GitHub Copilot", "Alibaba", "DashScope"], "alternates": {"html": "https://wpnews.pro/news/when-ai-reviewers-disagree-building-a-multi-agent-devsecops-tribunal-with-qwen", "markdown": "https://wpnews.pro/news/when-ai-reviewers-disagree-building-a-multi-agent-devsecops-tribunal-with-qwen.md", "text": "https://wpnews.pro/news/when-ai-reviewers-disagree-building-a-multi-agent-devsecops-tribunal-with-qwen.txt", "jsonld": "https://wpnews.pro/news/when-ai-reviewers-disagree-building-a-multi-agent-devsecops-tribunal-with-qwen.jsonld"}}