{"slug": "why-a-multi-agent-system-must-not-average", "title": "Why a multi-agent system must not average", "summary": "A developer described rebuilding a multi-agent analysis system by removing all voting and weighted averaging from the codebase, arguing that averaging heterogeneous judges produces a \"fabricated consensus\" no participant would defend. The redesigned architecture gives each persona the same canonical record with no shared drafts, escalates disagreements to a citation-based challenge stage, and allows runs to end unresolved or with no conclusion, published as-is.", "body_md": "We shipped four analysis personas, then deleted every vote and every weighted mean\n\nfrom the codebase.\n\nLook at how most multi-agent projects are built. Three roles read the same material, each\n\nwrites a conclusion, an orchestrator merges them into one answer.\n\nThe merge is usually a weighted mean, or a majority vote.\n\nIt seems obviously right — isn't that what ensembles do? Random forests average, voting\n\nclassifiers vote, bagging reduces variance. Same word, so the same move should apply.\n\nThe analogy is wrong, and the cost is not small.\n\nAn ensemble averages **homogeneous weak learners** with independent noise on a shared\n\nobjective. Averaging cancels the noise.\n\nA multi-agent team is **heterogeneous judges** whose value is precisely that they disagree:\n\none reads the shape of a distribution, one reads whether consensus has drifted, one exists\n\nto find the dissent nobody else produces. Average those three and you do not get a steadier\n\nanswer. You get a sentence sanded down until it has no position.\n\nThis is the part worth its own section.\n\n**When the members agree, averaging is the identity operation. Nothing happens, nothing is lost.**\n\n**When the members disagree, averaging is at full strength** — because at that moment the\n\nonly informative fact in the run is *that two informed readers disagreed*.\n\nThe operation activates precisely in the case where you most need the disagreement kept.\n\nTwo members read the same record:\n\nThe average is medium-width and centred. That shape was **proposed by nobody and would be defended by nobody**. You manufactured a conclusion no participant believes, and destroyed\n\nI call that a fabricated consensus. It is not even a compromise — a compromise at least\n\nadmits the disagreement happened.\n\nThe second trap is subtler: shared context.\n\nAs the system runs, the members' outputs start to look alike, and they converge nicely.\n\nIt reads like maturity. It is B having seen A's draft.\n\nThe moment B reads A's conclusion before reading the material, B's output stops being an\n\nindependent estimate and becomes a **revision** of A's. Two correlated estimates look like\n\nagreement and carry the information of one.\n\n**You paid for four members and received one.**\n\nSo each member gets the same canonical record and nothing else. No shared draft, no rolling\n\nsummary, no \"here is what the others think\".\n\nThe cost is real: independent members produce messier, more contradictory output. That mess\n\n**is** the signal. It is what you paid for.\n\nDisagreement cannot be averaged, so how does a run end?\n\nIt escalates to a **challenge** stage with one rule:\n\nA challenge is a citation contest, not a vote.\n\nA challenged claim has two exits:\n\nThere is no third path. The withdrawal must be recorded, because the withdrawal is the\n\nactual output of that round.\n\nIf both sides stand, neither is decided. Both are marked `unresolved`, and that marking\n\ntravels into the published result. An honest \"we could not settle this\" beats a fabricated\n\nmidpoint.\n\nThe distinction matters: a vote asks **who has more support**. A challenge asks **who can point at their evidence**. The first rewards confident phrasing. The second rewards having\n\nThe final stage reads the whole transcript and **can change the conclusion**, not merely\n\ncomment on it.\n\nReviewers without authority produce advisory notes, and advisory notes get ignored under\n\ndeadline. That is failure mode one.\n\nGrant authority instead and failure mode two arrives: reviewers with authority **kill good conclusions**, especially when their check is \"this feels off\".\n\nSo the reviewer's checks are fixed: **five audits, fixed order, each producing a diff against a named sentence.** No scores, no general impressions, no \"consider reviewing this\n\nOne more rule that feels wrong until you need it: **a run is allowed to end with no conclusion**, and that outcome is published as-is. The alternative — shipping a conclusion\n\nThe table below is the useful part of our architecture doc. The diagram is not.\n\n| Failure | Symptom | Cause and fix | \n|---|---|---|\n| Leaky independence | Members converge suspiciously often | Context leaked through the **runtime** , not the prompts | \n| Reviewer rubber-stamp | Zero overturns over many runs | Reviewer read the draft **after** it was formatted for output | \n| Persona collapse | Two \"different\" members write near-identical text | Both priors were restatements of \"be careful, consider many factors\" | \n| Confidence inflation | High confidence on single-source claims | Confidence was **member-reported** , never calibrated against the citation audit | \n| Challenge theatre | Disagreements resolve in one turn, always | The challenger had no evidence standard, so it always yielded first | \n\nFour of the five were prompt problems, not code problems. Only the first was structural.\n\nThat ratio is worth remembering when you debug your own team.\n\nWe run this in production. The public repository is **source-available, not open source**.\n\nPublic: the four personas, the orchestration schema, the client, and the reasoning above.\n\nNot public: the training prompts, the evaluation weights, the live data adapters, the\n\nsettlement logic.\n\nWe say this in the first line rather than letting you install it and discover it. A\n\nrepository that looks complete and is not damages a project more than the missing code ever\n\nwould.\n\nIt is not \"use more models\", and not \"add a reviewer\".\n\nIt is this: **when two informed judges disagree, the disagreement is the asset. Do not wash it out with a mean.**\n\nAny operation that smooths a disagreement deserves one question first — am I removing noise,\n\nor destroying evidence?\n\n[github.com/SoccerDreamAI/soccerdream-experts](https://github.com/SoccerDreamAI/soccerdream-experts)\n\n— the four personas, the orchestration schema, and the architecture rationale behind the\n\ntable above, in English and Chinese. Source-available: read it, fork it, and keep the\n\nattribution.\n\n*This is an analysis framework. It does not predict outcomes, is not advice, and is not intended for wagering.*", "url": "https://wpnews.pro/news/why-a-multi-agent-system-must-not-average", "canonical_source": "https://dev.to/soccerdreamai/why-a-multi-agent-system-must-not-average-3p5d", "published_at": "2026-11-01 12:17:10+00:00", "updated_at": "2026-11-01 12:22:21.271426+00:00", "lang": "en", "topics": ["ai-agents", "artificial-intelligence", "large-language-models"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/why-a-multi-agent-system-must-not-average", "markdown": "https://wpnews.pro/news/why-a-multi-agent-system-must-not-average.md", "text": "https://wpnews.pro/news/why-a-multi-agent-system-must-not-average.txt", "jsonld": "https://wpnews.pro/news/why-a-multi-agent-system-must-not-average.jsonld"}}