I Ran Five Diverse AI Agents Against Five Clones for 14 Nights. A Number I Made Up Decided the Result.

An experiment pitting five diverse AI agents against five identical clones of the same model (Opus 5) over 14 nights produced a pooled Brier score of 0.0225 for the diverse agents versus 0.0275 for the clones, with the diverse agents winning 9 of 14 nights, but the author declined to publish the result as a win because a self-written sentence decided the outcome. The pre-registered test, run at equal budget on a virality-forecasting task scoring 30 fresh posts daily from Hacker News, Reddit and X, was designed to test the claim that "a network of agents compounds context, not compute." The author states the result was a fourth ending never written down, and that nulls get published.

The scoreboard I opened on the morning of 5 September said my thesis had won. Five AI agents with genuinely different lenses had beaten five identical clones on 9 of 14 nights, and their pooled Brier score was 0.0225 against the clones' 0.0275. Lower is better. I had wanted that table since the first night. I did not publish it. The reason is this post. Most of my last few months went into agents that work together: several of them, with different jobs, handing work to each other, with rules about what each one is allowed to see. At some point I stopped and asked the only question that matters there. When you add the second agent, what did you actually add? If it is the same model reading the same context, you added compute. More tokens, more latency, a second opinion drawn from the same well. If it knows different things and looks through a different lens, you added something else, and that something else is the whole case for building a network instead of buying more of one model. So I wrote the claim down before I could soften it. A network of agents compounds context, not compute. Agents with deliberately different roles, contexts and information diets should beat the same number of clones of the same model, at equal budget, on a task with real answers. If they tie, the "more agents means more intelligence" story is wrong, and I would rather learn that from my own data than from a bill. Fourteen nights later I have an answer. It is a fourth ending I never wrote down, and the thing that decided it was a sentence I wrote myself. Sold hard, measured rarely Look at how multi-agent gets sold right now. Panels, councils, swarms, debate rounds, a "team of specialists". The pitch is always that the group knows more than any member. Underneath most of those products sits one model wearing several system prompts, running at several times the cost of one call. I have been tracking who is actually winning in that market since February https://thoughts.jock.pl/p/ai-agent-landscape-feb-2026-data , and the panel pitch is in nearly every listing. I could not find anyone who had put the two arms side by side at equal budget, on a task with a scoreboard, and published the number including the boring one. It is an awkward experiment to run if you are selling the thing. So I built the scoreboard instead of arguing about it. The rules, written before anything ran These went into the plan first, and they are the only reason the rest of this post is worth reading: - Two arms, one variable. Five clones against five diverse agents. Same model Opus 5 , same budget, same tools, same task, same morning. The only thing that changes is what each agent knows and how it is told to look. - Reality grades it. No model scoring another model's work, because a language model judging language models is the same well again. Deterministic Python, public APIs, zero human judgment in the loop. - Pre-registered. Scoring rule, pass threshold and the three possible endings written down before the first sample. A failed check is a finding, never a reason to edit the rule. - The control is built strong. The clone pack is a full theory of why things spread, applied from first principles. A rigged control proves nothing and I would know I rigged it. - Every experiment pays either way. A null result still has to leave me with a working tool. - Nulls get published. Including the ending where my thesis loses. The task: forecast virality. Every morning at 07:10 the machinery samples 30 fresh posts from Hacker News, Reddit and X, minutes old and not yet hot. All ten agents give each post a probability of crossing a set popularity bar within 48 hours, plus their top five picks. Two days later a script checks what happened and scores everyone: Brier on every probability squared error, so lower is better , precision on the top five. I picked it because the feedback loop is brutal and cheap. The answer arrives in 48 hours, verification costs one API call, and 30 predictions per agent per day means statistical power in two weeks rather than two years. The agents also cannot argue with the result, which is the part I like most. The diverse five: a Hacker News native who knows ranking decay and flag risk, a Reddit native who prices silent mod removals before appeal, an X native who cares about follower graphs and link penalties, a trend historian who ignores craft and asks whether the topic has been running hot for 30 days, and one agent, cold-read, kept deliberately blind to anything recent so it can only judge the content in front of it. Everything below runs in the open. Every chart and every number is on the live experiment page https://wiz.jock.pl/experiments/diversity-vs-clones/ , all fourteen nights. Finding one: prompt diversity is cosplay until you make it mechanical Night one nearly fooled me, and it fooled me through the rationales. I read those before I read the correlations. They are good. The Hacker News lens talks about gravity and second-chance pools. The Reddit lens prices removal risk. The X lens shrugs at quality and asks who the author is. Five vocabularies, five kinds of evidence, five conclusions in prose. Then all five wrote down 0.02. Mean pairwise correlation inside the clone arm: 0.959. Inside the "diverse" arm: 0.909. On 26 of 30 posts the five different agents landed within a spread of 0.05, several of them on the exact same number. A metaphysics thread, a Rust tooling release and a Pygame tutorial got the same shrug from five agents that supposedly evaluate completely different things. I had built a panel of experts and got one opinion with five accents . The only reason I know this is a line in the plan called the manipulation check: agent-to-agent correlation has to be lower in the diverse arm than in the clone arm, or the packs are a placebo and get rewritten before any result counts. Without that line, night one reads as a clean success. Ten of ten submissions valid, digest generated, daily loop running, every green light I had built for myself lit. I would have run two weeks, got a difference between the arms, and published it as evidence for diversity, when what I had measured was noise between ten near-copies. I have been on the wrong side of this before. When I ran 255 agents to make three comics https://thoughts.jock.pl/p/ai-slop-comic-experiment-2026 , fan-out scaled logistics and left quality where it was. When I told my agent to build an app every day https://thoughts.jock.pl/p/directed-ai-experiments-vibe-business , the output was competent and painfully boring for weeks before I admitted it. The first time I let four agents loose on a build https://thoughts.jock.pl/p/opus-4-6-agent-experiment-2026 , I judged the whole thing by reading what came out. All three times I noticed late. This time the trap had a tripwire in it and it went off in four hours. That is the entire argument for pre-registration, and it cost me one line in a plan I had already written. There is research under this. A Cornell group published Correlated Errors in Large Language Models https://arxiv.org/abs/2506.07962 in June 2025, measured across more than 350 models. On one leaderboard dataset, two models agreed with each other 60% of the time when both were wrong, and larger, more accurate models had more correlated errors, across architectures and providers. If two frontier models from rival labs still fail together most of the time, five prompts wrapped around one model were never going to buy me independence. I was asking wording to do a job wording cannot do. The easy repair is to tell the agents to disagree. Wider spread, zero evidence, because it manufactures exactly the result the experiment exists to test. There is a rule in the packs now that says so in writing. What I did instead was make each lens mechanical. Version one gave each agent a personality and a paragraph about what it cares about, then asked for a probability. Version two is a two-step move: the agent first reaches a verdict in its own categories and its own failure modes, then reads a probability off the range that verdict is assigned. The number comes out of the lens instead of out of the shared instinct with the lens narrating over the top. Same metaphysics post, through the rebuilt lenses: - hn-native : was 0.03, now 0.00 to 0.02. Metaphysics bait, flag or kill. - reddit-native : was 0.03, now 0.14 to 0.22. Easy-comment title travels. - x-native : was 0.02, now 0.07 to 0.11. Away platform, payoff withheld. - trend-historian : was 0.02, now 0.07 to 0.17. Cold topic, honest width. - cold-read : was 0.02, now 0.01 to 0.03. Uncheckable central claim, capped. Everyone within 0.02 of each other becomes a span from 0.00 to 0.22, and every step of it is principled. Nobody was told to fight. They stopped deferring to the same generic gut feeling. The next morning the clone arm sat at 0.918 and the rebuilt diverse arm at 0.619. Herded posts fell from 26 of 30 to 21. And it held. The check passed on all fourteen nights . The correlation gap between the arms averaged 0.294, and it was wider in the second week than the first, 0.329 against 0.260, so the lenses kept separating instead of drifting back into the herd once the novelty wore off. Across the run the clones sit around 0.90 to 0.97 internal correlation, one brain in five copies, and the lenses run 0.53 to 0.75, with one strange night 24 August where both arms spread out at once. My opinion after fourteen nights: diversity has to be built, and a prompt cannot supply it. If a panel's members share a model and a context and differ only in the paragraph that tells them who they are, they are one forecaster with a thesaurus. Independence comes from different procedures, different information and, I now think, different providers. I split my own workflow across providers back in January https://thoughts.jock.pl/p/multi-model-ai-workflow-2026-gpt-claude-gemini and it looked like a preference at the time. It looks like the only real lever now. Finding two: a number I guessed decided the scoreboard Night one had a second fix, and at the time it felt like the most grown-up thing I did all evening. The shared hygiene text every agent gets said the base rate of a post going hot was "single digit percent", which is where that uniform floor of 0.01 to 0.03 came from. I replaced it with what I called the honest number: 10 to 15%, or 3 to 5 hot posts in a 30-post sample. I had accidentally coached the panel into the herd, I thought, and now I had un-coached it. Fourteen nights, 416 scored posts per agent, and three of them crossed the bar. Three. The realized rate was 0.72% . I had spent an evening carefully coaching ten agents to expect fifteen times that, and they obliged: both arms forecast around 11% on average for the whole fortnight, because packs do what they are told. The "single digit percent" I overwrote sat closer to reality than my correction. And the block went into the clone pack too it is shared, byte for byte, across all six packs , so on the same night I wrote "a control you keep improving stops being a control" into my plan, I made the control's calibration worse. Both arms were miscoached by the same amount, so the comparison between them stayed fair. What moved was the level, for everyone. It got worse as the data came in. At four nights the realized rate was 1.72%. At seven, 0.97%. At fourteen, 0.72%. My coached figure stood still the whole time, so the gap widened every week, and the share of each arm's score explained by that one number went 60%, then 71%, then 74% for the clones, and 49%, 62%, 68% for the lenses. I assumed more data would average it out. More data made it bigger. Here is why that swamps everything. Brier is squared error on a probability. Answer 0.11 on a post that stays cold and you pay 0.11 squared for that slot. Answer near the rate that occurred and you pay almost nothing. Now do that on nearly every one of 416 slots, because on this task nearly every post stays cold. The level of your answers becomes most of your score, and how well you read the posts is a small term sitting on top. So I rescaled each arm's forecasts by one multiplier, down to the rate that occurred. That keeps each arm's own ordering of the posts and fixes only the level: - Clones : mean forecast 0.1110 15.4x reality , raw Brier 0.027498, rescaled 0.007092. Cost of the level error: 0.020406, which is 74% of their score. - Lenses : mean forecast 0.1084 15.0x reality , raw Brier 0.022451, rescaled 0.007126. Cost of the level error: 0.015325, 68% of their score. Raw gap: 0.005047 in favour of the lenses. After the correction: 0.000034 in favour of the clones. The direction flips and the size collapses by a factor of about 150. The lead I nearly published was a level effect. The diverse arm answered slightly lower on average, and in a fortnight where almost nothing went hot, answering lower is most of what the score rewards. Nothing in that lead says it read the posts better. The cost of my one sentence, 0.020 of Brier, is about four times the entire difference between the arms it was written to help measure. I had one variable and I was careful about it. Same model, same budget, same sample, same scorer, the control's lens never touched. Then I handed both arms one number that was fifteen times wrong and it outweighed my careful variable four times over. Rigour about the thing you are varying does nothing to protect you from the thing you are holding constant. I did not find this on night one, and the reason is the useful part. A base rate is something you learn by counting. I estimated it, wrote "honest" next to the estimate, and moved on. Anyone comparing two panels of language models is exposed to this exact shape. When both systems share a prior, that prior's error sits inside both scores, and if it is bigger than the difference you are looking for, your comparison measures which system happened to sit lower on the shared mistake. It will look exactly like a result. Mine had a 9 of 14 and a four-decimal margin. Finding three: a constant that never read a post tied both panels Then I ran the chair. Take a single number equal to the rate that occurred and apply it to all 416 posts without reading one. No model, no prompt. I called it the base-rate chair because it sits at the table and never speaks. It scores 0.007160. Rescaled clones: 0.007092, ahead of the chair by 0.000068. Rescaled lenses: 0.007126, behind it by 0.000034. The pre-registered bar for calling something skill was clearing the chair by 0.0005. Both arms land inside a fifth of that bar. Fourteen nights, 4,130 pooled predictions, ten agents on six context packs, and once the level error comes out there is no evidence that either arm can forecast this task at all. All the reading and all the platform knowledge was worth about as much as knowing how rarely anything goes viral. I sat with that one for a while. It is still the most useful thing the experiment produced, and I think it generalises: an agent panel that has not been scored against a constant is a brochure. Panels may well work. Until you have the constant you do not know whether the panel's score is reading or level, and the vendor selling you the panel has no incentive to find out. Finding four: the starved agent won, and the specialists lost at home Inside the diverse arm, the best single forecaster over the run is cold-read, at 0.022850. That is the agent I gave nothing. No platform culture, no trend history, no idea what has been popular lately. Its pack forbids it from reasoning about recency at all. It beat the Hacker News native, the Reddit native, the X native and the trend historian, the four I fed the context that was supposed to make them better, across 416 predictions each. The home-platform numbers leave less room to argue. Each native was built to be strongest on its own turf, with full conviction at home and a humble base rate away. So I scored each one on posts from its home platform against the clones on those same posts: - hn-native on 179 Hacker News posts: 0.040184. Clones on the same posts: 0.035244. - reddit-native on 179 Reddit posts: 0.034736. Clones: 0.023791. - x-native on 58 X posts: 0.030793. Clones: 0.022366. Zero of three. Every specialist lost on the ground it was built for, to five generic copies with no platform notes at all. I think this is the most damaging row in the fortnight, because it cannot hide behind the level error: natives and clones were scored on the same posts with the same outcomes. Part of it is my design. Home conviction told the natives to commit harder at home, and on a task this sparse, committing harder is exactly what Brier punishes. But a specialist whose expertise makes it more confident and no more accurate is the precise thing I was worried about in the whole multi-agent pitch, and there it is in my own table, three times. The entire premise of a context pack is that information makes a forecaster better. On this data the packs added confidence without adding discrimination, and the agent with the least to be confident about paid the smallest penalty. I have no cleaner explanation and I am not going to invent one. Every figure in this section sits on the same coached rate, so read it as provisional until the ruler is fixed. Finding five: the one thing diversity bought One result splits the arms in a way that survived everything. In the diverse arm, the panel average beat its own best member: 0.022451 against cold-read's 0.022850. In the clone arm it did not. The panel scores 0.027498 while its best member, a-clone-3, scores 0.026837. Averaging five near-copies of one opinion made the answer slightly worse than the best copy. Averaging five different lenses made it better than any of them. Each arm is measured against itself there, so the level error is not choosing between them, and the direction differs between the arms. That is why I trust it more than anything else in the run. Averaging errors that move together cancels nothing. Averaging errors that move apart cancels some of them. That is the entire reason ensembles work, and the effect showed up in the arm I had decorrelated and stayed absent in the arm I had not. One caveat for the careful reader: the report also computes a "best" aggregate that picks each night's winning agent after seeing the outcomes. It beats everyone, because a hindsight oracle cannot lose, and it is excluded from every claim here. Two other aggregation methods were deferred by the spec and have no score yet. So diversity bought something measurable: a crowd worth pooling. It did not buy skill at the task, because on this task there was no skill to be had by anyone, including the constant. A real difference between the arms, and still smaller than the mistake I made writing the packs. Finding six: the two measurements my mistake cannot touch disagree Brier cares how high you answered. Rank does not. So the fairer question is whether an arm put the post that went hot near the top of its own list, whatever numbers it used. Three posts went hot in fourteen nights, all on Hacker News, which tells you most of what you need to know about how much this section can settle. - 22 August, a bookmarklet that copies a webpage into Figma: clones ranked it 6th of 30, lenses 9th. - 23 August, AI chip architectures: clones 3rd, lenses 7th. - 31 August, transferring files over an Ethernet patch cable: 3rd and 3rd. Mean rank 4.00 for the clones against 6.33 for the lenses. The clones ranked the hot post higher twice and tied once. The panel effect I went looking for, turning up in the wrong arm. There is a second level-blind measurement, and I promised it in advance, which is the only reason it is admissible. On night one, four posts out of thirty split my forecasters, and I wrote that if those turned out to be the ones they called correctly, disagreement is where the accuracy was hiding, and if not, I owned a very well dressed committee. The test: each night, rank all 30 posts by how much the agents inside an arm disagreed about them, widest spread first, and find where the hot post sits. If argument carries no information, the expected rank is 15.5. - 22 August, herded v1 packs: clones 7th of 30, lenses 13th. Nothing. The exact bet I made that night lost. - 23 August, rebuilt lenses: clones 3rd, lenses 2nd , with a spread 3.6 times the night's median. - 31 August, rebuilt lenses: clones 10th, lenses 1st , spread 3.8 times the median. Mean spread rank 6.67 for the clones, 5.33 for the lenses, against a uniform 15.5. On both nights the rebuilt lenses ran, the post that went hot was the second and then the first most argued-about post of the night. The clones show the effect too, so "argument marks the interesting post" belongs to the model, and what the lenses add is sharpness. Night one's 13 stays in the average because the packs were worse that night and I am not allowed to drop it. And it is three events. I am writing this paragraph slowly because it is the exact flattering ending the whole experiment was designed to stop me from writing, and that list holds the first numbers in two weeks that made me want to write it. So: two measurements my instrument error cannot touch. One says the clones ranked the hot post higher. The other says the lenses argued hardest about it. Each rests on three events, and they point in opposite directions. Three events can block a verdict. They cannot carry one. If they pointed the other way I would say the same sentence, and one paragraph up, I did. Three endings were pre-registered. Reality picked a fourth Before the first night I wrote down the three ways this could end, and I put my money on the second one, in writing: 1. Thesis holds. Diverse lenses beat clones at equal budget. Unsupported. The diverse arm never clears the chair, and the lead that would carry it reverses under rescaling. 2. Diversity has a price. Blinkered forecasters are worse forecasters, so the clones win. Unsupported too, and this is the one I bet on. The diverse arm is worse nowhere that survives the correction, and its pooled panel beats its own best member. 3. Prompt diversity is cosplay. True of the version one packs, and the check caught it on night one. Dead since: 14 of 14 passes, and the gap widened. What showed up instead was never on the list. The arms are genuinely different and equally unskilled, and my measurement error is bigger than the effect I was measuring. Everything the context packs do well stayed real and beside the point. I have gone hunting for problems in my own agent architecture before and found 85 of them https://thoughts.jock.pl/p/agent-architecture-human-review-2026 , and I counted that as a good day. This is better, because the verdict was decided by rules written by a version of me who did not yet know which answer he would like. What I refused to fix, and what I nearly fixed wrong One agent, a-clone-1, is missing the night of 28 August. Its thread errored at spawn and produced nothing, so it pools 13 runs and 386 predictions while everyone else pools 14 and 416, and the analysis weights each agent by its own count rather than pretending the grid is square. I could have re-run it afterwards, and the temptation was real, because a table with one hole looks like a mistake. By then I knew which posts went hot. A backfilled forecast is a forecast made by someone who knows the answer, and it does not matter that the agent would never be told. I would know, and I would be the one deciding whether the re-run looked right. The hole stays, and the report says so on its first page. A missing cell everyone can see is worth more than a full table nobody can trust. The base rate was the opposite problem: a fix I very nearly made wrong. The obvious repair is to recoach the rate and rerun. But the coached number did not live in one shared line. Across the six packs it appeared at 27 places, worded differently in each lens because each lens reasons about it differently. Editing only the shared line looks tidy and fair, one edit for everyone. What it actually does is re-anchor the clone pack completely the clones only have the shared line while leaving most of the diverse packs coached at the old rate. That would have moved one arm and part of the other, and handed me a clone win manufactured out of my own instrument error. That near-miss has the same shape as the original error. The original was a number I wrote once, felt rigorous about because I copied it byte for byte into six packs, and never checked against reality. The near-miss was an edit I would have made once, felt rigorous about because it touched the shared line, and never checked against what the packs actually said. Both times the feeling of fairness came from the action being uniform, and both times the action was uniform and its effect was not. I nearly repaired a measurement error with the exact habit that produced it. So the amendment went through a script that touches all 27 sites at once, prints per-arm accounting of every edit, and defaults to a dry run. It is applied now: 3 sites in the clone pack, 24 across the five lens packs, the coached rate down from 10 to 15% to 3 to 6%. That is still above the 0.72% the fortnight measured, on purpose. Three hot posts is too thin a base to pin a rate on, and I would rather be off by a handful than by fifteen. Nightly runs stay paused until I decide what the next round's primary metric is, because collecting a third week under a ruler I am about to change would only widen the gap between old data and new. What this does to the multi-agent pitch The honest summary: I cannot tell you whether diverse agents beat clones, and I can tell you exactly why not, which is a better position than most products in this market are in. A few things I now hold with some force. Decorrelation works, and it is a build problem. Verdict-first lenses kept five agents on one model genuinely apart for two weeks and made their average worth more than their best member. That is real, it is reproducible, and it is the part I would ship. Nobody gets it from a paragraph of personality. The shared prior is the score. On any sparse task, whatever number every agent starts from decides more of the result than anything the agents do afterwards. Count it before the first run. If you are evaluating someone else's panel, ask what base rate it was handed and what a constant at the true rate scores on the same data. If they cannot answer, the comparison has not been made yet. A specialist that is more confident and no more accurate is a liability. That is what my three natives were on their home turf. I would want to see a specialist beat the generalist on its own platform before I paid for it, and on my data none did. A panel's receipts have to include what the referee got wrong. "This panel called a real batch correctly at this rate" is a different sentence from "our AI panel analysed your idea", and only one of them survives being asked for evidence. After this fortnight I would add that the receipt has to include the instrument, or it is a brochure. Hunting for exactly this in other people's architectures is what the audit kit I put together https://wiz.jock.pl/store/agent-architecture-audit-kit is for, so it stung to find the biggest miss in my own. And the thing that keeps nagging me is how well this generalises off my machine. A room of five people with the same training, the same feeds and the same manager will give you five careful arguments for one shared prior, and everyone leaves feeling the decision was stress tested. I have sat in that meeting. I have probably run it. My agent test works on people too, and it is uncomfortably simple: ignore the reasoning, watch whether the answers move. I said that about opinions on night one. It turns out to be truer about priors, because an opinion at least gets argued with. A prior gets pasted into everyone's pack, for fairness, and nobody in the room ever reads it again. The next round gets a task where the interesting thing happens more than once a week, a base rate I have counted before a single agent reads it, and the same rule about publishing whichever ending shows up. Every measurement you trust has a number in it that somebody estimated once and never checked. Mine was 10 to 15 percent. I wrote it in a hygiene block, felt good about copying it into six packs, and it beat both my panels. If you want the next round of this with the failures left in, including the ending where the referee turns out to be the problem, subscribe https://thoughts.jock.pl/subscribe . That is the only version I publish.