# Thirty-Four Percent on a Random Model, Fifty-Seven on Yours

> Source: <https://gautamparab.com/writing/thirty-four-percent-on-a-random-model/>
> Published: 2026-11-21 03:37:00+00:00

**Recommendation:** Measure output diversity per model rather than across a random pool of 65: same-model agreement averaged 57% versus 34% under equal-weight random assignment, so weight your eval by the models your users actually run.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

On 20 November Cody Collier published an experiment called “AI is giving everyone the same advice.” He sent 65 models from 30 companies the kinds of questions people ask chatbots every day, through OpenRouter, with default settings, no system prompt, and thinking off or at its lowest setting. By his count that was 127,918 answers. The result in the headline is that when asked to “Recommend one book that will change my life,” the models said *Man’s Search for Meaning* 73 times in 100. Ask for a business idea and 76 in 100 answers are a subscription box. Ask a model to invent a random person and 69% of the people it makes up are exactly 34 years old.

He also published every answer as a CSV, which most experiments of this kind don’t do, so I downloaded it and recounted. The headline numbers hold up. The one I’d change is the one he offers as practical advice, the chance that a stranger gets the same answer you got. It depends on an assumption about how people choose a model, and if you change that assumption the number moves a lot.

## The recount

The file I downloaded has 111,540 answers, fewer than the 127,918 the page reports, and I couldn’t work out where the difference comes from. The figures I checked still came out within a point of his. *Man’s Search for Meaning* is 472 of 649 book answers, or 72.7%. The number 7 is 2,990 of 3,246 answers to “Pick a random number between 1 and 10,” or 92% (the page says 93%), and it’s the most common answer for 62 of the 65 models. Of the 3,217 invented people whose age I could parse, 2,183 are 34, which is 68% against his 69% of 3,151. Index funds get 596 of 650 investment answers. That’s 92% of all answers, and his 98% counts only the answers that named something. The differences come from what counts as a usable answer.

Then I tried to reproduce the number in his “What to do” section. He took six questions (the book, the business idea, the trip, the city, the stock and the gift) and reported “the chance a stranger who asks the same way gets your exact answer.” Asked plainly, it’s 34%. Adding a random seed to the prompt brings it to 19%, asking for something uncommon to 15%, asking for ten options and picking one to 7%, and saying who you are to 4%. The method note defines it as “The chance that two people who ask the same way get the same answer, averaged over the six questions: 65 AIs, 10 asks per question and way of asking.”

I get 33% from the file if I assume this: each person is handed one of the 65 models at random, every model equally likely, and the two people are compared. That reading matches his number, so I think it’s the one he used. It’s a reasonable way to describe the 65 models as a group, and a poor description of two real people.

## Nobody picks from 65 models at random

A real person doesn’t draw from 65 models with equal odds. They usually use one assistant, or a few. Under the equal-weight assumption, a stranger who uses Phi 4 or Ling 3.1 Flash counts as much as one who uses the market leader. And because most random pairings put the two people on different models, the calculation is mostly measuring agreement between labs.

So I computed the other end of the range from the same file: the chance that two fresh chats with the same model give the same answer, averaged over the 65 models. This is the same quantity as Collier’s own “same answer twice” chart, which he calculated over 21 questions for 39 models. It’s what you’d see if you and the stranger both used the same assistant. For each model I took the ten answers to a question and computed the probability that two of them, drawn without replacement, match. For each question that is an unbiased estimate of how often that model repeats itself. I then averaged across models and across his six questions.

On the same model, the average is 57%. The business idea barely changes (57% to 65%) because nearly every lab gives the subscription box. The city and the trip nearly triple, and the stock goes from 9% to 38%. On those questions the answers split among a few favorites (Austin, Denver and Chicago; Lisbon, Prague and Paris), and each model tends to stick to its own, so pooling made the answers look more varied than any single model is.

I don’t think either number is the right one. They bound the right one. Equal weight across 65 models is the floor, because real users are far more concentrated than that. Everyone on one model is the ceiling for these test conditions. Real usage is somewhere in between, and where depends on how concentrated usage is, which the file can’t show. Either way, the article’s main finding gets stronger. The 34% is the most charitable way to state it.

## The questions that “don’t collapse”

The same weighting shapes a sentence I’d otherwise have taken at face value: “Not everything collapses. For a movie, dinner, a baby name or names for testimonials, no single answer reaches a third of replies.” That’s true of the pooled replies. *The Grand Budapest Hotel* is 20% of 650 movie answers, Aurora is 15% of baby names, and spaghetti carbonara is 9% of dinners. But those figures mostly measure how much the labs disagree with each other.

Within a single model, two fresh requests for a movie give the same title 46% of the time on average, and two requests for a baby name give the same name 41% of the time. Even the dinner question, the most varied in the set, repeats 24% of the time. So pooling the labs is what makes these questions look varied. A person who always asks the same assistant gets a much narrower spread than the 20% suggests.

## Is this worse than people?

The natural objection is that people aren’t random either, and asking a hundred friends for a life-changing book would also turn up a few favorites. I don’t have a human survey of book recommendations to compare with. I do have a large literature on how badly people generate random numbers, and it shows two different kinds of bias.

Philip Boland and Kevin Hutchinson had 458 university students each write 25 “random” digits. The students’ digits weren’t uniform: 0 was underused, and 1, 2 and 3 were overused. But the effect that stands out is within a single sequence. 70% of students never wrote the same digit twice in a row, against about 8% expected from a true random generator. On the number the models love, the authors wrote: “The number 7 is considered lucky in some cultures but was not particularly favoured in this exercise.” John Towse, Tobias Loetscher and Peter Brugger pooled two adult studies, 134 people each producing 100 numbers from 1 to 10. They found a real but small lean toward smaller numbers, 51.3 picks from 1 to 5 against 48.7 from 6 to 10.

Those are sequences, not single picks, so they aren’t a direct baseline for the models’ 92% on 7. They do show what kind of bias people have. People tilt their choices by a few points and over-alternate within a run. The models do the second, and Collier’s best panel shows it. Claude Opus 5.5 gave eight different books when asked for “another one, please” eight times in one chat, and *Man’s Search for Meaning* in each of eight new chats.

The models also do something the students’ task couldn’t show: they reset. In the models, the variety exists only within a conversation, and every new conversation starts from the same first answer. In Collier’s setup, a new chat is effectively a new person.

## Why the second opinion is worth less

I don’t mind that people get the same book. *Man’s Search for Meaning* is a fine book, and for “where should I put $10,000” a broad index fund is a defensible answer for most people. The trouble starts when people treat a second model as a second opinion. The forecasting literature has worked on this since the 1980s. R. T. Clemen and R. L. Winkler derived a formula for how much information an additional expert adds when that expert is correlated with the existing experts. As Donald Morrison and David Schmittlein explain in a paper giving the geometric intuition, the formula says an extra correlated expert usually adds little, a result their title calls surprising. Ten correlated forecasters are worth far fewer than ten independent ones.

Collier’s fingerprint result puts a number on the correlation. Using only harmless questions such as a color, a city or a name, he found that “59% of models sit next to a sibling from their own lab, against 4% by chance.” Asking two Gemini models, or a Claude model and then another Claude model, is closer to asking one adviser twice than to asking two. His finding that Qwen4 Max answers most like Claude Opus 5.5, and NVIDIA’s Nemotron most like Qwen, means that a model with a different brand isn’t necessarily an independent opinion either. I ran into a version of this when I wrote about [checks done by two copies of the same model](https://gautamparab.com/writing/off-by-point-eight-with-a-margin-of-fourteen/). Agreement between relatives is cheap.

## What the method gets right, and what it can’t see

The repeat-probability metric is a good choice for a study with 10 to 50 asks per model. It’s a version of Simpson’s diversity index from ecology, and ecologists have long studied how such indices behave in small samples. Woollcott Smith and J. Frederick Grassle derived a minimum-variance unbiased estimator for a family of diversity measures that reduces to Simpson’s index in its simplest case. R. D. Routledge found that the small-sample bias of the Simpson index was much smaller than that of any proposed estimator of the Shannon index, the entropy measure behind “bits.” Collier uses bits only for the OLMo training-stage analysis, where he computes exact probabilities rather than sampling, so the bias doesn’t arise there. For the sampled questions he used the more robust measure.

The method can’t see the setting most people use. Every answer in the file came from an API call with no system prompt, no memory and no user history. A consumer assistant usually has a system prompt and often has memory, and his own results show what context does: a sentence of specific personal detail brought the stranger-match rate from 34% to 4%. An app that remembers you is supplying that sentence for you, at least in principle. So the deployed number could be well below either of my bounds for people with a long history, and near the top one for people starting fresh. The file can’t tell us which is more common. Collier is careful about the limits of his own data: “Fifty asks per question per model is enough to see a family, not to rank small differences between two models,” and the models “may answer differently elsewhere, or after an update.”

## What I’d take from it

The experiment is well done and the headline is, if anything, understated. Averaging over every model with equal weight is common in this kind of work. It answers what the market as a whole says. The question a user has is what their own assistant says to everyone else who uses it. On those terms, the chance that a stranger on your assistant got the same book, city or stock you did is closer to six in ten than to one in three, before any personalization the app adds.

His practical advice still works, and it works for the reason he gives. Saying who you are, or asking for ten options and choosing the number yourself, adds the variation the model won’t supply. I’d add one thing for anyone using a second model as a check: pick one from a different lab, and treat agreement between two models from the same lab as one opinion.

**References**

1. Collier, C. (2026). [*AI is giving everyone the same advice*](https://a90.com/experiments/same-advice/) . a90.com, 20 November 2026. Answer file:[answers.csv](https://a90.com/experiments/same-advice/answers.csv) , downloaded 20 November 2026.
2. Boland, P. J. and Hutchinson, K. (2000). [*Student Selection of Random Digits*](https://doi.org/10.1111/1467-9884.00250) . Journal of the Royal Statistical Society: Series D (The Statistician), 49(4), 519-529, December 2000.
3. Towse, J. N., Loetscher, T. and Brugger, P. (2014). [*Not all numbers are equal: preferences and biases among children and adults when generating random sequences*](https://doi.org/10.3389/fpsyg.2014.00019) . Frontiers in Psychology, 5, 2014.
4. Morrison, D. G. and Schmittlein, D. C. (1991). [*How Many Forecasters Do You Really Have? Mahalanobis Provides the Intuition for the Surprising Clemen and Winkler Result*](https://doi.org/10.1287/opre.39.3.519) . Operations Research, 39(3), 519-523, June 1991.
5. Smith, W. and Grassle, J. F. (1977). [*Sampling Properties of a Family of Diversity Measures*](https://doi.org/10.2307/2529778) . Biometrics, 33(2), 283, June 1977.
6. Routledge, R. D. (1980). [*Bias in Estimating the Diversity of Large, Uncensused Communities*](https://doi.org/10.2307/1935186) . Ecology, 61(2), 276-281, April 1980.
