# How do you measure something that gives a different answer every time?

> Source: <https://dev.to/agustaon/how-do-you-measure-something-that-gives-a-different-answer-every-time-55m5>
> Published: 2026-07-29 06:44:55+00:00

I had a simple-sounding question: does ChatGPT recommend this business?

You'd think you just ask it. Ask ChatGPT "best personal injury law firm in NYC", see if the business is named, record yes or no.

That works exactly once. Ask again an hour later and you might get a different answer. Not slightly different — potentially a completely different set of firms and a completely different set of cited sources.

Which means the naive version of this measurement is worthless. You're not measuring visibility, you're sampling a distribution once and calling it a fact.

This is the same problem anyone gets when they try to test an LLM-backed feature. Your normal testing instinct — same input, assert on output — just doesn't apply. So here's how I ended up designing around it, and the numbers that came out, which surprised me.

I wanted to compare four assistants (GPT-4o, Claude Haiku 4.5, Gemini 2.5 Flash, Perplexity Sonar, all with web search on) across 10 buyer-intent questions in one vertical. Something like:

```
"Best personal injury law firm in New York City?"
"Top immigration lawyers in Mumbai?"
```

For each response I recorded two things: which businesses got named, and which URLs got cited. The cited sources come from each API's own citation metadata, so that part is structured — no scraping the prose.

First pass, the results looked dramatic. The four assistants barely agreed on anything. Different firms, different sources, almost no overlap.

Great finding. Except I couldn't publish it, because there was an obvious objection I couldn't answer:

**Maybe they weren't disagreeing with each other. Maybe each one was just disagreeing with itself.**

If a single assistant returns wildly different sources run to run, then "these four models cite different things" is a meaningless statement. You'd be measuring noise and calling it signal.

The fix is the same idea as a control group. Measure the thing you're worried about, separately, and see if it explains your result.

So: re-run all 10 questions **3× per assistant**, and compute two different numbers.

If those two numbers are close, the divergence is just non-determinism and I have nothing. If self-consistency is much higher, the models genuinely differ.

For "overlap" I used Jaccard similarity on the sets of cited domains — intersection over union. Standard, boring, and easy to defend when someone asks:

``` js
function jaccard(setA, setB) {
  const a = new Set(setA);
  const b = new Set(setB);
  if (a.size === 0 && b.size === 0) return 1;

  let intersection = 0;
  for (const x of a) if (b.has(x)) intersection++;

  const union = a.size + b.size - intersection;
  return intersection / union;
}
```

Cross-consistency for a question is the mean pairwise Jaccard across the 6 assistant pairs. Self-consistency is the mean pairwise Jaccard across the 3 runs of a single assistant. Then average across questions.

``` js
function meanPairwise(runs) {
  const scores = [];
  for (let i = 0; i < runs.length; i++) {
    for (let j = i + 1; j < runs.length; j++) {
      scores.push(jaccard(runs[i], runs[j]));
    }
  }
  return scores.reduce((a, b) => a + b, 0) / scores.length;
}
```

Two normalisation details that matter more than they sound:

**Normalise to registrable domain.** `www.example.com/page-a`

and `example.com/page-b`

are the same source for this purpose. If you don't collapse those, you'll measure URL formatting instead of source overlap and your numbers will be garbage.

**Decide what counts as a source and write it down.** I counted only domains that appeared in the API's citation metadata, not domains mentioned in prose. Defensible either way, but you have to pick one and say which.

So an assistant is roughly **7.5× more consistent with itself than with its rivals**. The divergence between models is real. The control did its job.

But look at that self-consistency number again, because it's the more interesting one. **Ask the same model the same question twice and half the cited sources change.**

And it isn't uniform:

| Assistant | Sources repeating on a second ask |
|---|---|
| Perplexity | ~72% |
| Claude | ~54% |
| Gemini | ~42% |
| ChatGPT | ~32% |

Roughly two-thirds of what ChatGPT cites changes between two identical questions. Perplexity is more than twice as stable.

That spread is the practically useful finding. If you're building anything that depends on retrieval consistency, your choice of provider changes how reproducible your system is by a factor of two — before you've written a line of your own code.

If you're testing anything LLM-backed, single-run assertions are lying to you. A few things I'd carry to any similar problem:

**Run n times and report a distribution, not a value.** n=3 was enough to separate signal from noise here. It won't be for everything — if your effect size is smaller, you need more runs. Sanity-check by increasing n and seeing whether your headline number moves.

**Measure your noise floor explicitly.** The self-consistency number *is* the noise floor. Any difference between two things you're comparing has to clear it to mean anything. Most "we compared model A and B" posts skip this, which is why most of them are unreliable.

**Assert on properties, not exact outputs.** "Response cites at least one source from an allowed list" survives non-determinism. "Response equals this string" does not.

**Pin what you can.** Model version, `temperature`

, whether web search is on, region. My biggest regret is not pinning model versions harder — a provider updating a model mid-study would have quietly invalidated everything, and I'd have had no way to detect it after the fact.

Two things worth being explicit about, since this space is full of overclaiming:

**This is a snapshot.** July 2026, one vertical, 10 questions. Retrieval behaviour changes with model updates. The 32% figure for ChatGPT is not a constant of nature — it's what I measured, in that window, with that method.

**More runs would be better.** n=3 is enough to establish that self-consistency >> cross-consistency, which is the claim I'm making. It is not enough to say Claude's 54% is meaningfully different from Gemini's 42%. Those two could easily swap places with more sampling, and I wouldn't defend that ordering.

If someone wants to run it with n=20 and tell me I'm wrong, genuinely great — that's a better dataset than mine.

Full write-up with all 10 questions, every business named, every source cited, and the method in more detail:

[https://greaterthanservices.com/blog/ai-search-study-law-firms](https://greaterthanservices.com/blog/ai-search-study-law-firms)

Disclosure: I run [Greater Than Services](https://greaterthanservices.com), which does this measurement as a product — the variance control exists because customers were reasonably asking "how do you know this isn't random?" The method above is the answer, and it's not proprietary, so take it and run it on your own vertical if it's useful.

*If you've built variance controls for LLM-dependent systems I'd like to hear how you handled it — particularly how you picked n, which is the part I'm least confident I got right.*