In our AI taste study, ten models from ten labs each built a site for the same ten briefs, and then people chose between the results in blind pairwise comparison: 450 matchups, 4,500 pairwise preference ranking annotations, 29 annotators, no model names on screen.
Every run carries a price tag. So we ask a question that usually gets answered by vibes: does spending more produce work people prefer?
muse-spark-1.3 built its sites for an average of $0.16 of inference. gpt-6-astra spent $1.53, about 9.5 times as much.
gpt-6-astra finished first overall at 64.3%. muse-spark-1.3 finished third at 60.3%.
Nine and a half times the spend bought four points of preference, and two places in a ten-model ranking.
| Model | Inference, average per site | Preference | Overall rank |
|---|---|---|---|
| muse-spark-1.3 | $0.161 | 60.3% | 3 |
| glm-5.3 | $0.178 | 53.2% | 6 |
| seed-2.0-code | $0.181 | 19.1% | 10 |
| deepseek-v4-pro | $0.256 | 56.2% | 4 |
| kimi-k3 | $0.271 | 53.3% | 5 |
| grok-4.6 | $0.345 | 62.7% | 2 |
| claude-sonnet-5 | $0.379 | 49.1% | 7 |
| gemini-3.8-flash | $0.427 | 32.8% | 9 |
| qwen3.8-2.4t | $0.536 | 48.9% | 8 |
| gpt-6-astra | $1.532 | 64.3% | 1 | Inference charges only, averaged over each model's ten briefs. Preference is the share of 900 blind pairwise votes each model won across the study.
Across models. Rank the ten models by what they spend and by how much people liked them, and the two orders barely relate: a rank correlation of +0.06. The third most expensive model in the study, gemini-3.8-flash, finished ninth overall.
Within a single model. Hold the model constant and look at its ten briefs: did the runs it spent more on turn out better? Averaged across all ten models, that correlation is +0.03. Four of the ten are negative. Whatever extra tokens a model burns on a particular brief, the result is not more likely to be chosen.
In time. Runs took between 162 and 1,604 seconds. The correlation between how long a run took and how well it did is −0.17, pointing gently the wrong way. The sites that took longest to produce did slightly worse.
Look at the bottom right of the chart. It is empty.
No run in the study costs more than $0.90 of inference and scores below 35%. Spending does not reliably produce work people prefer, but it does seem to buy out the worst outcomes. The failures are clustered among the cheap and mid-priced runs.
The takeaway here, then, is more nuanced than "cost does not matter." Cost buys a floor. It does not buy a ceiling, and the gap between the floor and the ceiling is where all the interesting variation lives.
The third cheapest model in the study, seed-2.0-code at $0.18 a site, finished last at 19.1%. It lost roughly four of every five votes it appeared in. This is largely because the model failed to generate a viable site for nearly every brief. Was this a cost issue? Probably not, but there’s no real way to know.
That matters because the obvious misreading of this data is "use the cheap model." The right reading is that price is close to uninformative in both directions. muse-spark-1.3 and seed-2.0-code cost within two cents of each other per site and finished third and tenth.
A price per token reflects parameter count, serving economics, and what a lab decided to charge. None of those is a measurement of whether the output is any good, and nobody ever claimed they were. The reason the question feels like it should have a yes is that in most of engineering, paying more for compute does buy a better answer, because the answer has a correctness criterion you can check.
Taste does not have one. There is no compiler, so there is nothing for the extra compute to optimize against.
If you are choosing a model for generated design work, the price list will not rank them for you, and neither will a capability benchmark: all 100 of these sites render, work, and pass. What does rank them is showing the output to people and asking which one they would choose. That is a slower instrument and a more expensive one, and on this evidence it is the only one that answers the question you are actually asking.
All 100 sites, their costs, and the full preference data are in the public dashboard.