{"slug": "what-happens-when-you-ask-8-ai-models-the-same-buying-question-every-month", "title": "What happens when you ask 8 AI models the same buying question every month", "summary": "A developer built a harness to measure the stability of AI model recommendations by asking eight models the same B2B software buying question every month. Across sixteen categories, the models never unanimously agreed on a single tool, and individual models contradicted their own previous picks 74% of the time. The project's data is open-source and updated monthly to track recommendation drift.", "body_md": "A while back I got annoyed at a specific genre of blog post: \"we asked ChatGPT what the best CRM is and here's the answer.\" One screenshot, one run, treated as if the model holds a stable opinion. It doesn't. So I built a small harness to measure that instead of hand-waving about it.\n\nThe setup is boring on purpose. Eight models. Sixteen B2B software categories (CRM, project management, email marketing, that kind of thing). For each category I ask every model the same plain question: what is the single best tool here. One pick, no hedging allowed. I log the raw response, the parsed pick, the model, the timestamp, and the exact prompt into a JSONL file. Then I do it again next month.\n\nTwo things fell out that I did not fully expect.\n\nFirst, across all sixteen categories the eight models never once agreed on the same tool. Not close-but-different. Zero unanimous picks out of sixteen. I assumed there would be at least a couple of categories where everyone converged on the obvious incumbent. Nope.\n\nSecond, and this is the one I keep chewing on: the models do not even agree with themselves. Ask the same model the same question in a fresh session and it swaps its own top pick around 74% of the time. Same model, same prompt, nothing changed but the session. Roughly three-in-four odds it contradicts what it told you yesterday.\n\nHere is the part I want other people to break. I do not know how much of that self-disagreement is temperature and how much is genuine ranking instability. My instinct is that dropping temperature to 0 will not actually fix it, it will just hide the wobble behind a deterministic-looking facade while the underlying ranking stays mush. But I have not run that ablation cleanly yet, and I would rather someone who does eval work for a living tell me I am wrong.\n\nThe reason I am posting the method and not only the numbers: a \"models disagree\" claim is worthless if you cannot reproduce it. So everything is open. The per-run JSONL, the prompt text, the model list, all of it, DOI'd under CC-BY so you can cite it or tear it apart: [https://data.deepsynthesis.org/](https://data.deepsynthesis.org/). I re-run the whole thing monthly, which means you can watch a specific category drift over time rather than trusting a frozen screenshot from whenever the author happened to hit send.\n\nA few implementation notes if you want to build your own version:\n\nThe uncomfortable takeaway for anyone doing AI-search or GEO work: if a model's pick is this unstable, \"we rank first when you ask ChatGPT\" is a coin that lands differently every flip. Optimizing for a single snapshot is optimizing for noise. Optimizing to show up across many runs and many models is the only thing that survives.\n\nIf you poke at the data and find a category where the parser is wrong, or a model I should add, tell me. That is the point of putting it out in the open.", "url": "https://wpnews.pro/news/what-happens-when-you-ask-8-ai-models-the-same-buying-question-every-month", "canonical_source": "https://dev.to/brainbootdev/what-happens-when-you-ask-8-ai-models-the-same-buying-question-every-month-5eh5", "published_at": "2026-07-11 23:54:13+00:00", "updated_at": "2026-07-12 00:13:25.630320+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["ChatGPT", "DeepSynthesis"], "alternates": {"html": "https://wpnews.pro/news/what-happens-when-you-ask-8-ai-models-the-same-buying-question-every-month", "markdown": "https://wpnews.pro/news/what-happens-when-you-ask-8-ai-models-the-same-buying-question-every-month.md", "text": "https://wpnews.pro/news/what-happens-when-you-ask-8-ai-models-the-same-buying-question-every-month.txt", "jsonld": "https://wpnews.pro/news/what-happens-when-you-ask-8-ai-models-the-same-buying-question-every-month.jsonld"}}