{"slug": "ask-chatgpt-or-gemini-the-same-best-x-twice-1-3-of-the-answer-changes", "title": "Ask ChatGPT or Gemini the same \"best X\" twice, ~1/3 of the answer changes", "summary": "A study of run-to-run consistency in web-search-enabled LLMs found that about 31% of recommended businesses changed when the same \"best X for a small business\" question was asked again minutes later, based on 60 responses from ChatGPT and Google Gemini collected on a single day. ChatGPT showed 87.2% consistency (SD 17.5, range 59–100%) while Gemini showed 51.9% (SD 11.8, range 33–67%), for an overall mean pairwise Jaccard overlap of 69.5%. The study's data and analysis code were released publicly, with 56% of business slots (71 of 127) named in every repetition and 27% (34 of 127) named in only one of three.", "body_md": "**How stable are the business recommendations you get from web-search-enabled LLMs when you ask the same question more than once?**\n\nThis repository releases the full response data and analysis code from an exploratory study of run-to-run consistency in ChatGPT and Google Gemini. We asked ten open-ended \"best X for a small business\" questions, three times each, to each assistant on a single day (**60 responses total**), extracted the set of businesses each answer recommended, and measured how much that set changed between identical repeated queries.\n\nThe short version: **about one recommended business in three changed when the same question was asked again minutes later.** But the instability is structured, not random. The top one to three names stay locked in; the lower-ranked slots rotate.\n\nThis is a small, deliberately-scoped study. The numbers are directional, not definitive, and we are upfront about the limits below. The reason we are releasing every response is so anyone can check the work.\n\n| Metric | Value | \n|---|---|\n| Overall run-to-run consistency (mean pairwise Jaccard overlap) | **69.5%** | \n| Equivalently, share of recommended businesses that changed on a repeat ask | **~31%** | \n| ChatGPT consistency (web search) | **87.2%** (SD 17.5, range 59–100%) | \n| Gemini consistency (Search grounding) | **51.9%** (SD 11.8, range 33–67%) | \n| Business-slots named in **every** repetition (stable core) | **56%** (71 of 127) | \n| Business-slots named in **only one** of three (volatile tail) | **27%** (34 of 127) | \n| Responses grounded in a live web search | **100%** (60/60) | \n\n**The structural finding is the interesting part.** A single \"AI is 69% consistent\" number hides the shape. The leading recommendations for a given question recur across every run (a stable core), while the fourth, fifth, and sixth positions shuffle each time (a volatile tail). For a smaller or newer business, the positions you could realistically reach are exactly the least stable ones.\n\n| File | What it is | \n|---|---|\n| [`data/study_final_2026-09-13.json`](/jjoseph18/ai-recommendation-consistency/blob/master/data/study_final_2026-09-13.json) | The complete raw run: all 60 responses with full answer text, extracted brand sets, citations, model IDs, latencies. This is the canonical source; the CSVs are derived from it. | \n| [`data/responses.csv`](/jjoseph18/ai-recommendation-consistency/blob/master/data/responses.csv) | One row per response (60 rows). Includes the full answer text, so the dataset is self-contained and independently verifiable. | \n| [`data/brand_mentions.csv`](/jjoseph18/ai-recommendation-consistency/blob/master/data/brand_mentions.csv) | Tidy/long format, one row per recommended business per response (290 rows). The easiest file to load for analysis. | \n| [`data/consistency_by_question.csv`](/jjoseph18/ai-recommendation-consistency/blob/master/data/consistency_by_question.csv) | Per-question, per-engine Jaccard consistency (20 rows). | \n| [`analyze.mjs`](/jjoseph18/ai-recommendation-consistency/blob/master/analyze.mjs) | The analysis script. Run it against the JSON to regenerate every headline number yourself. Node 18+, no dependencies. | \n| [`DATA_DICTIONARY.md`](/jjoseph18/ai-recommendation-consistency/blob/master/DATA_DICTIONARY.md) | Every column in every file, defined. | \n| [`CITATION.cff`](/jjoseph18/ai-recommendation-consistency/blob/master/CITATION.cff) | Citation metadata. | \n\nReproduce the headline stats:\n\n```\nnode analyze.mjs data/study_final_2026-09-13.json\n```\n\nTen open-ended \"best/top X for a small business\" questions, one per commercial category:\n\n| ID | Category | Question | \n|---|---|---|\n| q01 | PR distribution | What are the best PR distribution services for a small business? | \n| q02 | CRM | Top CRM platforms for startups | \n| q03 | Email marketing | Best email marketing services in 2026 | \n| q04 | Project management | Leading project management software for small teams | \n| q05 | Accounting | Best accounting software for freelancers | \n| q06 | Marketing agencies | Best digital marketing agencies for startups | \n| q07 | Website builders | Top website builders for a small business | \n| q08 | Live chat | Best live chat software for a small business website | \n| q09 | E-commerce | Top e-commerce platforms for a small online store | \n| q10 | Scheduling | Best appointment scheduling software for a small business | \n\nWe used **open** questions on purpose. Head-to-head comparison prompts (\"X vs Y\") were deliberately excluded: they can only return the two names you put in the prompt, so they trivially maximize apparent consistency and would inflate the topline. Open \"best X\" questions are also exactly the case a business wants to be recommended in.\n\n- **ChatGPT** via the OpenAI Responses API with the`web_search` tool enabled.\n- **Google Gemini** via the Generative Language API with Google Search grounding enabled.\n\nAll queries specified a United States locale and English responses. Each question was issued **three independent times** to each assistant, yielding 60 responses. All responses were collected on **2026-09-13** within a single session to hold model versions roughly constant. On this run, **all 60 responses were grounded in a live web search** (verified per response, not assumed).\n\nFrom each response we extracted the set of specific businesses recommended, using an automated language-model extraction pass instructed to include only named businesses presented as options, and to exclude media outlets, generic categories, and services named only as context.\n\nWe validated the extraction against the source text: **all 290 extracted business names appear verbatim in the response they were drawn from** (zero fabricated names). You can check this yourself, every answer's full text is in the dataset.\n\nFor each question and assistant, consistency is the **mean pairwise [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index)** of the recommended-business sets across the three repetitions:\n\n```\nJ(A, B) = |A ∩ B| / |A ∪ B|\n```\n\nWe count only repetitions that produced a shortlist of two or more businesses, so an empty or single-item answer can never masquerade as a \"different recommendation.\" Brand names are normalized (lowercased, non-alphanumerics stripped) before set comparison.\n\nWe separately count, per question-assistant, how many distinct businesses appeared in **all** repetitions (the stable core) versus **exactly one** (the volatile tail).\n\nThis is a **small exploratory study**, and treating it as more than that would be misusing it.\n\n- **Ten questions, three repetitions, two assistants, one day.** Enough to surface a pattern, not to estimate precise category-level or assistant-level rates.**The reported means carry no formal confidence interval, read them as directional.**\n- **Three repetitions quantize the per-question values coarsely** (a per-cell value can only land on a few discrete points). One cell (Gemini, e-commerce) rests on two repetitions rather than three, because the third named a single business and was excluded by the two-or-more rule.\n- **Automated extraction can under- or over-count borderline mentions** (for example treating \"Wave\" and \"Wave Accounting\" as distinct). This biases toward*understating* consistency, not inflating it.\n- **The \"most named\" brand tally excludes any brand named in a prompt** , so a model is never credited for repeating a name we handed it.\n- **Findings reflect two specific assistants on one day.** Assistant behavior evolves; a rerun months later will differ.\n- **This study was conducted by a commercial entity** (Pressfront) with an interest in the subject. That is exactly why the full method and every response are released here: so you don't have to take our word for any of it.\n\nIf you rerun it and get different numbers, that's a finding, not a contradiction. Open an issue or a PR.\n\nThe ten questions, the three-repetition procedure, and the Jaccard-overlap measure above are sufficient to reproduce the study end to end. `analyze.mjs` regenerates every published number from the raw JSON. If you have OpenAI and Gemini API keys and want to run a fresh collection (more questions, more repetitions, more engines, a later date), the method is fully specified in this README, and we'd genuinely like to see what you find.\n\nUseful extensions we haven't done: more repetitions per question (to get real confidence intervals), more engines (Perplexity, Claude, Copilot), longitudinal reruns of the same questions over weeks, and per-category sample sizes large enough to compare categories.\n\nIf you use this data, please cite it (see [`CITATION.cff`](/jjoseph18/ai-recommendation-consistency/blob/master/CITATION.cff)). The full writeup is in the preprint bundled here: [`ai-recommendation-consistency-preprint.pdf`](/jjoseph18/ai-recommendation-consistency/blob/master/ai-recommendation-consistency-preprint.pdf), and the DOI-registered version is on Zenodo:\n\nPressfront Research (2026). *Run-to-Run Consistency of Business Recommendations from Web-Search-Enabled Large Language Models: An Exploratory Study.* Zenodo. [https://doi.org/10.5281/zenodo.22738861](https://doi.org/10.5281/zenodo.22738861)\n\n**DOI:** [`10.5281/zenodo.22738861`](https://doi.org/10.5281/zenodo.22738861) (this version) · [`10.5281/zenodo.22738860`](https://doi.org/10.5281/zenodo.22738860) (all versions)\n\nFor the companion writeup and further analysis: [pressfront.co/research](https://pressfront.co/research)\n\nData and code released under [CC BY 4.0](/jjoseph18/ai-recommendation-consistency/blob/master/LICENSE) — use it freely, including commercially, just credit the source.", "url": "https://wpnews.pro/news/ask-chatgpt-or-gemini-the-same-best-x-twice-1-3-of-the-answer-changes", "canonical_source": "https://github.com/jjoseph18/ai-recommendation-consistency", "published_at": "2026-09-13 23:31:00+00:00", "updated_at": "2026-09-13 23:52:04.112034+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-products"], "entities": ["ChatGPT", "Google Gemini", "OpenAI", "Google"], "alternates": {"html": "https://wpnews.pro/news/ask-chatgpt-or-gemini-the-same-best-x-twice-1-3-of-the-answer-changes", "markdown": "https://wpnews.pro/news/ask-chatgpt-or-gemini-the-same-best-x-twice-1-3-of-the-answer-changes.md", "text": "https://wpnews.pro/news/ask-chatgpt-or-gemini-the-same-best-x-twice-1-3-of-the-answer-changes.txt", "jsonld": "https://wpnews.pro/news/ask-chatgpt-or-gemini-the-same-best-x-twice-1-3-of-the-answer-changes.jsonld"}}