Measuring Brand Mentions in LLM Answers Is Easy. Measuring Them Well Isn’t.

A 48-prompt, four-engine measurement run on 7 November 2026 returned 240 answers and 3,775 raw citations, which fell to 1,404 when each cited domain was counted once per answer, according to a first-person account published in the free weekly newsletter Mention Impossible. The author reports that vendor-owned sources were 15.8% of raw citations but 20.7% of unique-per-answer citations, and that in six "best alternatives to X" prompts the excluded brand was named in 33 of 34 answers and named first in 28, making the brand a user wants to avoid look like the top recommendation. In email marketing, Mailchimp was named in 21 of 40 answers, but 10 of those 21 mentions came from two of eight prompts that contain the word "Mailchimp"; on the other 30 answers ActiveCampaign led with 12 mentions, ahead of Mailchimp's 11 and Klaviyo's 10.

Disclosure: I publish these measurements in a free weekly newsletter, Mention Impossible, and no brand mentioned here is a client or sponsor. Much of this text was drafted with an AI writing assistant, then edited by me and checked against the raw result tables. “Share of mention” in AI answers is turning into a marketing metric. People want to know how often ChatGPT, Gemini, Claude or Perplexity name their product when a buyer asks for a recommendation. Collecting the data is not hard. Producing numbers that mean what they appear to mean is harder. This post covers six traps I ran into while building a small weekly measurement, with figures from one run so you can see how large each effect is. The panel is 48 fixed English prompts, eight per category across six software categories: CRM, email marketing, project management, ecommerce platforms, website builders and AI writing tools. The engines run through their official APIs with web search enabled: | Engine | Model / setting | Prompts | Runs per prompt ||---|---|---|---|| OpenAI | gpt-6.1-sol | 24 shared | 1 || Anthropic | claude-sonnet-5-5 | 24 shared | 1 || Google | gemini-3.8-flash | all 48 | 1 || Perplexity | Agent API "fast" preset | all 48 | 3 | The first four prompts in each category are shared by all four engines. The run on 7 November 2026 returned 240 answers with no failures. The processing is deliberately simple: A simplified version of the counting step: python import refrom collections import defaultdictfrom urllib.parse import urlparse python def mentions answer text, brands : """Return {brand: rank}, where rank = order of first appearance.""" hits = for brand, aliases in brands.items : positions = m.start for a in aliases for m in re.finditer rf"\b{re.escape a }\b", answer text, re.I if positions: hits.append min positions , brand return {brand: i + 1 for i, , brand in enumerate sorted hits } python def unique domains citation urls : """Count each cited domain once per answer.""" return {urlparse u .hostname.removeprefix "www." for u in citation urls} Each of these choices turned out to matter. Across the 240 answers, the APIs returned 3,775 citations. Counting each domain once per answer leaves 1,404. The reduction is not spread evenly. Average raw citations per answer: | Engine | Raw citations per answer | Range ||---|---|---|| Gemini | 52.3 | 9–86 || Claude | 18.9 | 12–28 || ChatGPT | 7.2 | 3–12 || Perplexity | 4.4 | 2–11 | On raw counts, Gemini’s habits would dominate any pooled source statistic. The headline number also changes: vendor-owned sources are 15.8% of raw citations but 20.7% of unique-per-answer citations. Neither number is wrong, but they answer different questions. I report the unique version, because the question I care about is “did this answer rely on this site?”, not “how many times did the API list it?”. “Best alternatives to X” is a common buyer question, so the panel includes six prompts of that form: alternatives to HubSpot, Mailchimp, Asana, Shopify, WordPress and Jasper. Those prompts produced 34 answers. The excluded brand was named in 33 of them. It was also named first in 28, because answers usually name the brand while restating the question before they list any alternatives. On a first-appearance rank metric, the brand the user is trying to avoid looks like the top recommendation. The same effect leaks into share of mention. In email marketing, Mailchimp was named in 21 of 40 answers, the category’s highest share. But two of the eight email prompts contain the word “Mailchimp”: the alternatives question and a “Klaviyo vs Mailchimp vs Omnisend” comparison. Together they produced 10 of Mailchimp’s 21 mentions. On the other 30 answers the order changes: ActiveCampaign 12, Mailchimp 11, Klaviyo 10. Alt text: Two-panel chart. Left: horizontal bars showing how often the excluded brand was named first in “alternatives” questions: Mailchimp 6 of 6, Jasper 6 of 6, HubSpot 5 of 6, Shopify 5 of 6, WordPress 3 of 4, Asana 3 of 6. Right: Perplexity question–brand pairs by number of runs in which the brand appeared: all 3 runs 131, 2 runs 24, 1 run 30. Fix: tag every prompt that contains a brand name, and report those prompts separately or exclude them from rankings. For rank, consider “first brand recommended” rather than “first brand mentioned”. That needs a classifier or manual labels, not a regex. Perplexity is cheap enough to run three times per prompt, which gives a direct look at run-to-run variance. The top pick is fairly stable. Perplexity named the same brand first in all three runs on 42 of 48 prompts. By category, that ranged from 8 of 8 in email marketing to 6 of 8 in AI writing tools. The full list of brands is much less stable. Across the three runs there were 185 question–brand pairs that appeared at least once: So 54 of 185 pairs, 29%, did not appear in every run of the same question. A brand at the bottom of a recommendation list can appear or disappear between runs without anything changing in the world. ChatGPT, Claude and Gemini ran once per prompt in this setup, so their noise is unmeasured, not absent. If budget allows only one engine with repeats, the repeats still tell you roughly how much movement to ignore. On the 24 prompts every engine answered, all four named the same brand first on 10. It is tempting to read that as a consensus rate. But 8 of those 24 shared prompts name a brand. On those eight, the engines agreed on 4. On the other 16, they agreed on 6. The brand-named prompts make agreement easier, partly through the effect in Trap 2. Without them, agreement is 6 of 16, a bit under 40%. That is closer to “the engines usually disagree on the top pick”. Share of mention counts an answer if a brand appears anywhere in it. Engines differ a lot in how many brands they list: | Engine | Distinct brands per answer ||---|---|| Claude | 6.3 || Gemini | 4.7 || ChatGPT | 3.8 || Perplexity | 3.3 | A brand is more likely to be “mentioned” by Claude simply because Claude lists nearly twice as many brands as Perplexity. Comparing raw share of mention across engines partly compares answer length. Fix: report share of mention per engine, and alongside it a position-aware metric, such as presence in the top three. Then a longer list doesn’t count as a stronger endorsement. With three runs per prompt and all 48 prompts, Perplexity supplies 24 of the 40 answers in each category. Gemini supplies 8, ChatGPT 4 and Claude 4. A pooled share of mention is therefore mostly a Perplexity number, with the other engines as a minority. That may be fine if you think Perplexity represents your buyers. It isn’t fine by accident. Fix: either weight engines explicitly, for example equally or by estimated user share, or don’t pool, and report each engine with its own n. With 40 answers per category, one answer moves a share by 2.5 percentage points. A rough 95% margin for a share around 50% is about ±15 points at n = 40, and about ±18 at n = 30. That is a normal approximation, and with the repeated runs inside it the true uncertainty is probably larger. So most week-to-week moves in a panel this size are noise. I currently treat moves under about 10 points as noise and anything bigger as a lead to check, not a finding. A better approach is to publish proper intervals, for example by bootstrapping over prompts and runs, instead of a rule of thumb. Most of the work in measuring LLM brand recommendations is in the counting rules, not the API calls. Prompt wording, citation de-duplication, repeat runs, answer length and pooling weights each moved the numbers in this sample by as much as the week-to-week changes people tend to report. Make those choices explicitly and publish them, and the numbers become comparable from week to week. 1 WhiskeyJack, Mention Impossible: Method and weekly tables 2026 , https://data.mentionimpossible.com/method/ https://data.mentionimpossible.com/method/ WhiskeyJack, Mention Impossible Measuring Brand Mentions in LLM Answers Is Easy. Measuring Them Well Isn’t. https://pub.towardsai.net/measuring-brand-mentions-in-llm-answers-is-easy-measuring-them-well-isnt-d00af2d111c5 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.