cd /news/generative-engine-optimization/how-llm-search-engines-pick-sources-… · home › topics › generative-engine-optimization › article
[ARTICLE · art-167653] src=pub.towardsai.net ↗ pub= topic=generative-engine-optimization verified=true sentiment=· neutral

How LLM Search Engines Pick Sources: A Practical Retrieval Checklist for Brands

Brands seeking citations in ChatGPT search, Claude, Perplexity and Google AI Overviews must optimize for passage-level retrieval rather than page-level ranking, according to a practitioner checklist from search consultant Adrian Grabski. Grabski separates search crawlers from training crawlers — OpenAI's OAI-SearchBot versus GPTBot, Anthropic's Claude-SearchBot versus ClaudeBot — and cites Profound data from 240 million citations between April and July 2025 showing ChatGPT source overlap with Bing fell to about 8% while overlap with Google rose to about 33%. He also notes a 2025 Search Engine Land analysis that found answer-style blocks in 72.4% of cited posts and a reported figure that ChatGPT draws 44.2% of citations from the first third of a page.

by read7 min views3 publishedNov 11, 2026

When a language-model search engine answers a question, it is not remembering your brand. In ChatGPT search, Claude, Perplexity and Google AI Overviews, the model first retrieves a small set of documents, then generates an answer from those documents. If your page never enters that set, no amount of clever wording will get you cited.

I have worked in search since 2002, first in classical SEO and now also in generative engine optimization (GEO). The useful frame is the retrieval pipeline: crawl, chunk, retrieve, generate, cite. Here is a checklist for each stage.

A typical LLM search stack works roughly like this. A query is expanded into several sub-queries (query fan-out). Each sub-query hits a search index or a live crawl. The returned pages are split into passages. A retriever scores those passages, a generator writes an answer, and a citation is attached to the passages that most supported the claim.

You therefore compete at passage level, not page level, and a block at any earlier stage wastes later work. That is why I put accessibility first in the priority order I use in my GEO and AI search guide: bot access, then citable paragraphs, then entity consistency and earned mentions. That 80/20 is a heuristic from implementations, not a measured law.

The most expensive confusion is treating every AI user-agent as a training crawler. OpenAI uses GPTBot to collect training data and OAI-SearchBot to fetch pages for ChatGPT search — the bot that actually drives citations. Anthropic splits the same way with ClaudeBot and Claude-SearchBot. PerplexityBot both indexes and cites. AI Overviews and AI Mode have no special crawler; they use Googlebot’s Search index. Blocking Google-Extended (Gemini training) will not remove you from AI Overviews. Bingbot still matters as one intake path into ChatGPT. Profound (240 million citations, April–July 2025) reported that overlap of ChatGPT sources with Bing had fallen to around 8% in three months, while overlap with Google had risen to around 33%.

A citation-friendly robots.txt allows the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) and may disallow training bots if that is your policy. Blocking GPTBot does not hurt ChatGPT citations, provided OAI-SearchBot remains allowed. The sitemap in robots.txt must return HTTP 200; a 404 is worse than no sitemap. llms.txt is inexpensive hygiene, not a retrieval lever: Google stated in a 15 May 2026 guide that it does not use it.

robots.txt is also not the only gate. Cloudflare Bot Fight Mode and “block AI bots” policies can intercept crawlers regardless of robots.txt. Then check whether the content exists in the raw HTML. Most AI search crawlers do not execute JavaScript, so a page of empty shells is blank for ChatGPT, Claude and Perplexity. Googlebot is the exception: it renders JavaScript, and AI Overviews use its index. Confirm with curl that a distinctive sentence appears in the raw HTML and that the sitemap returns 200. A forged User-Agent is only a partial test, because CDNs verify crawler IPs; use server or CDN logs, not Google Analytics. Do not hide key facts in images. After a URL change, old addresses can still be cited for two to four months. When this cheapest stage is broken, later work will not show up in measurements.

Once a page is fetched, it is sliced into chunks, and the passage that wins a sub-query is the one that can stand alone as an answer. I call the editorial pattern an Answer Capsule: a direct answer plus one or two context sentences, immediately under the heading, typically up to about 60 words (the source analysis measured the lead sentence at roughly 120–150 characters).

This is a writing heuristic, not a secret ranking factor. A 2025 Search Engine Land analysis found such a block in 72.4% of cited posts — a correlation, not proof of cause. One reported figure is that ChatGPT takes 44.2% of its citations from the first third of a page. Liu et al. (TACL 2024) described a U-shaped “lost in the middle” effect, with accuracy in the middle dropping by more than 30 percentage points. Put the answer first.

A chunk-ready paragraph carries its own thesis, its own fact, and, if needed, its own definition. If you could paste it into any article in your industry without changing its meaning, delete it. Google’s official guidance is explicit: you do not need to write in a special format for AI.

Retrievers also have to decide who a source is. If your site, public profiles and bylines disagree about the company name, address, person or product, the system has no stable entity to attach a citation to. Hallucinations about a brand often start as an information gap: when the model cannot find a consistent fact, it fills one in.

Publish a dated source-of-truth page. Use one official name. Keep name, address and phone identical across the site, Google Business Profile and directories. Sign work with a real person, a short bio and the same profile URLs. In Organization and Person schema, sameAs is the field that ties those profiles together. Entity work does not replace content; it reduces the chance that a retrieved passage looks like a different organisation.

Structured data is the most oversold item on this list. In a controlled Ahrefs test on 1,885 pages, adding schema produced no citation lift (a −4.6% movement, with visibility falling in both groups). That is evidence of no positive effect, not that schema is harmful. SE Ranking (129,000 domains) found pages without FAQPage markup cited more often than pages with it (4.2× versus 3.6×).

Implement schema anyway for classic search, authorship and entity building. Organization (with sameAs), Person, Article and, where relevant, Product/Offer and BreadcrumbList are the types that earn their keep. They must describe what is visible on the page: a language model cannot cite a fact that exists only in a script tag.

Start by validating what you already have. In a Digital Applied audit of 5,000 sites, 71% had some schema and only 22% had error-free schema — 49% of the sample shipped validation errors. Fixing broken markup is more urgent than adding an exotic type. Do not count on FAQPage for rich results: Google has been sunsetting them from 7 May 2026.

There is no Search Console for ChatGPT. A single screenshot is an anecdote. SparkToro ran 2,961 prompts with about 600 volunteers and found that the chance of getting the same list of brands twice in a row was lower than 1 in 100. AI Overviews also replace their content for the same query in around 70% of cases.

I use four metrics, per platform: Share of Voice, average mention position, Citation Rate (answers with a live link) and Citation Links (unique URLs on your domain that get cited). If Citation Links sits at one or two, the system knows only your homepage. Five runs are enough for a rough test; 10–20, spread across days and sessions, before an editorial decision. Report a rate with a range. Measure platforms separately: Profound’s analysis of 100,000 prompts found only about 11% overlap in citations across platforms. If you monitor only one system, start with ChatGPT. Goodie (2026) put it at 62.5% of measurable B2B AI referral traffic, a proxy.

A heuristic I use in projects, not a published standard: if Citation Rate stays below 5–10% after 90 days despite working crawlers and solid pages, invest in third-party mentions rather than more on-page rewrites. Also score what the model says about you: visibility that spreads false claims is a negative result.

The accessibility and indexation checks I start with in an SEO audit still come first. The protocol I document for a GEO audit adds: 20–50 real customer questions, repeated 10–20 times on ChatGPT, Gemini, Perplexity and AI Overviews.

Stage

What must be true

Typical failure

Crawl

Search bots allowed; facts in raw HTML

Training-bot block copied onto OAI-SearchBot; JS-only HTML; CDN “block AI bots” Chunk

Self-contained answer under the heading

The claim sits in the middle of a long page

Entity

One name, consistent facts, aligned sameAs

Conflicting contact details or authorship

Schema

Valid JSON-LD that mirrors visible text

Schema added in the hope it will create citations

Measure

Repeated prompts, per platform, with ranges

One ChatGPT screenshot treated as proof

This week: fix robots.txt and CDN bot policies, put a direct answer under real-question headings, and run those questions five times on two platforms, saving the cited URLs.

About the author. Adrian Grabski is an independent SEO specialist and GEO consultant based in Łódź, Poland, working in search since 2002. He runs SEO and GEO audits, consulting and in-house training, and founded Search AI Academy, a free community of more than 2,100 practitioners.

How LLM Search Engines Pick Sources: A Practical Retrieval Checklist for Brands was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

── more in #generative-engine-optimization 4 stories · sorted by recency
typemill.net · · #generative-engine-optimization
Second Brain
── more on @chatgpt 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-llm-search-engin…] indexed:0 read:7min 2026-11-11 · —