I shipped llms.txt, JSON-LD and AI crawler allowances. Here's what each one actually does.

A developer building an answer engine optimization site for SearchD implemented llms.txt, JSON-LD, and AI crawler allowances, then tested whether each actually delivers the promised citation benefits. Testing 12 AI crawler user agents against the site returned HTTP 200 across the board, but the developer found that robots.txt allowances prove nothing about edge-level bot blocking, that Google states it ignores llms.txt, and that an Ahrefs study of 1,885 pages found no citation lift from schema markup.

I am building a site whose entire job is to be quoted by AI assistants. It is the marketing site for SearchD https://searchd.ai , an answer engine optimization practice, which makes the site both the product and the test case. Three things get recommended everywhere for this: let the AI crawlers in, publish an llms.txt , ship JSON-LD. I implemented all three. Then I went looking for evidence that each one does what the advice says it does. Only one of the three can fail silently. The other two turned out to do a different job from the one the advice promises. A robots.txt that allows GPTBot proves nothing. The file is a request to a well-behaved crawler; it has no effect on the layer in front of it. If your CDN has a bot-management rule, the crawler can be challenged or blocked at the edge and never get as far as reading your robots file. The only way to know is to ask as each crawler and look at the status code: for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User PerplexityBot \ Google-Extended Applebot-Extended CCBot Amazonbot Bingbot Googlebot; do printf "%-20s %s\n" "$ua" "$ curl -s -o /dev/null -w '%{http code}' -A "$ua" https://example.com/ " done Run against my own site on 11 September 2026: GPTBot 200 OAI-SearchBot 200 ChatGPT-User 200 ClaudeBot 200 Claude-User 200 PerplexityBot 200 Google-Extended 200 Applebot-Extended 200 CCBot 200 Amazonbot 200 Bingbot 200 Googlebot 200 A 403 or a 503 here outranks any amount of markup work, because it means nothing downstream of it can matter. Passing the check does not get you cited. It removes a way of being invisible that you would otherwise never see. The status code leaves two things open. It says nothing about whether the response body is a JavaScript shell, so fetch one and read it: a 200 carrying an empty <div id="root" is a 200 that says nothing. It also says nothing about whether the crawler that matters is on your list. The names change. Claude-SearchBot and Perplexity-User were both added to mine after the first pass. llms.txt is a markdown index of your site at /llms.txt , meant to give a model a clean map instead of making it parse your nav. Generating it is cheap if your content is already in a collection. This is an Astro endpoint, about forty lines: python // src/pages/llms.txt.ts import type { APIRoute } from 'astro'; import { getCollection } from 'astro:content'; import { SITE } from '../config'; const line = title: string, path: string, note: string = - ${title} ${SITE.domain}${path} : ${note} ; export const GET: APIRoute = async = { const answers = await getCollection 'answers' ; const glossary = await getCollection 'glossary' ; const body = ${SITE.name} ${SITE.description} Answers ${answers.map a = line a.data.question, /answers/${a.id} , a.data.description .join '\n' } Glossary ${glossary.map t = line t.data.term, /glossary/${t.id} , t.data.definition .join '\n' } ; return new Response body, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } } ; }; Google states plainly https://developers.google.com/search/docs/fundamentals/ai-optimization-guide that Search ignores these files, and that having one neither helps nor harms. No major engine has published a commitment to read it. I keep mine. It is generated, so it costs nothing to maintain, and it makes a useful index when I point an agent at my own site. Nobody has shown it does harm either. None of that amounts to a model reading it, and if you are writing a plan for someone else, llms.txt does not belong in the column marked citations . This is the one I expected to defend and could not. Ahrefs ran a controlled study across 1,885 pages May 2026 https://ahrefs.com/blog/schema-ai-citations/ and after matched controls found no citation lift from schema markup: +2.4% on AI Mode, +2.2% on ChatGPT, and −4.6% on AI Overviews. Those sit inside the noise. If you are adding schema to get quoted more often, the published evidence does not support you. Schema still earns its place, for a different job. It is how you tell a machine that the "Liam Hwang" on the byline, the one on the author page and the one in the organization's founder field are one person rather than three. Get that wrong and you have manufactured the entity confusion you were trying to prevent. The rule that makes it work is one @graph per page, with @id values that are stable URIs rather than page-local strings: js // src/lib/schema.ts const id = path: string = ${SITE.domain}${path} ; export const ORG ID = id '/ organization' ; export const authorPath = slug: string = /authors/${slug} ; export const authorId = slug: string = id ${authorPath slug } person ; export const person = a: Author = { '@type': 'Person' as const, '@id': authorId a.slug , // the same URI on every page that names them name: a.name, url: id authorPath a.slug , // Writing for the site is not an employment claim. ... a.slug === FOUNDER.slug ? { worksFor: { '@id': ORG ID } } : {} , } ; A person's canonical URI is their author page, including the founder's. An id derived from a role /about founder moves the day the role does, and it makes the founder a different kind of thing from everyone else who writes there. Every page then references the node instead of redeclaring it author: { '@id': authorId slug } , founder: { '@id': authorId FOUNDER.slug } , and the layout emits one <script type="application/ld+json" carrying the whole graph. Getting this wrong produces two bugs. Emitting a second Person node for someone who already has one is the split you were trying to prevent, so the helper that adds a post's author filters out the founder, whose node the layout already includes. Describing something in schema that is not visible on the page has the same shape: the markup and the rendered page have to agree, or you have told two stories about one object. All three are shipped on my site. None of them is the reason a page gets retrieved and quoted. Crawler access, llms.txt and JSON-LD are upstream plumbing, and the binding constraint sits earlier than any of them. A page that is not indexed cannot be retrieved by anything that searches, however clean its markup is. Google Search Console reports that in one screen, as a count of submitted URLs against indexed ones, and on a young domain the gap between those two numbers will tell you more than any markup audit. So the order is: run the crawler check, because it is the only one of the three that can cost you everything without saying so. Then read your index coverage, and keep reading it. Do the other two because they are cheap and correct, and file them under plumbing rather than citations.