{"slug": "i-shipped-llms-txt-json-ld-and-ai-crawler-allowances-here-s-what-each-one-does", "title": "I shipped llms.txt, JSON-LD and AI crawler allowances. Here's what each one actually does.", "summary": "A developer building an answer engine optimization site for SearchD implemented llms.txt, JSON-LD, and AI crawler allowances, then tested whether each actually delivers the promised citation benefits. Testing 12 AI crawler user agents against the site returned HTTP 200 across the board, but the developer found that robots.txt allowances prove nothing about edge-level bot blocking, that Google states it ignores llms.txt, and that an Ahrefs study of 1,885 pages found no citation lift from schema markup.", "body_md": "I am building a site whose entire job is to be quoted by AI assistants. It is the marketing site for [SearchD](https://searchd.ai), an answer engine optimization practice, which makes the site both the product and the test case.\n\nThree things get recommended everywhere for this: let the AI crawlers in, publish an `llms.txt`, ship JSON-LD. I implemented all three. Then I went looking for evidence that each one does what the advice says it does.\n\nOnly one of the three can fail silently. The other two turned out to do a different job from the one the advice promises.\n\nA `robots.txt` that allows `GPTBot` proves nothing. The file is a request to a well-behaved crawler; it has no effect on the layer in front of it. If your CDN has a bot-management rule, the crawler can be challenged or blocked at the edge and never get as far as reading your robots file.\n\nThe only way to know is to ask as each crawler and look at the status code:\n\n```\nfor ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User PerplexityBot \\\n Google-Extended Applebot-Extended CCBot Amazonbot Bingbot Googlebot; do\n printf \"%-20s %s\\n\" \"$ua\" \"$(curl -s -o /dev/null -w '%{http_code}' -A \"$ua\" https://example.com/)\"\ndone\n```\n\nRun against my own site on 11 September 2026:\n\n```\nGPTBot 200\nOAI-SearchBot 200\nChatGPT-User 200\nClaudeBot 200\nClaude-User 200\nPerplexityBot 200\nGoogle-Extended 200\nApplebot-Extended 200\nCCBot 200\nAmazonbot 200\nBingbot 200\nGooglebot 200\n```\n\nA `403` or a `503` here outranks any amount of markup work, because it means nothing downstream of it can matter. Passing the check does not get you cited. It removes a way of being invisible that you would otherwise never see.\n\nThe status code leaves two things open. It says nothing about whether the response body is a JavaScript shell, so fetch one and read it: a 200 carrying an empty `
` is a 200 that says nothing. It also says nothing about whether the crawler that matters is on your list. The names change. `Claude-SearchBot` and `Perplexity-User` were both added to mine after the first pass.\n\n`llms.txt` is a markdown index of your site at `/llms.txt`, meant to give a model a clean map instead of making it parse your nav. Generating it is cheap if your content is already in a collection. This is an Astro endpoint, about forty lines:\n\n``` python\n// src/pages/llms.txt.ts\nimport type { APIRoute } from 'astro';\nimport { getCollection } from 'astro:content';\nimport { SITE } from '../config';\n\nconst line = (title: string, path: string, note: string) =>\n `- [${title}](${SITE.domain}${path}): ${note}`;\n\nexport const GET: APIRoute = async () => {\n const answers = await getCollection('answers');\n const glossary = await getCollection('glossary');\n\n const body = `# ${SITE.name}\n\n> ${SITE.description}\n\n## Answers\n${answers.map((a) => line(a.data.question, `/answers/${a.id}`, a.data.description)).join('\\n')}\n\n## Glossary\n${glossary.map((t) => line(t.data.term, `/glossary/${t.id}`, t.data.definition)).join('\\n')}\n`;\n\n return new Response(body, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } });\n};\n```\n\nGoogle [states plainly](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) that Search ignores these files, and that having one neither helps nor harms. No major engine has published a commitment to read it.\n\nI keep mine. It is generated, so it costs nothing to maintain, and it makes a useful index when I point an agent at my own site. Nobody has shown it does harm either. None of that amounts to a model reading it, and if you are writing a plan for someone else, llms.txt does not belong in the column marked *citations*.\n\nThis is the one I expected to defend and could not.\n\nAhrefs ran a controlled study across 1,885 pages ([May 2026](https://ahrefs.com/blog/schema-ai-citations/)) and after matched controls found no citation lift from schema markup: +2.4% on AI Mode, +2.2% on ChatGPT, and −4.6% on AI Overviews. Those sit inside the noise. If you are adding schema to get quoted more often, the published evidence does not support you.\n\nSchema still earns its place, for a different job. It is how you tell a machine that the \"Liam Hwang\" on the byline, the one on the author page and the one in the organization's `founder` field are one person rather than three. Get that wrong and you have manufactured the entity confusion you were trying to prevent.\n\nThe rule that makes it work is one `@graph` per page, with `@id` values that are stable URIs rather than page-local strings:\n\n``` js\n// src/lib/schema.ts\nconst id = (path: string) => `${SITE.domain}${path}`;\n\nexport const ORG_ID = id('/#organization');\nexport const authorPath = (slug: string) => `/authors/${slug}`;\nexport const authorId = (slug: string) => id(`${authorPath(slug)}#person`);\n\nexport const person = (a: Author) => ({\n '@type': 'Person' as const,\n '@id': authorId(a.slug), // the same URI on every page that names them\n name: a.name,\n url: id(authorPath(a.slug)),\n // Writing for the site is not an employment claim.\n ...(a.slug === FOUNDER.slug ? { worksFor: { '@id': ORG_ID } } : {}),\n});\n```\n\nA person's canonical URI is their author page, including the founder's. An id derived from a role (`/about#founder`) moves the day the role does, and it makes the founder a different kind of thing from everyone else who writes there.\n\nEvery page then references the node instead of redeclaring it (`author: { '@id': authorId(slug) }`, `founder: { '@id': authorId(FOUNDER.slug) }`), and the layout emits one `