{"slug": "how-to-make-your-next-js-site-appear-in-chatgpt-and-any-llm", "title": "How to make your Next.js site appear in ChatGPT (and any LLM)", "summary": "A developer's guide explains how to configure Next.js sites for Generative Engine Optimization (GEO), distinguishing between training crawlers like GPTBot and search bots like OAI-SearchBot. The article details that blocking GPTBot does not prevent ChatGPT Search citations, and provides a robots.txt policy to allow search bots while disallowing training bots.", "body_md": "You blocked `GPTBot`\n\nin `robots.txt`\n\nto keep your content out of training runs — and then wondered why ChatGPT Search never cites your docs. Those are **different systems**. OpenAI’s own crawler docs say each bot is independent: allowing `OAI-SearchBot`\n\nkeeps you eligible for ChatGPT search answers while disallowing `GPTBot`\n\nopts you out of foundation-model training.\n\nThis article is a Next.js App Router playbook for **Generative Engine Optimization (GEO)**: how answer engines discover pages, which user-agents actually matter, how to configure `robots.ts`\n\nand sitemaps, how to keep HTML crawlable, and what `llms.txt`\n\ndoes — and does not — guarantee.\n\nTreat “appearing in an LLM” as three separate pipelines:\n\n| Pipeline | What it does | Typical bots / tokens |\n|---|---|---|\nTraining crawl |\nCollects public pages that may enter future model training |\n`GPTBot` , `ClaudeBot` , `Google-Extended` (token), Common Crawl’s `CCBot`\n|\nSearch / answer index |\nBuilds or refreshes retrieval so answers can cite your URLs |\n`OAI-SearchBot` , `Claude-SearchBot` , `PerplexityBot` , classic `Googlebot` / `Bingbot` (and partners) |\nUser-triggered fetch |\nDownloads a specific URL because a human asked for it (or pasted a link) |\n`ChatGPT-User` , `Claude-User` , `Perplexity-User`\n|\n\nBlocking the training bot does **not** automatically block the search bot. OpenAI states this explicitly for `GPTBot`\n\nvs `OAI-SearchBot`\n\n. Anthropic documents the same split for `ClaudeBot`\n\n, `Claude-SearchBot`\n\n, and `Claude-User`\n\n. Perplexity documents `PerplexityBot`\n\nfor search indexing and `Perplexity-User`\n\nfor live fetches.\n\nChatGPT Search can also partner with third-party search providers. OpenAI’s help center documents that rewritten queries may be sent to partners such as **Bing** (and others listed in that article). Independently, OpenAI recommends allowing ** OAI-SearchBot** if you want to appear in ChatGPT search answers. Practical implication: keep search bots allowed\n\nFrom [OpenAI’s crawler overview](https://developers.openai.com/api/docs/bots):\n\nUsed to surface websites in ChatGPT’s search features. Sites opted out of `OAI-SearchBot`\n\n**will not be shown in ChatGPT search answers**, though they can still appear as plain navigational links. OpenAI recommends allowing it in `robots.txt`\n\nand permitting its [published IP ranges](https://openai.com/searchbot.json). Changes can take about **24 hours** to propagate.\n\nCrawls content that may be used to train generative foundation models. Disallowing `GPTBot`\n\nsignals that content should not be used for that training. It does **not** control ChatGPT Search eligibility.\n\nUsed when ChatGPT or Custom GPTs fetch a page because of a user action. OpenAI notes it is **not** used for automatic web crawling, **not** used to decide Search inclusion, and that **robots.txt rules may not apply** because the fetch is user-initiated. Manage Search with `OAI-SearchBot`\n\n; treat `ChatGPT-User`\n\nas a separate live-fetch channel.\n\nOnly visits pages submitted as ads on ChatGPT; not used to train foundation models. Relevant if you run ChatGPT ads — ignore it for organic GEO.\n\nA common, defensible policy for content sites that want citations but not training:\n\n```\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: GPTBot\nDisallow: /\n```\n\n[Anthropic’s help center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (updated April 2026) defines three bots:\n\n`ClaudeBot`\n\n`Claude-SearchBot`\n\n`Claude-User`\n\nAnthropic honors `robots.txt`\n\n(including non-standard `Crawl-delay`\n\n) and warns that **IP-blocking alone is unreliable** because it can prevent the bot from reading your `robots.txt`\n\n.\n\n[Perplexity’s crawler docs](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) recommend allowing ** PerplexityBot** so your site can appear in Perplexity search results.\n\n`Perplexity-User`\n\n`perplexitybot.json`\n\n/ `perplexity-user.json`\n\n) — robots.txt alone is not enough when the edge drops the request.[ Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) is a\n\nImportant trade-off: for OpenAI you can allow search and disallow training separately. For Google, `Google-Extended`\n\ncovers **both** Gemini training and Gemini grounding. Allow it if you want Gemini apps to ground on your content; disallow it if you want to opt out of those Gemini uses (Search itself stays separate via `Googlebot`\n\n).\n\n`app/robots.ts`\n\nApp Router can generate `/robots.txt`\n\nfrom a typed file. Official docs: [robots.txt file convention](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots).\n\nExample that keeps **search/citation** bots allowed, optionally opts out of **training**, fences private routes, and advertises the sitemap:\n\n``` python\n// app/robots.ts\nimport type { MetadataRoute } from 'next'\n\nconst SITE = 'https://example.com'\n\nexport default function robots(): MetadataRoute.Robots {\n  return {\n    rules: [\n      {\n        userAgent: '*',\n        allow: '/',\n        disallow: ['/api/', '/admin/', '/drafts/'],\n      },\n      // ChatGPT Search + live fetch\n      { userAgent: 'OAI-SearchBot', allow: '/' },\n      { userAgent: 'ChatGPT-User', allow: '/' },\n      // Training opt-out (optional — remove this rule to allow training)\n      { userAgent: 'GPTBot', disallow: '/' },\n      // Claude\n      { userAgent: 'Claude-SearchBot', allow: '/' },\n      { userAgent: 'Claude-User', allow: '/' },\n      { userAgent: 'ClaudeBot', disallow: '/' },\n      // Perplexity\n      { userAgent: 'PerplexityBot', allow: '/' },\n      { userAgent: 'Perplexity-User', allow: '/' },\n      // Gemini grounding/training token (allow if you want Gemini apps to use you)\n      { userAgent: 'Google-Extended', allow: '/' },\n    ],\n    sitemap: `${SITE}/sitemap.xml`,\n    host: SITE,\n  }\n}\n```\n\nAfter deploy, open `https://your-domain/robots.txt`\n\nand confirm the groups look right. Spoofed user-agents exist — for verification, OpenAI and Perplexity publish IP range JSON files; Anthropic currently points publishers at robots.txt rather than relying on IP blocks.\n\nCrawlers need URLs to discover. Next.js can generate `/sitemap.xml`\n\nfrom `app/sitemap.ts`\n\n([docs](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap)):\n\n``` python\n// app/sitemap.ts\nimport type { MetadataRoute } from 'next'\n\nexport default async function sitemap(): Promise<MetadataRoute.Sitemap> {\n  const posts = await getPublishedPosts() // your data layer\n\n  return [\n    { url: 'https://example.com', lastModified: new Date(), priority: 1 },\n    ...posts.map((p) => ({\n      url: `https://example.com/blog/${p.slug}`,\n      lastModified: p.updatedAt ?? p.publishedAt,\n      changeFrequency: 'monthly' as const,\n      priority: 0.7,\n    })),\n  ]\n}\n```\n\nThen:\n\nIf ChatGPT Search partners with Bing for some queries, a healthy Bing index is still cheap insurance. A healthy Google index still matters for Google Search, AI Overviews / AI Mode, and anything that grounds on Google’s index.\n\nAnswer engines and classic crawlers are more reliable when the **important prose is in the first HTML response**. App Router Server Components and SSR help; shipping an empty shell that only fills after client JavaScript is a classic way to look invisible.\n\nPractical checklist for Next.js:\n\n`<title>`\n\n, meta description, and Open Graph tags via the Metadata API — they travel with the document.`alternates.languages`\n\nin the sitemap when you ship locales.`200`\n\nresponses for public pages — timeouts and soft 404s waste crawl budget.Structured data (JSON-LD for `Article`\n\n, `FAQPage`\n\n, `Organization`\n\n, etc.) does not replace good prose, but clear headings, short definitional paragraphs, tables, and FAQ sections match how Bing’s own GEO guidance describes content that is easier to cite accurately.\n\nTechnical access is necessary but not sufficient. Pages that get cited tend to:\n\n`##`\n\n/ `###`\n\nheadings that match how people ask questionsYou do not need a new CMS. You need pages that are **the best extractable answer** for a specific intent.\n\n`llms.txt`\n\n: useful map, not a ranking switch\nJeremy Howard’s [llms.txt proposal](https://llmstxt.org/) suggests a Markdown file at `/llms.txt`\n\n: site name as `#`\n\nheading, a short `>`\n\nsummary, then `##`\n\nsections with curated absolute links. Optional companion files (for example full concatenated context) exist in the ecosystem. Docs platforms (and many agent workflows) already use this pattern.\n\nWhat it is **not**: a formal standard enforced by OpenAI, Google, or Anthropic for citation ranking. It does not replace `robots.txt`\n\n. It cannot block crawlers. Treat it as a **curated table of contents** for agents and humans who fetch `/llms.txt`\n\non demand — especially documentation sites — not as a guaranteed ChatGPT ranking lever.\n\nIn Next.js you can start with `public/llms.txt`\n\n, or generate it from a Route Handler:\n\n``` js\n// app/llms.txt/route.ts\nexport function GET() {\n  const body = `# Acme Docs\n> Official documentation for the Acme API and SDKs.\n\n## Docs\n- [Quick start](https://example.com/docs/quickstart.md): Install and make your first request\n- [Auth](https://example.com/docs/auth.md): API keys and OAuth\n\n## Optional\n- [Changelog](https://example.com/changelog): Release history\n`\n\n  return new Response(body, {\n    headers: {\n      'Content-Type': 'text/plain; charset=utf-8',\n      'Cache-Control': 'public, max-age=3600',\n    },\n  })\n}\n```\n\nIf you also serve Markdown mirrors of key pages (the proposal’s `.md`\n\nsuffix idea), agents get cleaner context than parsing your marketing HTML.\n\n`GPTBot`\n\nand assuming you left ChatGPT Search`OAI-SearchBot`\n\nallowed for Search answers.`robots.txt`\n\n, hostile WAF`curl`\n\ndoes not show the answer text, many crawlers will not either.`llms.txt`\n\nas access control`Google-Extended`\n\nwhile expecting Gemini grounding**Invest in GEO when** you publish public expertise (docs, tutorials, comparisons, research) and want referral traffic or brand citation from ChatGPT, Claude, Perplexity, Copilot, or Gemini.\n\n**De-prioritize (or fully opt out) when** the product is private, paywalled, legally sensitive, or you deliberately do not want model/training or answer-engine reuse — then disallow the relevant bots and accept lower AI visibility.\n\nNo. Per OpenAI, Search visibility is governed by ** OAI-SearchBot**.\n\n`GPTBot`\n\nis the training-oriented crawler. You can disallow `GPTBot`\n\nand still allow `OAI-SearchBot`\n\n.OpenAI documents third-party search partners (including Bing) **and** its own `OAI-SearchBot`\n\n. Do both: allow `OAI-SearchBot`\n\nand keep a healthy presence in major indexes. Do not rely on a single secondary blog’s claim about exclusive Bing or Google dependency.\n\n`llms.txt`\n\nmake ChatGPT cite me?\nThere is no public commitment from major AI labs that `llms.txt`\n\ncontrols ChatGPT citation ranking. It is still worth shipping for docs and agent UX. Citations still depend on crawl access, indexability, and content quality.\n\nNot automatically. Decide per pipeline: search/citation vs training vs user fetch. Many content sites allow search bots, allow or disallow training consciously, and keep sensitive paths disallowed for everyone.\n\nNo — Server Components and the Metadata / sitemap / robots file conventions are well suited to crawlable HTML. Problems come from client-only rendering patterns, not from App Router itself.\n\nTo show up in ChatGPT Search and other LLM answers: **allow the search crawlers**, **do not let your WAF undo robots.txt**, **ship crawlable HTML from Next.js**, **submit sitemaps** (and IndexNow where it helps), and **write pages that answer specific questions clearly**. Optionally add `llms.txt`\n\nas a curated map. Optionally disallow training bots if that is your policy — without confusing them for search bots.\n\nShip the `OAI-SearchBot`\n\nallow rule this week if it is missing. Everything else compounds on top of being fetchable.", "url": "https://wpnews.pro/news/how-to-make-your-next-js-site-appear-in-chatgpt-and-any-llm", "canonical_source": "https://dev.to/gustavo_garcia_dev/how-to-make-your-nextjs-site-appear-in-chatgpt-and-any-llm-3mph", "published_at": "2026-08-03 02:11:43+00:00", "updated_at": "2026-08-03 02:39:17.445682+00:00", "lang": "en", "topics": ["generative-ai", "ai-products", "developer-tools", "large-language-models"], "entities": ["OpenAI", "Anthropic", "Perplexity", "Next.js", "GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot"], "alternates": {"html": "https://wpnews.pro/news/how-to-make-your-next-js-site-appear-in-chatgpt-and-any-llm", "markdown": "https://wpnews.pro/news/how-to-make-your-next-js-site-appear-in-chatgpt-and-any-llm.md", "text": "https://wpnews.pro/news/how-to-make-your-next-js-site-appear-in-chatgpt-and-any-llm.txt", "jsonld": "https://wpnews.pro/news/how-to-make-your-next-js-site-appear-in-chatgpt-and-any-llm.jsonld"}}