{"slug": "make-a-nuxt-or-next-js-site-readable-by-ai-llms-txt-robots-txt-markdown", "title": "Make a Nuxt or Next.js site readable by AI: llms.txt, robots.txt, Markdown", "summary": "A developer added AI-crawler support to the SEO plugin of Easy CMS, an embedded CMS for Nuxt and Next.js, generating robots.txt rules grouped by crawler purpose (training, search, user), an /llms.txt Markdown index plus /llms-full.txt, and per-post .md routes. The robotsTxt() helper defaults to allowing all AI crawlers and repeats base admin and API disallow rules in every user-agent group, since crawlers follow only the most specific matching group. The developer notes that crawler user-agent names change frequently and should not be copied from blog posts.", "body_md": "Ask an AI assistant about a page on your site and one of three things happens. It answers from what it learned in training, it answers from a search index it built earlier, or it fetches your page right now. In all three, a crawler read your HTML at some point. What it was allowed to read, and how easily, is the part you can shape.\n\nI built four pieces for that into the SEO plugin of Easy CMS, my embedded CMS for Nuxt and Next.js ([the first post in this series](https://easycms.dev/blog/payload-style-cms-nuxt) introduces it). The ideas work in any server-rendered site:\n\n`robots.txt` rules for AI crawlers, by what they do.`/llms.txt`, a short Markdown index of the site, and `/llms-full.txt`, the whole site in one file.`/posts/hello.md`.\n\"AI crawler\" covers three jobs:\n\n| Group | What it does | If you block it | \n|---|---|---|\n| `training` | Collects text to train models. | Your text stays out of future training sets. | \n| `search` | Indexes pages for AI search and for answers that cite and link them. | Your pages stop showing up as cited sources. | \n| `user` | Opens a page because a person asked an assistant to read it. | Assistants can't fetch your page on request. Some vendors say these may not always follow robots.txt. | \n\nEach company publishes its own user-agent names, and they change, so don't copy a list from a blog post. Easy CMS keeps them in one exported constant, `AI_CRAWLERS`, grouped as above and updated in each release. The [AI search guide](https://easycms.dev/docs/ai-search) shows the current list.\n\nA common choice: stay out of training, stay visible in AI answers.\n\n``` js\n// app/robots.txt/route.ts (Next.js)\nimport { robotsTxt } from '@easy-cms/plugin-seo'\nimport config from '@/easy-cms.config'\nexport function GET() {\n  // AI training crawlers stay out; AI search and assistants may still read and cite posts.\n  const text = robotsTxt({ config, ai: { training: false } })\n  return new Response(text, { headers: { 'content-type': 'text/plain; charset=utf-8' } })\n}\n```\n\nThe output, shortened:\n\n```\nUser-agent: *\nAllow: /api/cms/media/file/\nDisallow: /admin/\nDisallow: /api/cms/\nUser-agent: <one line for each crawler in the training group>\n…\nDisallow: /\nSitemap: https://example.com/sitemap.xml\n```\n\nTwo choices need explaining.\n\n**Everything is allowed by default.** If an upgrade started blocking a group, sites would quietly drop out of AI answers. Blocking is the site owner's business decision, so it's opt-in.\n\n**Each group repeats the base rules.** A crawler follows only the most specific group that names it, without merging in `User-agent: *`. Add a group for one bot to disallow `/search`, and that bot may now crawl your admin and API. `robotsTxt()` adds the admin and API rules to every group it writes, including the ones you add with `rules: [{ userAgent: 'SomeBot', disallow: ['/'] }]`. If you write robots.txt by hand, copy your base rules into each group.\n\n[llms.txt](https://llmstxt.org) is a proposed standard: a Markdown file at the root that says what the site is and where its main pages are. Here's what Easy CMS generates:\n\n```\n# My Blog\n> A blog about building websites in Thailand.\n\n## Posts\n- [Hello](https://example.com/posts/hello.md): What this blog is about.\n\n## Pages\n- [My Blog](https://example.com/)\n```\n\nYou configure it in the plugin and serve it from a route:\n\n```\n// easy-cms.config.ts (excerpt)\nseoPlugin({\n  collections: ['posts'],\n  generateURL: ({ doc }) => `/posts/${doc.slug}`,\n  llms: {\n    description: 'A blog about building websites in Thailand.',\n    limit: 100, // newest pages per collection (the default)\n    markdownURL: ({ doc, collection }) =>\n      collection === 'posts' && doc.slug ? `/posts/${doc.slug}.md` : null,\n  },\n})\njs\n// server/routes/llms.txt.ts (Nuxt)\nimport { llmsTxt } from '@easy-cms/plugin-seo'\nexport default defineEventHandler(async (event) => {\n  setHeader(event, 'content-type', 'text/markdown; charset=utf-8')\n  return llmsTxt(await useEasyCMS())\n})\n```\n\nLinks are made absolute from a `{ siteURL }` you pass, or else from `admin.siteURL` in the config, or `serverURL` when `admin.siteURL` is empty.\n\nThe choices behind it:\n\n`hreflang`, so every language would just repeat each page. It uses the default locale, or `llms.locale`.` markdownURL`.` generateURL` stays the address of the HTML page, because the sitemap and the search preview depend on it.\n`llmsFullTxt(cms)`) is for tools that want the whole site in one request: every page's Markdown, newest first. It stops at about 5 MB (`maxBytes`) and ends with a line pointing to `/sitemap.xml` for the rest.\nHTML wraps the text in navigation, footers and scripts. Markdown is just the text. `docMarkdown()` writes one document as Markdown: title, description, URL, published and updated dates, then the body.\n\nThe body is rich text and `textarea` fields at the top level, plus any text inside blocks and arrays. Top-level `text` fields are skipped: they're usually a slug or a name, not content. If that guess is wrong for a collection, pass your own function in the plugin's `markdown` option.\n\nIn Next.js, a dynamic segment can't carry a `.md` suffix, so a rewrite maps the URL onto a route:\n\n``` js\n// next.config.ts\nrewrites: async () => [{ source: '/posts/:slug.md', destination: '/md/posts/:slug' }],\njs\n// app/md/posts/[slug]/route.ts\nimport { getEasyCMS } from '@easy-cms/next'\nimport { docMarkdown } from '@easy-cms/plugin-seo'\nimport config from '@/easy-cms.config'\nexport const dynamic = 'force-dynamic'\nexport async function GET(_request: Request, { params }: { params: Promise<{ slug: string }> }) {\n  const { slug } = await params\n  const cms = await getEasyCMS(config)\n  // Read as a visitor: published posts only.\n  const { docs } = await cms.find('posts', {\n    where: { slug: { equals: decodeURIComponent(slug) } },\n    limit: 1,\n    overrideAccess: false,\n    user: null,\n  })\n  const post = docs[0]\n  if (!post || post.meta.noindex) return new Response('Not found', { status: 404 })\n  // The address on the site from the config (admin.siteURL), not from the request's Host.\n  const url = `${cms.config.admin.siteURL.replace(/\\/+$/, '')}/posts/${post.slug}`\n  return new Response(docMarkdown(cms, { collection: 'posts', doc: post, url }), {\n    headers: { 'content-type': 'text/markdown; charset=utf-8' },\n  })\n}\n```\n\nIn Nuxt, a server middleware matches `/^\\/posts\\/([^/]+)\\.md$/` and does the same. Three details matter: read with `overrideAccess: false, user: null`, so only what a visitor may see comes out. Respect `noindex`: a page you hid from search shouldn't reappear as Markdown. And take the address from the config, not the request: a client can send any `Host` header, and a cached response would pass that address on to every reader.\n\n**Why not `Accept: text/markdown` on the normal URL?** One URL with two formats looks cleaner, but many caches key on the URL alone. Unless every layer between you and the reader honors `Vary: Accept`, the first Markdown response gets cached and served to the next browser, or the reverse. A separate `.md` URL can't be mixed up.\n\nCrawlers come back on their own schedule. [IndexNow](https://www.indexnow.org) lets you tell Bing, Yandex and the other engines that take part about a changed URL right away. Bing's index also feeds ChatGPT search and Copilot, which is why I put it in this plugin.\n\n```\nseoPlugin({\n  // …\n  indexNow: { key: process.env.INDEXNOW_KEY }, // 8–128 letters, digits or dashes, e.g. a UUID\n})\n```\n\nEach part of the behavior has a reason:\n\n`previousDoc` that `afterChange` hooks get is that draft. So the plugin reads the published state in `beforeChange` and compares after the save. That costs one extra read per edit, and only when IndexNow is on.`delay`), then sent as one request per host, up to 10,000 URLs each. Publishing 40 posts in one go means one request, not 40.`https` URLs only.`localhost`, IP addresses, hostnames without a dot, and `.test`, `.local`, `.example`, `.invalid` and `.internal` domains are skipped. Dev servers and test runs stay quiet with no config. A staging site on a real public domain `INDEXNOW_KEY` unset there.\nIndexNow checks ownership with a key file at `/<key>.txt`, and accepts only URLs under that file's location, so it must sit at the root. The examples answer it with `indexNowKeyFile(cms, pathname)`: a catch-all `app/[file]/route.ts` in Next.js, a middleware in Nuxt. A plain file in `public/` works too.\nOne trade-off I accepted: the 5-second queue lives in memory. On serverless, if the function stops before it flushes, those URLs are lost. Set a shorter | Piece | What to do | The trap | \n|---|---|---|\n| robots.txt | One group per crawler category; the names come from each vendor's docs. | A named group ignores `*` , so repeat your base rules in it. | \n| llms.txt | Title, one-line summary, newest pages per section, one locale. | Listing every URL and every language. | \n| Markdown pages | A separate `.md` URL, read as an anonymous visitor. | Content negotiation behind a cache; leaking drafts or noindex pages. | \n| IndexNow | Ping on visitor-visible changes, batched, public https only. | Pinging on every draft save, or from staging. | \n\nllms.txt is a proposal, not a standard every assistant reads. I can't tell you which ones use it today, or whether any of this changes where you rank. What I can say: each file costs one route, and none exposes anything a visitor couldn't already read.\n\n[TODO: your story — one line on what you saw (e.g. which crawlers fetched /llms.txt or .md pages) after turning this on for a real site, or delete this line]\n\nIf you want the reference, the [AI search guide](https://easycms.dev/docs/ai-search) has the Nuxt and Next.js versions of every route, and `examples/next-blog` and `examples/nuxt-blog` in the repository run them all.\n\nEasy CMS is MIT-licensed: [github.com/maritonx/easy-cms](https://github.com/maritonx/easy-cms) · [docs](https://easycms.dev/docs). If you try it, tell me what got in your way in the [feedback issue](https://github.com/maritonx/easy-cms/issues/137).", "url": "https://wpnews.pro/news/make-a-nuxt-or-next-js-site-readable-by-ai-llms-txt-robots-txt-markdown", "canonical_source": "https://dev.to/kanawoot_korsang_5be7c94a/make-a-nuxt-or-nextjs-site-readable-by-ai-llmstxt-robotstxt-markdown-1m27", "published_at": "2026-11-14 06:12:37+00:00", "updated_at": "2026-11-14 06:21:14.971728+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "generative-engine-optimization", "ai-search", "developer-tools"], "entities": ["Easy CMS", "Nuxt", "Next.js", "@easy-cms/plugin-seo"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/make-a-nuxt-or-next-js-site-readable-by-ai-llms-txt-robots-txt-markdown", "markdown": "https://wpnews.pro/news/make-a-nuxt-or-next-js-site-readable-by-ai-llms-txt-robots-txt-markdown.md", "text": "https://wpnews.pro/news/make-a-nuxt-or-next-js-site-readable-by-ai-llms-txt-robots-txt-markdown.txt", "jsonld": "https://wpnews.pro/news/make-a-nuxt-or-next-js-site-readable-by-ai-llms-txt-robots-txt-markdown.jsonld"}}