Make a Nuxt or Next.js site readable by AI: llms.txt, robots.txt, Markdown

A developer added AI-crawler support to the SEO plugin of Easy CMS, an embedded CMS for Nuxt and Next.js, generating robots.txt rules grouped by crawler purpose (training, search, user), an /llms.txt Markdown index plus /llms-full.txt, and per-post .md routes. The robotsTxt() helper defaults to allowing all AI crawlers and repeats base admin and API disallow rules in every user-agent group, since crawlers follow only the most specific matching group. The developer notes that crawler user-agent names change frequently and should not be copied from blog posts.

Ask an AI assistant about a page on your site and one of three things happens. It answers from what it learned in training, it answers from a search index it built earlier, or it fetches your page right now. In all three, a crawler read your HTML at some point. What it was allowed to read, and how easily, is the part you can shape. I built four pieces for that into the SEO plugin of Easy CMS, my embedded CMS for Nuxt and Next.js the first post in this series https://easycms.dev/blog/payload-style-cms-nuxt introduces it . The ideas work in any server-rendered site: robots.txt rules for AI crawlers, by what they do. /llms.txt , a short Markdown index of the site, and /llms-full.txt , the whole site in one file. /posts/hello.md . "AI crawler" covers three jobs: | Group | What it does | If you block it | |---|---|---| | training | Collects text to train models. | Your text stays out of future training sets. | | search | Indexes pages for AI search and for answers that cite and link them. | Your pages stop showing up as cited sources. | | user | Opens a page because a person asked an assistant to read it. | Assistants can't fetch your page on request. Some vendors say these may not always follow robots.txt. | Each company publishes its own user-agent names, and they change, so don't copy a list from a blog post. Easy CMS keeps them in one exported constant, AI CRAWLERS , grouped as above and updated in each release. The AI search guide https://easycms.dev/docs/ai-search shows the current list. A common choice: stay out of training, stay visible in AI answers. js // app/robots.txt/route.ts Next.js import { robotsTxt } from '@easy-cms/plugin-seo' import config from '@/easy-cms.config' export function GET { // AI training crawlers stay out; AI search and assistants may still read and cite posts. const text = robotsTxt { config, ai: { training: false } } return new Response text, { headers: { 'content-type': 'text/plain; charset=utf-8' } } } The output, shortened: User-agent: Allow: /api/cms/media/file/ Disallow: /admin/ Disallow: /api/cms/ User-agent: <one line for each crawler in the training group … Disallow: / Sitemap: https://example.com/sitemap.xml Two choices need explaining. Everything is allowed by default. If an upgrade started blocking a group, sites would quietly drop out of AI answers. Blocking is the site owner's business decision, so it's opt-in. Each group repeats the base rules. A crawler follows only the most specific group that names it, without merging in User-agent: . Add a group for one bot to disallow /search , and that bot may now crawl your admin and API. robotsTxt adds the admin and API rules to every group it writes, including the ones you add with rules: { userAgent: 'SomeBot', disallow: '/' } . If you write robots.txt by hand, copy your base rules into each group. llms.txt https://llmstxt.org is a proposed standard: a Markdown file at the root that says what the site is and where its main pages are. Here's what Easy CMS generates: My Blog A blog about building websites in Thailand. Posts - Hello https://example.com/posts/hello.md : What this blog is about. Pages - My Blog https://example.com/ You configure it in the plugin and serve it from a route: // easy-cms.config.ts excerpt seoPlugin { collections: 'posts' , generateURL: { doc } = /posts/${doc.slug} , llms: { description: 'A blog about building websites in Thailand.', limit: 100, // newest pages per collection the default markdownURL: { doc, collection } = collection === 'posts' && doc.slug ? /posts/${doc.slug}.md : null, }, } js // server/routes/llms.txt.ts Nuxt import { llmsTxt } from '@easy-cms/plugin-seo' export default defineEventHandler async event = { setHeader event, 'content-type', 'text/markdown; charset=utf-8' return llmsTxt await useEasyCMS } Links are made absolute from a { siteURL } you pass, or else from admin.siteURL in the config, or serverURL when admin.siteURL is empty. The choices behind it: hreflang , so every language would just repeat each page. It uses the default locale, or llms.locale . markdownURL . generateURL stays the address of the HTML page, because the sitemap and the search preview depend on it. llmsFullTxt cms is for tools that want the whole site in one request: every page's Markdown, newest first. It stops at about 5 MB maxBytes and ends with a line pointing to /sitemap.xml for the rest. HTML wraps the text in navigation, footers and scripts. Markdown is just the text. docMarkdown writes one document as Markdown: title, description, URL, published and updated dates, then the body. The body is rich text and textarea fields at the top level, plus any text inside blocks and arrays. Top-level text fields are skipped: they're usually a slug or a name, not content. If that guess is wrong for a collection, pass your own function in the plugin's markdown option. In Next.js, a dynamic segment can't carry a .md suffix, so a rewrite maps the URL onto a route: js // next.config.ts rewrites: async = { source: '/posts/:slug.md', destination: '/md/posts/:slug' } , js // app/md/posts/ slug /route.ts import { getEasyCMS } from '@easy-cms/next' import { docMarkdown } from '@easy-cms/plugin-seo' import config from '@/easy-cms.config' export const dynamic = 'force-dynamic' export async function GET request: Request, { params }: { params: Promise<{ slug: string } } { const { slug } = await params const cms = await getEasyCMS config // Read as a visitor: published posts only. const { docs } = await cms.find 'posts', { where: { slug: { equals: decodeURIComponent slug } }, limit: 1, overrideAccess: false, user: null, } const post = docs 0 if post || post.meta.noindex return new Response 'Not found', { status: 404 } // The address on the site from the config admin.siteURL , not from the request's Host. const url = ${cms.config.admin.siteURL.replace /\/+$/, '' }/posts/${post.slug} return new Response docMarkdown cms, { collection: 'posts', doc: post, url } , { headers: { 'content-type': 'text/markdown; charset=utf-8' }, } } In Nuxt, a server middleware matches /^\/posts\/ ^/ + \.md$/ and does the same. Three details matter: read with overrideAccess: false, user: null , so only what a visitor may see comes out. Respect noindex : a page you hid from search shouldn't reappear as Markdown. And take the address from the config, not the request: a client can send any Host header, and a cached response would pass that address on to every reader. Why not Accept: text/markdown on the normal URL? One URL with two formats looks cleaner, but many caches key on the URL alone. Unless every layer between you and the reader honors Vary: Accept , the first Markdown response gets cached and served to the next browser, or the reverse. A separate .md URL can't be mixed up. Crawlers come back on their own schedule. IndexNow https://www.indexnow.org lets you tell Bing, Yandex and the other engines that take part about a changed URL right away. Bing's index also feeds ChatGPT search and Copilot, which is why I put it in this plugin. seoPlugin { // … indexNow: { key: process.env.INDEXNOW KEY }, // 8–128 letters, digits or dashes, e.g. a UUID } Each part of the behavior has a reason: previousDoc that afterChange hooks get is that draft. So the plugin reads the published state in beforeChange and compares after the save. That costs one extra read per edit, and only when IndexNow is on. delay , then sent as one request per host, up to 10,000 URLs each. Publishing 40 posts in one go means one request, not 40. https URLs only. localhost , IP addresses, hostnames without a dot, and .test , .local , .example , .invalid and .internal domains are skipped. Dev servers and test runs stay quiet with no config. A staging site on a real public domain INDEXNOW KEY unset there. IndexNow checks ownership with a key file at /<key .txt , and accepts only URLs under that file's location, so it must sit at the root. The examples answer it with indexNowKeyFile cms, pathname : a catch-all app/ file /route.ts in Next.js, a middleware in Nuxt. A plain file in public/ works too. One trade-off I accepted: the 5-second queue lives in memory. On serverless, if the function stops before it flushes, those URLs are lost. Set a shorter | Piece | What to do | The trap | |---|---|---| | robots.txt | One group per crawler category; the names come from each vendor's docs. | A named group ignores , so repeat your base rules in it. | | llms.txt | Title, one-line summary, newest pages per section, one locale. | Listing every URL and every language. | | Markdown pages | A separate .md URL, read as an anonymous visitor. | Content negotiation behind a cache; leaking drafts or noindex pages. | | IndexNow | Ping on visitor-visible changes, batched, public https only. | Pinging on every draft save, or from staging. | llms.txt is a proposal, not a standard every assistant reads. I can't tell you which ones use it today, or whether any of this changes where you rank. What I can say: each file costs one route, and none exposes anything a visitor couldn't already read. TODO: your story — one line on what you saw e.g. which crawlers fetched /llms.txt or .md pages after turning this on for a real site, or delete this line If you want the reference, the AI search guide https://easycms.dev/docs/ai-search has the Nuxt and Next.js versions of every route, and examples/next-blog and examples/nuxt-blog in the repository run them all. Easy CMS is MIT-licensed: github.com/maritonx/easy-cms https://github.com/maritonx/easy-cms · docs https://easycms.dev/docs . If you try it, tell me what got in your way in the feedback issue https://github.com/maritonx/easy-cms/issues/137 .