# What Is llms.txt? How to Set It Up on WordPress (2026 Guide) > Source: > Published: 2026-09-20 18:19:03+00:00 In September 2024, Jeremy Howard — the guy behind fast.ai and a former president of Kaggle — published a short proposal. He called it llms.txt, and the idea was almost annoyingly simple: give AI models a plain-text map of your website so they don't have to guess what's important. No one paid much attention at first. Then Cloudflare added one. Then Stripe. Then Anthropic, Vercel, Supabase, and Zapier. Today, over 780 websites have published an llms.txt file, and three separate community directories track who's adopted it. The question isn't really *whether* llms.txt will matter. It's whether you'll set yours up before or after your competitors do. Think of your website from an AI model's perspective for a second. When ChatGPT or Perplexity needs to answer a question about your product, it sends a crawler to your site. That crawler sees HTML — headers, footers, navigation menus, cookie banners, newsletter popups, sidebar widgets. Buried somewhere in that noise is the content that actually matters. Your XML sitemap doesn't help much here. It lists every URL on the site equally — the homepage, the privacy policy, that test page from 2019 you forgot to delete. There's no hierarchy, no context, no "start here." llms.txt solves this by giving AI models exactly what they need: a structured Markdown file at `yoursite.com/llms.txt` that says "here's what this site is about, and here are the pages that matter most." That's it. A curated table of contents for machines that read. An llms.txt file is just Markdown with a specific structure: ``` # Your Site Name > A one-line summary of what your site does. Some optional context about your project, company, or product. ## Documentation - [Getting Started](https://yoursite.com/docs/start): Setup guide for new users - [API Reference](https://yoursite.com/docs/api): Complete API documentation ## Blog - [Latest Release](https://yoursite.com/blog/v2): What's new in version 2.0 ``` The rules are minimal: `[Page Name](URL): optional description` That's the entire spec. Jeremy Howard intentionally kept it simple because "websites already have sitemaps, robots.txt, and meta tags. What they don't have is a curated overview written specifically for language models." The standard actually defines two files: **llms.txt** is the summary — your site's elevator pitch plus links to key pages. Think of it as the table of contents. Most llms.txt files are a few hundred lines at most. **llms-full.txt** is the entire book. It contains the full Markdown content of your key pages, concatenated into one massive file. Cloudflare's llms-full.txt is 3.7 million tokens. Anthropic's is 481,000 tokens. Vercel's has been described as "a 400,000-word novel." Why would you want that? Because an AI model with a large enough context window can ingest your entire documentation in one shot — no crawling, no parsing HTML, no missing pages. For most WordPress sites, llms.txt alone is enough. But if you run a documentation-heavy site or want AI to have deep access to your content, llms-full.txt is worth considering. The short answer: mostly developer-facing companies. The long answer is more interesting. **Stripe** organizes their llms.txt by product category, with an "Optional" section for niche tools like Stripe Climate. Their file tells AI agents exactly how to retrieve plain-text versions of documentation pages. **Cloudflare** structures theirs by product line — Workers, Pages, R2, and so on. Each entry has a description that helps AI models decide which page to fetch based on the user's question. **Anthropic** (the company behind Claude) maintains both files. Their llms.txt is 8,364 tokens; the full version is 481,349 tokens covering their entire API documentation. **Vercel** went a step further and published a proposal for inline LLM instructions in HTML — extending the llms.txt idea into individual pages. Other adopters include Supabase, Zapier, Modal, Coinbase, and hundreds of smaller companies tracked across three community directories. Here's where most articles about llms.txt get vague. We won't. **The data is mixed.** Search Engine Land tracked 10 websites for 180 days — 90 days before adding llms.txt, 90 days after. The results: But here's the catch: the two sites that grew had also launched new content and PR campaigns around the same time. The llms.txt file alone didn't drive the increase — it was part of a broader effort. **Adoption is still early.** Rankability scanned the top 1,000 most-visited websites globally. As of mid-2025, only 0.3% had an llms.txt file. That's 3 out of 1,000. **No AI provider has officially committed** to reading llms.txt. OpenAI, Anthropic, Google — none have confirmed their crawlers consistently follow llms.txt instructions. Google's John Mueller compared the standard to the deprecated keywords meta tag. So why bother? Three reasons: **Low effort, no downside.** Setting up llms.txt takes 5 minutes. There's zero risk. It won't hurt your rankings, slow your site, or cause conflicts with anything. **The direction is clear.** Even if no one reads llms.txt *today*, AI search is growing at 34% CAGR. Gartner predicts 25% of search volume will shift away from traditional engines by late 2026. When AI providers *do* formalize how they read websites, llms.txt (or something very similar) will be the standard. **First-mover advantage is real.** If only 0.3% of top sites have llms.txt, that means 99.7% of your competitors don't. Early adopters set the baseline that later standards build on. This trips people up, so here's a clear comparison: | | **robots.txt** | **XML Sitemap** | **llms.txt** | |---|---|---|---| | **Purpose** | Controls crawler *access* | Lists *all* URLs for indexing | Curated overview for *AI understanding* | | **Audience** | Search engine crawlers | Search engine crawlers | Language models (LLMs) | | **Format** | Custom syntax | XML | Markdown | | **Content** | Allow/Disallow rules | URLs + metadata | Descriptions + key page links | | **Analogy** | Security guard | Phone book | Tour guide | They serve completely different purposes. You need all three — robots.txt controls who gets in, the sitemap lists what exists, and llms.txt explains what matters. You have two options: manual or plugin. Manual works, but it's tedious to maintain. Here's why. Create a file called `llms.txt` in your WordPress root directory (same place as `wp-config.php`): ``` # Your Site Name > Brief description of your website and what it offers. ## Pages - [Home](https://yoursite.com/): Main landing page - [About](https://yoursite.com/about/): Company information - [Services](https://yoursite.com/services/): What we offer ## Blog - [Latest Post](https://yoursite.com/blog/latest/): Description ``` Upload it via FTP or your hosting file manager. **The problem:** Every time you publish a new post, update a page, or change your site structure, you need to manually edit this file. For a site with 50+ posts, maintaining llms.txt by hand gets old fast. You also need to handle URL rewrites so WordPress serves the file correctly, which means editing `.htaccess` or adding a custom rewrite rule in `functions.php`. This is where things get practical. Several WordPress plugins now generate llms.txt automatically. The setup typically looks like: **What Prime SEO does specifically:** Prime SEO includes llms.txt generation as part of its AI Settings module. Here's what it offers beyond basic generation: The llms.txt and AI bots features are free — no Pro license required. [**Install Prime SEO on WordPress.org**](https://wordpress.org/plugins/prime-seo/) — takes about 2 minutes. Not every page belongs in your llms.txt. The whole point is curation — helping AI focus on what matters. **Include:** **Skip:** `noindex` **Good descriptions matter.** Don't just list URLs — add context: ``` ## Documentation - [Getting Started](https://yoursite.com/docs/start): Step-by-step installation and configuration guide for new users - [API Reference](https://yoursite.com/docs/api): Complete REST API documentation with code examples in Python, JavaScript, and PHP ``` That description tells an AI model *exactly* what it'll find at each URL, so it can pick the right source for the user's question. Here's a shortened version of our own llms.txt at [prime-seo-plugin.com/llms.txt](https://prime-seo-plugin.com/llms.txt) (the full text of the same pages is in [llms-full.txt](https://prime-seo-plugin.com/llms-full.txt)): ``` # Prime SEO > Free WordPress SEO plugin for Google and AI search: meta tags, XML sitemap, > schema markup, redirects, llms.txt and control over 12 AI crawlers. ## Documentation - [Getting Started](https://prime-seo-plugin.com/docs/getting-started/): Install Prime SEO, run the setup wizard, and start optimizing your site - [LLMs.txt](https://prime-seo-plugin.com/docs/llms-txt/): Generate and manage LLMs.txt files for your WordPress site - [AI Bots Manager](https://prime-seo-plugin.com/docs/ai-bots-manager/): Control how AI search engines crawl your WordPress site - [Schema Markup](https://prime-seo-plugin.com/docs/schema-markup/): Add JSON-LD structured data to your WordPress site ## Product - [Prime SEO PRO](https://prime-seo-plugin.com/pro.html): AI-written SEO titles and descriptions, bulk generation, video sitemap ``` Notice how each link has a description that tells the AI model what the page contains. An AI answering "How do I set up schema markup in Prime SEO?" knows exactly which link to pull from. Here's the quick version for people who skim (no judgment — we do it too): `yoursite.com/llms.txt` and verify it loads llms.txt is still a proposal, not an official web standard. It doesn't have W3C backing or formal support from major AI providers. But the momentum is real. Three community directories track adoption. GitHub discussions are active. Major tech companies are implementing it on their documentation sites. And the underlying problem — AI models struggling to understand website structure — isn't going away. Whether the final standard is called llms.txt or something else, the concept of "a curated, machine-readable summary of your website" is becoming essential. Setting it up now costs nothing and positions your site for whatever comes next. The sites that prepared for mobile-first indexing early didn't regret it. The same pattern is playing out with AI search readiness. **FAQ** **Is llms.txt required for WordPress?** No. It's optional, and your site will function perfectly without it. But adding it improves your chances of being correctly cited when AI search engines reference your content. **Will llms.txt improve my Google rankings?** No. Google does not use llms.txt for ranking. It specifically targets AI language models like ChatGPT, Claude, and Perplexity. Your traditional SEO setup (sitemap, meta tags, schema) remains separate. **Do I need both llms.txt and llms-full.txt?** For most sites, llms.txt alone is sufficient. llms-full.txt is useful if you have extensive documentation and want AI models to access the full content without crawling each page individually. **How often should I update llms.txt?** Every time you add or remove important content from your site. If you use a plugin like Prime SEO, this happens automatically when you publish or update posts. **Is llms.txt the same as robots.txt?** No. robots.txt controls which crawlers can access your site. llms.txt tells AI models what your site is about and which pages are most important. They serve different purposes, and you need both.