{"slug": "making-a-website-ai-friendly-what-the-bots-actually-fetched-from-mine", "title": "Making a website AI friendly: what the bots actually fetched from mine", "summary": "A first-party audit of one website's Cloudflare logs from 30 September to 17 October 2026 found that no verified AI or search crawler fetched the site's llms.txt file in 17 days, while plain HTML pages drew 15,431 requests in 18 days and Markdown copies drew 432. ClaudeBot alone fetched the sitemap 122 times, and seven verified crawlers made 143 of the 152 non-owner requests for the Markdown copies, most likely via the link in each page's head. The author also reported that 113 requests claiming to be PerplexityBot were unverified by Cloudflare, and that the site's MCP server saw about 300 connections and 29 tool calls over three and a half days, with clients requesting a dozen discovery addresses it does not have.", "body_md": "# Making a website AI friendly: what the bots actually fetched from mine\n\nI kept reading lists of files to add to make a website “AI friendly”, and I wanted to know whether anything actually reads them. This site has most of them: a `robots.txt` that welcomes AI crawlers, an `llms.txt`, a Markdown copy of every page, full-text feeds, structured data, an AI Catalog and an MCP server of its own. So on 18 October 2026 I looked at what actually came through each door. I read 18 days of the site’s Cloudflare logs (30 September to 17 October 2026), three and a half days of its MCP server’s telemetry, and the standards and vendor documentation behind each door as they stood that day. Some of it surprised me.\n\n## In brief\n\n1. **Most AI crawler traffic went to the oldest doors: pages, `robots.txt` and the sitemap.** ClaudeBot alone fetched the sitemap 122 times in 18 days.\n2. **No verified AI or search crawler fetched `llms.txt` in 17 days.** The one verified AI fetch came from Claude-User, which fetches when a person asks. Google Search says it doesn’t use the file, and the larger studies find that its readers are coding agents.\n3. **The Markdown copies were found the day after they went live.** Seven verified crawlers made 143 of the 152 requests for them that weren’t mine, most likely through the link in each page’s head.\n4. **Feeds are read by feed readers.** Most of their requests only asked whether anything had changed.\n5. **An MCP server is reached by MCP clients and directories, not by crawlers.** In three and a half days it saw about 300 connections and 29 tool calls, and clients asked for a dozen discovery addresses it doesn’t have. No mainstream client finds a site’s server on its own yet.\n6. **A crawler’s name in a request proves nothing.** 113 requests said they came from PerplexityBot without Cloudflare verifying them. Count verified bots only.\n7. **You can measure most of this for free, with limits.** Cloudflare’s Free plan keeps 31 days of counts per crawler and per path, but not the`Accept` header and not referrers.\n\n## The doors, and who knocks\n\nMachines come to a website for different reasons, and each reason favours different doors. The big operators document three kinds of visitor, and keep them apart: a training crawler (OpenAI’s GPTBot, Anthropic’s ClaudeBot, Amazonbot), a search crawler that builds the index AI answers cite (OAI-SearchBot, Applebot, Googlebot), and a fetcher that gets one page because a person asked (ChatGPT-User, Claude-User, Perplexity-User). Two more show up in the logs and the studies: coding agents such as Claude Code and Codex, and plain feed readers. An agent that uses the Model Context Protocol (MCP) comes in by yet another door.\n\nThis is how they lined up on this site, by what each kind of visitor actually fetched. The training and search crawlers used the same doors, so they share a box.\n\nI followed fourteen verified crawlers and fetchers by name: eleven that Cloudflare files as AI crawlers, AI search or AI assistants, and three search crawlers, OpenAI’s and Exa’s, that it files under search engines. Counting how many of the 14 used each door at least once gives a rough map of what matters. Keep in mind that the doors weren’t open equally long: pages, `robots.txt` and the sitemap for all 18 days, `llms.txt` for 17, the feeds for 13 to 17, the Markdown copies and `llms-full.txt` for 8, and the AI Catalog for 1.\n\n## Plain HTML still does most of the work\n\nThe page itself is the door every crawler uses, so it’s the one to get right first. All 14 crawlers fetched pages, and nothing else on the site came close in volume: 15,431 page requests in 18 days, against 432 for the Markdown copies.\n\nWhat helps a machine read a page is mostly what has always helped. The extractors are documented in their code: Mozilla’s Readability, the library behind Firefox’s reader view, reads the page’s JSON-LD first, then its Open Graph and `article:` meta tags, and drops `<footer>`, `<aside>` and hidden elements. trafilatura, the extractor Hugging Face used for its FineWeb training set, reads the same meta tags and JSON-LD. When I ran one of this site’s articles through Jina Reader, a service that turns a page into text for language models, its title and date came from the page’s JSON-LD, and the footer with the author line was gone.\n\nThe one thing a modern site can easily get wrong is rendering. In [Vercel’s measurement of December 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler), OpenAI’s, Anthropic’s and Perplexity’s crawlers didn’t run JavaScript, while Googlebot and Applebot did, and [Anthropic’s web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool) says it doesn’t support pages rendered with JavaScript. A page whose text is built in the browser can look fine in Google and be empty to most AI crawlers. This site is static HTML with no client-side rendering, one `<main>`, real headings and JSON-LD built from the same content files as the page, which is the safe ground.\n\nWhat you don’t need, according to Google, is anything special. Its guide to [optimizing for generative AI features in Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says AI Overviews and AI Mode need no special markup, no special schema.org, and no `llms.txt` or Markdown files, and it lists overfocusing on structured data among the myths. [Bing’s guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a), which cover Copilot too, say structured data may support clearer grounding but guarantees nothing, and ask for markup that matches the visible page. The studies of schema and AI citations I found are vendors’ own and mostly found no lift.\n\n## robots.txt: say yes or no by purpose\n\nA `robots.txt` file is where you tell crawlers what they may fetch, and since the operators split their bots by purpose, you can answer each purpose separately. The format is [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) (September 2022), and it’s worth remembering what the RFC itself says: the rules are not a form of access control.\n\nTwo details change how much it can do for you.\n\n- **User-triggered fetchers may not follow it.** OpenAI says`robots.txt` rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. Anthropic’s page is the exception: it says all three of its bots honour`robots.txt` , Claude-User included.\n- **A name in the user agent can be anyone.** Cloudflare verifies a bot by a published IP list, reverse DNS or, more and more, a signature on the request ([Web Bot Auth](https://datatracker.ietf.org/wg/webbotauth/about/) , built on HTTP Message Signatures, RFC 9421, which OpenAI’s agent and some of Google’s agent requests already use). On this site, leaving images and scripts aside, 113 requests named PerplexityBot, 112 Kimi-SearchBot, 85 Bingbot and 75 Googlebot without Cloudflare verifying them. Some are spoofed, some come from operators that aren’t on the verified list, and the logs can’t tell which.\n\nThere’s also a newer layer that says what may be done with content after it’s fetched. Cloudflare’s [Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) (September 2025) adds a `Content-Signal` line to `robots.txt` with three answers, `search`, `ai-input` (using the content to ground an AI answer) and `ai-train`. The IETF’s AI Preferences working group is drafting something similar, a `Content-Usage` rule and HTTP header with `search`, `ai-use` and `train-ai`; both drafts missed their August 2026 milestone. Neither has an operator on record yet: none of the crawler documentation I read mentions either one, and Cloudflare’s scan found `Content-Signal` on 2.4% of about 109,000 top domains.\n\nMy own file is four lines: every crawler is allowed, and the `Content-Signal` line says yes to search, AI answers and AI training. I want the site read and cited, so the choice was easy. One practical note if you’re on Cloudflare: its managed `robots.txt` would add `ai-train=no`, so I keep it off.\n\n## Sitemap and IndexNow: say what changed\n\nA sitemap with honest dates is the cheapest way to tell crawlers what changed, and the logs show it gets read: ClaudeBot fetched this site’s sitemap 122 times in 18 days, Googlebot 19 and GPTBot 10. [Google](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) uses `lastmod` only when it’s consistently accurate and ignores `changefreq` and `priority`; [Bing](https://blogs.bing.com/webmaster/July-2025/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search) says the same and calls sitemaps a foundational signal for Copilot too. Here, each page’s `lastmod` is the date its content last changed, never the build date.\n\nIndexNow is a protocol Bing and Yandex launched in October 2021: a site pings one search engine when a URL is added, changed or deleted, and that engine shares the ping with all the others. The [registry](https://www.indexnow.org/searchengines.json) lists Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot. Google isn’t part of it. Bing’s guidelines tie it directly to Copilot: timely IndexNow pings reduce outdated URLs in Copilot answers. No OpenAI document says IndexNow reaches ChatGPT search, so I’d treat that as unproven. This site sends the addresses each deploy changed, straight after the deploy.\n\n## llms.txt is cheap, and coding agents are its readers\n\n`llms.txt` is a Markdown file at a site’s root that gives language models a short, curated map of the site: a title, a one-line summary and lists of links. Answer.AI proposed it in September 2024, and [version 2](https://llmstxt.org/) (August 2026) adds a way to find it from any page, a `rel=\"describedby\"` link. `llms-full.txt`, one file with the whole site in it, is a convention from documentation platforms, not part of the proposal. This site has had an `llms.txt` since 1 October and an `llms-full.txt` since 10 October, both generated from the same content files as the pages.\n\nIn 17 days, no verified AI or search crawler fetched it. Of 75 requests for `/llms.txt`, 44 were my own tools, 24 came from browsers and clients that didn’t identify themselves, 5 from Chrome’s Lighthouse, 1 from an agent directory’s bot, and 1 from Claude-User. Not one came from GPTBot, ClaudeBot, OAI-SearchBot, Applebot, Amazonbot, Googlebot or Bingbot.\n\nThat matches what I found elsewhere:\n\n- **Google Search doesn’t use it** , by its own guide, which adds that keeping one for other services neither helps nor harms a site in Search. Google’s Search Advocate wrote in June 2025 that no AI system used`llms.txt` .\n- **[Ahrefs looked at 137,210 sites](https://ahrefs.com/blog/llmstxt-study/)** in May 2026: 28% had a valid file, 97% of those files got no request that month, and AI retrieval bots made 1.1% of the requests the rest got. GPTBot was the most frequent named bot, and Claude Code came second.\n- **Coding agents do read it.**[Gauge found Codex](https://www.withgauge.com/blog/does-codex-read-llms-txt/) opening`llms.txt` in about a third of its sessions, through`curl` , which doesn’t say who’s asking. A count by crawler name, like mine, misses those reads.\n- **Chrome’s Lighthouse checks it.** Its new Agentic Browsing category audits the file’s form when there is one, and doesn’t penalise a site without one.\n\nSo I’d keep an `llms.txt`, because it’s cheap and coding agents use it, but I wouldn’t expect it to change how search crawlers or AI answers see a site.\n\n## Markdown copies were found within a day\n\nA Markdown copy of each page is the door that surprised me most. Since 10 October, every page here has a copy at its own address with `.md` added (`/about.md` for `/about`), served as `text/markdown`, written from the same content files as the page rather than converted from its HTML, and kept out of search results with `noindex`. Each page names its copy in its head:\n\n```\n<link rel=\"alternate\" type=\"text/markdown\" href=\"https://pournasserian.com/about.md\">\n```\n\nVerified crawlers started fetching the copies on 11 October, the day after they went live. Of 432 requests, 280 were my own tools and checks; of the other 152, seven verified crawlers made 143.\n\nNone of these crawlers had fetched `llms.txt`, so they didn’t find the copies there. The link in the head is the likely route, though the logs can’t show how a crawler found an address. A month of logs published on dri.es, another personal site, showed the same pattern: crawlers reached its Markdown only through the `.md` addresses and the head link, with Amazonbot, OAI-SearchBot and GPTBot fetching the most, in that order. Googlebot and Bingbot fetched no copy here.\n\nThe other way to serve Markdown is content negotiation. A client sends `Accept: text/markdown` to the page’s own address, and the server answers with Markdown and `Vary: Accept` (the `text/markdown` type is [RFC 7763](https://www.rfc-editor.org/rfc/rfc7763); the negotiation is ordinary HTTP, [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110)). Developer tools are the ones asking: Claude Code sends `text/markdown, text/html, */*`, and OpenCode and VS Code’s chat ask for Markdown first. No crawler operator documents sending it, and [Cloudflare’s scan](https://blog.cloudflare.com/agent-readiness/) found 3.9% of about 200,000 top domains answering it. Since 18 October this site answers it with the same copies, through a small Worker in front of the pages, so the copy’s address comes back in a `Content-Location` header. In its first 49 minutes, one verified crawler used it: ExaSearchBot asked for an article as Markdown and got the copy. That’s far too early to say more.\n\nIf your site is on Cloudflare, [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) does the negotiation for you by converting your HTML at the edge. It’s a switch on Pro plans and above (this site is on the Free plan, where the switch shows “Pro plan”), and it removes headers, footers and navigation as it converts. I chose my own copies because they hold the content files’ words exactly, diagrams as their source included, but conversion is a reasonable shortcut when you don’t have Markdown sources.\n\n## Feeds with the whole post\n\nA feed with the whole post in it is an old door that still gets plenty of use, mostly by feed readers. The two feeds here carry each post in full, in `content:encoded`. In 18 days they got 691 requests: 572 from clients Cloudflare doesn’t verify, most of them feed readers (one ingesting service alone made 140), 29 from my own tools, a few dozen from Flipboard, small search engines and other named bots, and 1 to 4 each from ten verified AI crawlers, ClaudeBot, GPTBot, Amazonbot, Applebot and OAI-SearchBot among them. Most of the traffic is a reader asking whether anything changed: in a 30-day count, 401 of 551 answers on `/rss.xml` were `304 Not Modified`.\n\nNone of the AI operators’ crawler pages mention feeds, so I can’t say any of them reads feeds by design. Google and Bing both accept a feed as a sitemap, which is reason enough to keep one.\n\n## An MCP server and the AI Catalog\n\nAn MCP server is the one door that lets an agent ask a site a question rather than read its pages one by one. MCP is an open standard that connects AI applications to outside systems; this site runs [mcp-my-site](https://pournasserian.com/open-source/mcp-my-site), the open-source server I built, at `ai.pournasserian.com`, with a `search` tool that ranks the whole site and a `fetch` tool that returns one page as Markdown, from an index rebuilt each time the site deploys. The [agents page](https://pournasserian.com/agents) explains how to connect to it.\n\nFinding a site’s server is the part that isn’t settled yet. The intended route, from the [MCP Server Cards proposal](https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/seps/2127-mcp-server-cards.md) (final since 6 October 2026), goes like this:\n\nAn AI Catalog is a JSON file at `/.well-known/ai-catalog.json` that lists a site’s AI artifacts, such as MCP server cards. The card gives the server’s name, version and address, deliberately without its tools, and the live `server/discover` and `tools/list` calls give the rest (I wrote about [`server/discover`](https://pournasserian.com/writing/mcp-2026-server-discover) in the MCP series). This site has carried its catalog since 17 October. On that first day, one crawler fetched it: OAI-SearchBot.\n\nThe server’s own telemetry shows who does come. In three and a half days (14 to 18 October), clients opened about 300 connections (308 `initialize` calls, 281 tool lists) and made 29 tool calls, 15 searches and 14 fetches. Many of the clients named themselves as MCP or agent directories and rankers. They also asked for a dozen discovery addresses the server doesn’t have: `/.well-known/agent-card.json` (the address A2A, another agent protocol, uses), `/.well-known/mcp.json` and `/.well-known/mcp/server-card.json` (from earlier Server Card drafts), `/.well-known/ard.json` (a rival proposal), and `/mcp/server-card`, where the final proposal puts a server’s card and where mine doesn’t serve one yet. Everyone is guessing at once.\n\nNo mainstream client follows the catalog by itself yet. Claude Code, Claude.ai, VS Code and Cursor all take a server’s URL from you, or from their own gallery; the only client code for Server Cards I found was in open pull requests. So if you run a server, expect to hand out its address yourself, as the [agents page](https://pournasserian.com/agents) does. The [44 tools in my earlier roundup](https://pournasserian.com/writing/mcp-servers-for-websites-2026) are the ways to get one without writing it.\n\n## Measuring it yourself\n\nYou can see most of this without paying for anything, as long as you count at the edge and count verified bots only.\n\n- **Cloudflare.**[AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) is on every plan. On the Free plan it identifies crawlers by user agent; the GraphQL Analytics API behind it keeps 31 days of requests by user agent, verified bot category, path and status, which is everything in this article. It doesn’t show the`Accept` header, so you can’t see who negotiates Markdown, and referrers are a paid feature, so you can’t see visits that AI answers send you.\n- **JavaScript analytics can’t see crawlers.** Cloudflare Web Analytics, like any tag in the page, counts browsers that run its script, and most AI crawlers don’t run scripts.\n- **Google Search Console.** Clicks from AI Overviews and AI Mode are counted with other web clicks and can’t be separated. Its Generative AI report, on every site since 31 August 2026, shows impressions only.\n- **Bing Webmaster Tools.** Its AI Performance report (in preview since February 2026) shows how often Copilot and Bing’s AI summaries cite your pages, and for which grounding queries, but not clicks.\n- **Lighthouse.** The Agentic Browsing category in Lighthouse 13.5 (also in PageSpeed Insights) checks the accessibility tree, layout shift,`llms.txt` , the AI Catalog and WebMCP. This site scored 3 of 4: its catalog entry lacks a`displayName` that one of two competing specs requires.\n- **Your own server’s telemetry.** If you run an MCP server, count calls by method and channel. Mine doesn’t log what people search for, because queries can carry personal data.\n\n## A checklist\n\nIf I were starting a site today, this is the order I’d do it in, by what the evidence says gets read:\n\n1. **Serve the text in the HTML.** No client-side rendering of content, one`<main>` , real headings, and JSON-LD that matches the page.\n2. **Write a `robots.txt` that answers each purpose.** Add a`Content-Signal` line if you have a view, knowing nobody has committed to reading it yet.\n3. **Keep a sitemap with honest `lastmod` dates,** and ping IndexNow when you deploy.\n4. **Publish a Markdown copy of each page** and name it in the head with`rel=\"alternate\"` . Answer`Accept: text/markdown` with`Vary: Accept` if you can.\n5. **Put the whole post in your feeds.**\n6. **Add an `llms.txt`** for coding agents. It’s cheap; just don’t expect search crawlers to read it.\n7. **If you want agents to search you, run an MCP server** and publish an AI Catalog, and give people the address yourself.\n8. **Measure at the edge,** and count only the bots your CDN verified.\n\n## Method and caveats\n\n- The figures come from Cloudflare’s GraphQL Analytics API for `pournasserian.com` , 30 September 2026 00:00 UTC to 18 October 2026 00:00 UTC, read on 18 October 2026, and from the AI Crawl Control dashboard the same day. The dataset is sampled; the bot counts are close to exact and the totals are Cloudflare’s estimates. The MCP server’s figures are Application Insights counts from 14 to 18 October 2026, with no query text.\n- “Verified” means Cloudflare put the request in a verified bot category. The fourteen crawlers I followed by name leave out a few verified AI assistants that send a generic browser user agent. Requests that only named a crawler are counted apart, and so are my own tools by their user agents (`curl` , the site’s indexer, audit scripts, Claude Code and the Claude app), along with one evening’s check that fetched 165 Markdown copies. Someone else using`curl` would have been counted as me.\n- The doors were open for different lengths of time, and Markdown negotiation, the site’s search box and the agents page went live on 18 October, outside the period; the only figure for them is the first 49 minutes of Markdown negotiation, 15:31 to 16:20 UTC that day. The figures are counts of requests, not reads: a fetch says nothing about what a crawler did with the page.\n- This is one small personal site. A larger or busier site will see more crawlers and different proportions. The studies I cite are mostly by vendors measuring their own platforms, and I’ve named them as such.\n- The standards, the operators’ documentation and Cloudflare’s features are as I read them on 18 October 2026. Several are moving: `llms.txt` v2, the IETF’s AI Preferences drafts, Web Bot Auth, MCP Server Cards and Lighthouse’s Agentic Browsing audits were all changed within the last few months.", "url": "https://wpnews.pro/news/making-a-website-ai-friendly-what-the-bots-actually-fetched-from-mine", "canonical_source": "https://pournasserian.com/writing/ai-friendly-website-2026", "published_at": "2026-10-18 00:00:00+00:00", "updated_at": "2026-10-18 16:49:06.971882+00:00", "lang": "en", "topics": ["ai-crawlers", "generative-engine-optimization", "agent-protocols", "structured-data", "ai-search"], "entities": ["Cloudflare", "ClaudeBot", "PerplexityBot", "GPTBot", "OAI-SearchBot", "Claude-User", "Model Context Protocol", "Anthropic"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/making-a-website-ai-friendly-what-the-bots-actually-fetched-from-mine", "markdown": "https://wpnews.pro/news/making-a-website-ai-friendly-what-the-bots-actually-fetched-from-mine.md", "text": "https://wpnews.pro/news/making-a-website-ai-friendly-what-the-bots-actually-fetched-from-mine.txt", "jsonld": "https://wpnews.pro/news/making-a-website-ai-friendly-what-the-bots-actually-fetched-from-mine.jsonld"}}