An LLM-friendly website checklist: eight steps and a test for each

A checklist published on the author's site lays out eight steps for making a website readable and citable by AI crawlers and agents, each paired with a shell-command test, drawing on 18 days of the site's Cloudflare logs (30 September to 17 October 2026) and vendor documentation as of 18 October 2026. The evidence includes ClaudeBot fetching the site's sitemap 122 times in 18 days, seven verified crawlers finding Markdown page copies the day after they went live, and 113 requests claiming to be PerplexityBot that Cloudflare had not verified. The checklist recommends serving page text in the HTML because most AI crawlers do not run JavaScript, splitting robots.txt rules by crawler purpose, and counting only verified bots at the edge.

An LLM-friendly website checklist: eight steps and a test for each My longer article on making a website AI friendly https://pournasserian.com/writing/ai-friendly-website-2026 ends with a checklist: eight things I’d do, in the order the evidence says they get read. The evidence was 18 days of this site’s Cloudflare logs 30 September to 17 October 2026 and the standards and vendor documentation as they stood on 18 October 2026. A list only helps if you can tell when an item is done, so here it is again, short, with a check for each step that you can run against your own site in a minute or two. LLM-friendly, friendly to large language models LLMs , means here what AI friendly means there: a site that AI crawlers and agents can find, read and cite. The checks are shell commands. Set your site’s address once, and the rest can be pasted as they are: SITE=https://example.com In brief 1. Serve the text in the HTML. Most AI crawlers don’t run JavaScript, so text built in the browser can be invisible to them. 2. Write a robots.txt that answers each purpose. The big operators split their crawlers into training, search and fetching for a person, so you can say yes or no to each. 3. Keep a sitemap with honest lastmod dates, and ping IndexNow when you deploy. ClaudeBot fetched this site’s sitemap 122 times in 18 days. 4. Publish a Markdown copy of each page and name it in the head. Seven verified crawlers found this site’s copies the day after they went live. 5. Put the whole post in your feeds. Feeds are read mostly by feed readers, and most of their requests only ask whether anything changed. 6. Add an llms.txt for coding agents. No verified AI or search crawler fetched this site’s in 17 days, but coding agents do read the file. 7. If you want agents to search you, run an MCP server and publish an AI Catalog. No mainstream client finds a site’s server on its own yet, so hand out the address yourself. 8. Measure at the edge, and count only verified bots. 113 requests here said they came from PerplexityBot without Cloudflare verifying them. Serve the text in the HTML The text of each page has to be in the HTML your server sends, because most AI crawlers don’t run JavaScript. In Vercel’s measurement of December 2024 https://vercel.com/blog/the-rise-of-the-ai-crawler , OpenAI’s, Anthropic’s and Perplexity’s crawlers didn’t, while Googlebot and Applebot did, so a page whose text is built in the browser can look fine in Google and be empty to most AI crawlers. The rest is what has always helped a machine read a page: one <main , real headings, and JSON-LD that matches what the page shows. Fetch a page the way a crawler without JavaScript does, and look for a few plain words from a paragraph you can see in the browser: curl -s "$SITE/about" | grep -c "a few plain words from the page" curl -s "$SITE/about" | grep -o "<main" | wc -l curl -s "$SITE/about" | grep -c "application/ld+json" The first should print 1 or more; 0 means the text arrives later, from a script. Pick words without apostrophes or quotes, which the HTML may write differently. The second should print 1, and the third at least 1. To see what a text extractor keeps, ask Jina Reader https://jina.ai/reader/ , a service that turns a page into text for language models. The page’s title and its text should be there: curl -s "https://r.jina.ai/$SITE/about" | head -20 Write a robots.txt that answers each purpose A robots.txt file lets you answer each kind of crawler separately, because the big operators split theirs by purpose: training crawlers GPTBot, ClaudeBot, Amazonbot , search crawlers that build the index AI answers cite OAI-SearchBot, Applebot, Googlebot , and fetchers that get one page because a person asked ChatGPT-User, Claude-User, Perplexity-User . Its rules are not a form of access control, and the fetchers may not follow them: OpenAI says they may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, while Anthropic says all three of its bots honour them, Claude-User included. Cloudflare’s Content Signals Policy https://blog.cloudflare.com/content-signals-policy/ adds a Content-Signal line that says what may be done with the content once it’s fetched, with three answers, search , ai-input and ai-train , though none of the crawler documentation I read mentions it yet. Mine says yes to all three: User-agent: Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Sitemap: https://pournasserian.com/sitemap.xml Check the file crawlers actually get, not the one in your repository. On Cloudflare, for example, the managed robots.txt would add ai-train=no to it. curl -s "$SITE/robots.txt" Look for a User-agent group for each crawler you treat differently, and make sure the group says what you mean for everyone else. Keep the sitemap’s dates honest, and ping IndexNow A sitemap is read often, so its lastmod dates are worth getting right: ClaudeBot fetched this site’s sitemap 122 times in 18 days, Googlebot 19 times and GPTBot 10. Google https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap uses lastmod only when it’s consistently accurate, so each page’s date should be the day its content last changed, never the build date. IndexNow is the other half. It’s a protocol that lets a site ping one search engine when an address is added, changed or deleted, and that engine shares the ping with the others in its registry https://www.indexnow.org/searchengines.json : Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot. Bing’s guidelines say timely pings reduce outdated addresses in Copilot answers. Google isn’t part of it. curl -s "$SITE/robots.txt" | grep -i "^sitemap" curl -s "$SITE/sitemap.xml" | grep -o "<lastmod ^< " | sort | uniq -c | sort -rn | head The first should name your sitemap. The second counts the addresses that share each date: if one date covers nearly all of them, it’s probably the day you last built the site. If your sitemap is an index of other sitemaps, run it on each of those. For IndexNow, the key file at your site’s root should print your key, and each ping your deploy sends should get 200 or 202 back, both of which mean the ping was received: curl -s "$SITE/your-key.txt" Publish a Markdown copy of each page A Markdown copy of each page, named in the page’s head, was the door crawlers found fastest on this site: seven verified crawlers fetched the copies, starting the day after they went live, and Amazonbot most of all. Here each copy sits at its page’s address with .md added, is served as text/markdown , is kept out of search results with noindex , and is named in the page’s head: <link rel="alternate" type="text/markdown" href="https://pournasserian.com/about.md" curl -s "$SITE/about" | grep -o '<link ^ text/markdown ^ ' curl -s -o /dev/null -D - "$SITE/about.md" | grep -i "^content-type" The first should print the link, and the second text/markdown . You can also answer a request for Markdown at the page’s own address, which is how coding agents ask: Claude Code sends Accept: text/markdown, text/html, / . The server answers with Markdown and Vary: Accept , and sends a browser, which asks for text/html , the HTML it always had, with the same Vary: Accept , so a cache keeps the two apart. curl -s -o /dev/null -D - -H "Accept: text/markdown" "$SITE/about" | grep -iE "^ content-type|vary " curl -s -o /dev/null -D - "$SITE/about" | grep -iE "^ content-type|vary " The first should print text/markdown and Vary: Accept , the second text/html and the same Vary: Accept . If your site is on a Cloudflare Pro plan or above, Markdown for Agents https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ does the negotiation for you by converting your HTML at the edge. Put the whole post in your feeds A feed that carries each post in full is still read, mostly by feed readers, and most of their requests only ask whether anything changed: in a 30-day count, 401 of 551 answers on this site’s /rss.xml were 304 Not Modified . Google and Bing both accept a feed as a sitemap, which is one more reason to keep it. curl -s "$SITE/rss.xml" | grep -o "<item " | wc -l curl -s "$SITE/rss.xml" | grep -o "<content:encoded " | wc -l curl -s -o /dev/null -D - "$SITE/rss.xml" | grep -iE "^ etag|last-modified " In an RSS feed the first two numbers should match: one full text for each post. The third should print an ETag or a Last-Modified header, which is what lets a reader ask whether anything changed. Send the ETag back, and the answer should be 304 : curl -s -o /dev/null -w "%{http code}\n" -H 'If-None-Match: "the-etag-you-got"' "$SITE/rss.xml" Add an llms.txt for coding agents An llms.txt is cheap to add and coding agents read it, but don’t expect search or AI crawlers to: in 17 days, no verified AI or search crawler fetched this site’s. It’s a Markdown file at a site’s root that gives language models a short map of the site, a title, a one-line summary and lists of links, as the proposal https://llmstxt.org/ describes. Gauge found Codex https://www.withgauge.com/blog/does-codex-read-llms-txt/ opening one in about a third of its sessions, while Google Search says it doesn’t use the file. curl -s "$SITE/llms.txt" | head -5 It should open with a title and a summary line, with the lists of links under headings after them. Lighthouse’s Agentic Browsing category, also in PageSpeed Insights https://pagespeed.web.dev/ , checks the file’s form when there is one and doesn’t penalise a site without one. Run an MCP server and publish an AI Catalog An MCP server lets an agent search your site rather than read it page by page, but you’ll have to hand out its address yourself, because no mainstream client finds a site’s server on its own yet. MCP, the Model Context Protocol, is an open standard that connects AI applications to outside systems. An AI Catalog is a JSON file at /.well-known/ai-catalog.json that lists a site’s AI artifacts, such as its MCP server’s card, the route the MCP Server Cards proposal https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/seps/2127-mcp-server-cards.md sets out. This site runs mcp-my-site https://pournasserian.com/open-source/mcp-my-site , the open-source server I built, and the 44 tools in my earlier roundup https://pournasserian.com/writing/mcp-servers-for-websites-2026 are the ways to get one without writing it. curl -s "$SITE/.well-known/ai-catalog.json" | head -12 It should print JSON with an entry of type application/mcp-server-card+json for your server. Then connect a client to the server’s address and list its tools; the official MCP Inspector https://github.com/modelcontextprotocol/inspector does it, and I compared it with the other testing tools in an earlier article https://pournasserian.com/writing/mcp-server-testing-tools-2026 . Last, check that a person can find the address on your site, as the agents page https://pournasserian.com/agents gives it here. Measure at the edge, and count only verified bots Count crawlers in your CDN’s or your server’s logs, and only the ones verified, because a crawler’s name in a request proves nothing: 113 requests here said they came from PerplexityBot without Cloudflare verifying them. Analytics that run as JavaScript in the page can’t see them, since most AI crawlers don’t run scripts. Cloudflare verifies a crawler by a published IP list, reverse DNS or a signature on the request; count what passes, and count a request that only names a crawler apart. On Cloudflare, AI Crawl Control https://developers.cloudflare.com/ai-crawl-control/ is on every plan, and the GraphQL Analytics API https://developers.cloudflare.com/analytics/graphql-api/ behind it keeps 31 days on the Free plan. This query lists who fetched one path in a week. requestSource: "eyeball" leaves out Cloudflare’s own requests, the host filter leaves out your zone’s other hostnames, and an empty verifiedBotCategory means Cloudflare didn’t verify the request, whatever its user agent says: query { viewer { zones filter: { zoneTag: "YOUR ZONE ID" } { httpRequestsAdaptiveGroups filter: { datetime geq: "2026-10-11T00:00:00Z" datetime lt: "2026-10-18T00:00:00Z" requestSource: "eyeball" clientRequestHTTPHost: "example.com" clientRequestPath: "/llms.txt" } limit: 20 orderBy: count DESC { count dimensions { verifiedBotCategory userAgent } } } } } Off Cloudflare, you can check a crawler by reverse DNS yourself. Look up the address a request came from: Google’s crawlers https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests resolve to a name under googlebot.com , google.com or googleusercontent.com , and that name resolves back to the same address. nslookup 66.249.66.1 nslookup crawl-66-249-66-1.googlebot.com Method and caveats - The steps, their order and every figure come from the longer article https://pournasserian.com/writing/ai-friendly-website-2026 , which read this site’s Cloudflare logs from 30 September to 17 October 2026 and the standards and documentation as they stood on 18 October 2026. Its own method and caveats apply here. - The checks are new in this article. I ran each one against this site on 18 October 2026: the shell commands with curl in bash they’re written for a Unix shell such as bash or zsh , the GraphQL query against this site’s Cloudflare zone, and the two lookups against the Googlebot address shown. I didn’t send an IndexNow ping for this article; 200 and 202 are the answers IndexNow’s documentation lists for a ping it received. - A check that passes says a door is open, not that anyone uses it, and a count of requests says nothing about what a crawler did with a page. - This is one small personal site. A larger or busier site will see more crawlers and different proportions, and several of the standards here were still changing when I wrote this.