{"slug": "serve-markdown-to-ai-crawlers-from-static-html-pages-with-a-few-htaccess-lines", "title": "Serve Markdown to AI crawlers from static HTML pages with a few .htaccess lines", "summary": "A developer published a set of Apache .htaccess rules that serve a Markdown version of each static HTML page to known AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot and others) or to any client sending an Accept: text/markdown header, while browsers and Googlebot continue to receive HTML. The setup adds a text/markdown MIME type, a Vary header on Accept and User-Agent for correct caching, and a rel=\"alternate\" discovery link in the page head, with curl commands for verifying bot, browser and content-negotiation responses. The developer notes that actual AI-assistant usage of the Markdown files is not yet measurable, and that stale caching caused a query-string bug during testing.", "body_md": "AI assistants read web pages, and HTML is a noisy format for them: navigation, scripts, styles and cookie banners around a few paragraphs of content. A Markdown version of the same page is cheaper to read and easier to quote. I added one to the static pages of my site and it took a handful of Apache rules. Here is the setup, with the mistakes I made.\n\nDisclosure: the pages I used this on belong to my app, Teleprompter Offline & Private, and I also build a WordPress plugin that does the same job for WordPress.\n\n**The idea**\n\nFor every page `/some-page/index.html`, put a file `/some-page/index.md` next to it with the same content. Then serve the Markdown file when the request comes from a known AI crawler or when the client asks for it with an `Accept: text/markdown` header. Everyone else, browsers and Googlebot included, keeps getting the HTML.\n\n**The rules**\n\n```\nAddType \"text/markdown; charset=utf-8\" .md\n\nRewriteEngine On\nRewriteCond %{REQUEST_URI} /$\nRewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI}index.md -f\nRewriteCond %{HTTP_ACCEPT} text/markdown [NC,OR]\nRewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider) [NC]\nRewriteRule ^(.*)$ $1index.md [E=MDPAGE:1,L]\n\nHeader append Vary \"Accept, User-Agent\" env=MDPAGE\n```\n\nThree details matter:\n\n`index.md` are touched. Pages without one are served as usual.`Vary` header.`Accept` and `User-Agent`. Without `[NC,OR]`.\nI also added a discovery link in the page head, so crawlers that read HTML can find the Markdown:\n\n```\n\n```\n\n**Test it with curl**\n\n```\n# browser: HTML\ncurl -s -A \"Mozilla/5.0\" https://example.com/some-page/ | head -3\n# bot user agent: Markdown\ncurl -s -A \"GPTBot\" https://example.com/some-page/ | head -3\n# content negotiation: Markdown\ncurl -s -H \"Accept: text/markdown\" https://example.com/some-page/ | head -3\n# check the headers\ncurl -sI -A \"GPTBot\" https://example.com/some-page/ | grep -i \"content-type\\|vary\"\n```\n\n**Mistakes I made**\n\n`?x=1` returned Markdown. After clearing the cache everything worked. Test with a cache-busting query string, and purge after changes.\n**Does it work?**\n\nI cannot tell yet how much AI assistants use it. What I can verify is the delivery: browsers get HTML, bots get Markdown, caches vary correctly. I also list the pages in the sitemap and keep the usual structured data in the HTML, because normal search still matters.\n\nWhere I use it: the pages of my teleprompter app. The landing page is [https://www.a-devgroup.com/teleprompter/](https://www.a-devgroup.com/teleprompter/) and its Markdown version is at [https://www.a-devgroup.com/teleprompter/index.md](https://www.a-devgroup.com/teleprompter/index.md). The app itself, a free offline teleprompter for iPhone with voice-follow scrolling, is on the App Store: [https://apps.apple.com/ua/app/teleprompter-offline-private/id6808964093](https://apps.apple.com/ua/app/teleprompter-offline-private/id6808964093)\n\nIf you run WordPress instead of static files, the plugin I build does the negotiation for posts and pages: [https://www.a-devgroup.com/adg-ai-markdown/](https://www.a-devgroup.com/adg-ai-markdown/)\n\nQuestion for you: do you serve anything special to AI crawlers, or do you leave them the plain HTML?", "url": "https://wpnews.pro/news/serve-markdown-to-ai-crawlers-from-static-html-pages-with-a-few-htaccess-lines", "canonical_source": "https://dev.to/andrew_gnasevych_eceaa0ef/serve-markdown-to-ai-crawlers-from-static-html-pages-with-a-few-htaccess-lines-5e95", "published_at": "2026-11-12 09:49:07+00:00", "updated_at": "2026-11-12 09:51:32.703584+00:00", "lang": "en", "topics": ["ai-crawlers", "agent-protocols", "structured-data", "developer-tools"], "entities": ["GPTBot", "ClaudeBot", "PerplexityBot", "CCBot", "Bytespider", "Apache", "Googlebot", "Teleprompter Offline & Private"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/serve-markdown-to-ai-crawlers-from-static-html-pages-with-a-few-htaccess-lines", "markdown": "https://wpnews.pro/news/serve-markdown-to-ai-crawlers-from-static-html-pages-with-a-few-htaccess-lines.md", "text": "https://wpnews.pro/news/serve-markdown-to-ai-crawlers-from-static-html-pages-with-a-few-htaccess-lines.txt", "jsonld": "https://wpnews.pro/news/serve-markdown-to-ai-crawlers-from-static-html-pages-with-a-few-htaccess-lines.jsonld"}}