Serve Markdown to AI crawlers from static HTML pages with a few .htaccess lines

A developer published a set of Apache .htaccess rules that serve a Markdown version of each static HTML page to known AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot and others) or to any client sending an Accept: text/markdown header, while browsers and Googlebot continue to receive HTML. The setup adds a text/markdown MIME type, a Vary header on Accept and User-Agent for correct caching, and a rel="alternate" discovery link in the page head, with curl commands for verifying bot, browser and content-negotiation responses. The developer notes that actual AI-assistant usage of the Markdown files is not yet measurable, and that stale caching caused a query-string bug during testing.

AI assistants read web pages, and HTML is a noisy format for them: navigation, scripts, styles and cookie banners around a few paragraphs of content. A Markdown version of the same page is cheaper to read and easier to quote. I added one to the static pages of my site and it took a handful of Apache rules. Here is the setup, with the mistakes I made. Disclosure: the pages I used this on belong to my app, Teleprompter Offline & Private, and I also build a WordPress plugin that does the same job for WordPress. The idea For every page /some-page/index.html , put a file /some-page/index.md next to it with the same content. Then serve the Markdown file when the request comes from a known AI crawler or when the client asks for it with an Accept: text/markdown header. Everyone else, browsers and Googlebot included, keeps getting the HTML. The rules AddType "text/markdown; charset=utf-8" .md RewriteEngine On RewriteCond %{REQUEST URI} /$ RewriteCond %{DOCUMENT ROOT}%{REQUEST URI}index.md -f RewriteCond %{HTTP ACCEPT} text/markdown NC,OR RewriteCond %{HTTP USER AGENT} GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider NC RewriteRule ^ . $ $1index.md E=MDPAGE:1,L Header append Vary "Accept, User-Agent" env=MDPAGE Three details matter: index.md are touched. Pages without one are served as usual. Vary header. Accept and User-Agent . Without NC,OR . I also added a discovery link in the page head, so crawlers that read HTML can find the Markdown: <link rel="alternate" type="text/markdown" href="https://dev.to/some-page/index.md" title="Markdown version" Test it with curl browser: HTML curl -s -A "Mozilla/5.0" https://example.com/some-page/ | head -3 bot user agent: Markdown curl -s -A "GPTBot" https://example.com/some-page/ | head -3 content negotiation: Markdown curl -s -H "Accept: text/markdown" https://example.com/some-page/ | head -3 check the headers curl -sI -A "GPTBot" https://example.com/some-page/ | grep -i "content-type\|vary" Mistakes I made ?x=1 returned Markdown. After clearing the cache everything worked. Test with a cache-busting query string, and purge after changes. Does it work? I cannot tell yet how much AI assistants use it. What I can verify is the delivery: browsers get HTML, bots get Markdown, caches vary correctly. I also list the pages in the sitemap and keep the usual structured data in the HTML, because normal search still matters. Where I use it: the pages of my teleprompter app. The landing page is https://www.a-devgroup.com/teleprompter/ https://www.a-devgroup.com/teleprompter/ and its Markdown version is at https://www.a-devgroup.com/teleprompter/index.md https://www.a-devgroup.com/teleprompter/index.md . The app itself, a free offline teleprompter for iPhone with voice-follow scrolling, is on the App Store: https://apps.apple.com/ua/app/teleprompter-offline-private/id6808964093 https://apps.apple.com/ua/app/teleprompter-offline-private/id6808964093 If you run WordPress instead of static files, the plugin I build does the negotiation for posts and pages: https://www.a-devgroup.com/adg-ai-markdown/ https://www.a-devgroup.com/adg-ai-markdown/ Question for you: do you serve anything special to AI crawlers, or do you leave them the plain HTML?