AI assistants read web pages, and HTML is a noisy format for them: navigation, scripts, styles and cookie banners around a few paragraphs of content. A Markdown version of the same page is cheaper to read and easier to quote. I added one to the static pages of my site and it took a handful of Apache rules. Here is the setup, with the mistakes I made.
Disclosure: the pages I used this on belong to my app, Teleprompter Offline & Private, and I also build a WordPress plugin that does the same job for WordPress.
The idea
For every page /some-page/index.html, put a file /some-page/index.md next to it with the same content. Then serve the Markdown file when the request comes from a known AI crawler or when the client asks for it with an Accept: text/markdown header. Everyone else, browsers and Googlebot included, keeps getting the HTML.
The rules
AddType "text/markdown; charset=utf-8" .md
RewriteEngine On
RewriteCond %{REQUEST_URI} /$
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI}index.md -f
RewriteCond %{HTTP_ACCEPT} text/markdown [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider) [NC]
RewriteRule ^(.*)$ $1index.md [E=MDPAGE:1,L]
Header append Vary "Accept, User-Agent" env=MDPAGE
Three details matter:
index.md are touched. Pages without one are served as usual.Vary header.Accept and User-Agent. Without [NC,OR].
I also added a discovery link in the page head, so crawlers that read HTML can find the Markdown:
<link rel="alternate" type="text/markdown" href="https://dev.to/some-page/index.md" title="Markdown version">
Test it with curl
curl -s -A "Mozilla/5.0" https://example.com/some-page/ | head -3
curl -s -A "GPTBot" https://example.com/some-page/ | head -3
curl -s -H "Accept: text/markdown" https://example.com/some-page/ | head -3
curl -sI -A "GPTBot" https://example.com/some-page/ | grep -i "content-type\|vary"
Mistakes I made
?x=1 returned Markdown. After clearing the cache everything worked. Test with a cache-busting query string, and purge after changes.
Does it work?
I cannot tell yet how much AI assistants use it. What I can verify is the delivery: browsers get HTML, bots get Markdown, caches vary correctly. I also list the pages in the sitemap and keep the usual structured data in the HTML, because normal search still matters.
Where I use it: the pages of my teleprompter app. The landing page is https://www.a-devgroup.com/teleprompter/ and its Markdown version is at https://www.a-devgroup.com/teleprompter/index.md. The app itself, a free offline teleprompter for iPhone with voice-follow scrolling, is on the App Store: https://apps.apple.com/ua/app/teleprompter-offline-private/id6808964093
If you run WordPress instead of static files, the plugin I build does the negotiation for posts and pages: https://www.a-devgroup.com/adg-ai-markdown/
Question for you: do you serve anything special to AI crawlers, or do you leave them the plain HTML?