# Serve Markdown to AI crawlers from static HTML pages with a few .htaccess lines

> Source: <https://dev.to/andrew_gnasevych_eceaa0ef/serve-markdown-to-ai-crawlers-from-static-html-pages-with-a-few-htaccess-lines-5e95>
> Published: 2026-11-12 09:49:07+00:00

AI assistants read web pages, and HTML is a noisy format for them: navigation, scripts, styles and cookie banners around a few paragraphs of content. A Markdown version of the same page is cheaper to read and easier to quote. I added one to the static pages of my site and it took a handful of Apache rules. Here is the setup, with the mistakes I made.

Disclosure: the pages I used this on belong to my app, Teleprompter Offline & Private, and I also build a WordPress plugin that does the same job for WordPress.

**The idea**

For every page `/some-page/index.html`, put a file `/some-page/index.md` next to it with the same content. Then serve the Markdown file when the request comes from a known AI crawler or when the client asks for it with an `Accept: text/markdown` header. Everyone else, browsers and Googlebot included, keeps getting the HTML.

**The rules**

```
AddType "text/markdown; charset=utf-8" .md

RewriteEngine On
RewriteCond %{REQUEST_URI} /$
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI}index.md -f
RewriteCond %{HTTP_ACCEPT} text/markdown [NC,OR]
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|PerplexityBot|CCBot|Bytespider) [NC]
RewriteRule ^(.*)$ $1index.md [E=MDPAGE:1,L]

Header append Vary "Accept, User-Agent" env=MDPAGE
```

Three details matter:

`index.md` are touched. Pages without one are served as usual.`Vary` header.`Accept` and `User-Agent`. Without `[NC,OR]`.
I also added a discovery link in the page head, so crawlers that read HTML can find the Markdown:

```
<link rel="alternate" type="text/markdown" href="https://dev.to/some-page/index.md" title="Markdown version">
```

**Test it with curl**

```
# browser: HTML
curl -s -A "Mozilla/5.0" https://example.com/some-page/ | head -3
# bot user agent: Markdown
curl -s -A "GPTBot" https://example.com/some-page/ | head -3
# content negotiation: Markdown
curl -s -H "Accept: text/markdown" https://example.com/some-page/ | head -3
# check the headers
curl -sI -A "GPTBot" https://example.com/some-page/ | grep -i "content-type\|vary"
```

**Mistakes I made**

`?x=1` returned Markdown. After clearing the cache everything worked. Test with a cache-busting query string, and purge after changes.
**Does it work?**

I cannot tell yet how much AI assistants use it. What I can verify is the delivery: browsers get HTML, bots get Markdown, caches vary correctly. I also list the pages in the sitemap and keep the usual structured data in the HTML, because normal search still matters.

Where I use it: the pages of my teleprompter app. The landing page is [https://www.a-devgroup.com/teleprompter/](https://www.a-devgroup.com/teleprompter/) and its Markdown version is at [https://www.a-devgroup.com/teleprompter/index.md](https://www.a-devgroup.com/teleprompter/index.md). The app itself, a free offline teleprompter for iPhone with voice-follow scrolling, is on the App Store: [https://apps.apple.com/ua/app/teleprompter-offline-private/id6808964093](https://apps.apple.com/ua/app/teleprompter-offline-private/id6808964093)

If you run WordPress instead of static files, the plugin I build does the negotiation for posts and pages: [https://www.a-devgroup.com/adg-ai-markdown/](https://www.a-devgroup.com/adg-ai-markdown/)

Question for you: do you serve anything special to AI crawlers, or do you leave them the plain HTML?