Expose every documentation page's raw Markdown source at a predictable URL — via a .md suffix on the canonical URL, content negotiation, or both. Agents pull source instead of parsing HTML.
What it is #
A per-page Markdown source endpoint exposes the original, unrendered Markdown of a content page at a predictable URL. An agent — an LLM, a documentation index, a CLI tool — fetches the Markdown instead of the HTML and gets the same content without the lossy round trip through DOM parsing, ad scaffolding, navigation chrome, and JavaScript hydration.
Two conventions are emerging, and the best sites do both.
1. URL suffix. Append .md
to the page path.
GET /docs/getting-started/ HTTP/1.1
GET /docs/getting-started.md HTTP/1.1
The first returns HTML. The second returns the raw Markdown source with a text/markdown
content type.
2. Content negotiation. Send Accept: text/markdown
to the canonical (HTML) URL and get Markdown back.
GET /docs/getting-started/ HTTP/1.1
Accept: text/markdown
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Content-Location: /docs/getting-started.md
Vary: Accept
The Content-Location
header tells the client the canonical Markdown URL; Vary: Accept
tells caches the response depends on the request’s Accept
header.
Why it matters #
Lossless source. HTML extraction loses heading semantics, code-block language hints, callout syntax, and inline metadata. Markdown gives the agent exactly what the author wrote.Smaller payloads. A typical Markdown page is 5–20× smaller than its rendered HTML once you remove styling, scripts, and shared chrome.No JavaScript runtime. Crawlers that don’t run JS still get full content.Aligned withllms.txt
.llms.txtandllms-full.txtpoint at pages —.md
makes the pointed-to representation directly usable.Stable signal of intent. Sites that expose.md
are advertising “we want to be readable by agents,” which is itself a useful discovery signal.
Anthropic’s documentation, Stripe’s, and a growing number of others ship the .md
suffix. Treat the canonical URL with Accept: text/markdown
as the next layer up — same content, the URL stays canonical, caches behave correctly.
How to implement #
Ship the suffix at minimum. For every Markdown-sourced page, serve the same Markdown at the URL with .md
appended.
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Cache-Control: public, max-age=3600
X-Markdown-Tokens: 1240
Include a YAML frontmatter block with the page’s metadata so agents get structured context alongside the prose: title
, url
(the canonical HTML URL), updated
, sources
, licence
. Don’t ship implementation details an agent doesn’t need (build timestamps, internal IDs).
Hint the size. Add an X-Markdown-Tokens
response header carrying a rough token-count estimate for the body (any reasonable tokeniser — tiktoken
’s cl100k_base
is a fine default). The header is not a registered standard, but a growing number of agent-friendly sites ship it because it lets a caller decide whether to inline the page into a prompt, summarise it first, or skip it for a smaller one — without down the body. Recompute it whenever the Markdown changes; do not invent the number.
Advertise it in <head>. Add a discovery link to the HTML version so crawlers and tools find the Markdown without guessing the URL pattern.
<link rel="alternate" type="text/markdown"
href="https://example.com/docs/getting-started.md"
title="Getting started — Markdown source">
Add content negotiation if your edge supports it. Cloudflare Pages Functions, Workers, Vercel Edge Middleware, Netlify Edge Functions, Fastly Compute, and Nginx can all branch on Accept
. The middleware:
- Matches the page URL pattern.
- If
Accept
containstext/markdown
, fetches the.md
asset internally and returns its body withContent-Type: text/markdown
,Content-Location: /…/page.md
, andVary: Accept
. - Otherwise serves HTML, but still appends
Vary: Accept
so caches don’t conflate the two representations of the same URL.
Mention it in llms.txt. Document both modes near the top of the file so agents can opt in on first read.
Extend it to aggregate views, not just individual pages. The same Accept: text/markdown
negotiation works for any HTML view backed by Markdown. This site ships all three modes on every spec page and also negotiates two aggregate URLs: /
returns llms.txt, and /checklist/ returns a copy-and-paste
Common mistakes #
- Shipping the suffix but forgetting
Content-Type: text/markdown
. Browsers will then download the file instead of displaying it, and agents may misclassify it. - Doing content negotiation without
Vary: Accept
. Caches will return whichever representation they cached first to every subsequent client, regardless of what they asked for. - Returning Markdown that still contains template includes, unresolved shortcodes, or framework-specific syntax (e.g.
{{ component }}
). Serve the fully-rendered Markdown, not the source-of-source. - Letting the Markdown drift from the HTML. The two should be generated from the same source at the same time, in the same build.
- Putting the
.md
behind authentication while the HTML is public. The two representations must have the same access policy.
Verification #
curl -i https://example.com/docs/getting-started.md
returns200
withContent-Type: text/markdown; charset=utf-8
.curl -i -H 'Accept: text/markdown' https://example.com/docs/getting-started/
returns Markdown withContent-Location
andVary: Accept
.- The HTML response to the same URL also carries
Vary: Accept
. - View source on any page — there is a
<link rel="alternate" type="text/markdown" …>
in<head>
. - The
.md
body contains the same headings, paragraphs, and code blocks as the rendered HTML, with frontmatter at the top.