# Turning any website into clean Markdown chunks for RAG, without a headless browser

> Source: <https://dev.to/agebreakcoprog/turning-any-website-into-clean-markdown-chunks-for-rag-without-a-headless-browser-7al>
> Published: 2026-11-22 21:03:32+00:00

**Recommendation:** Skip the headless browser for server-rendered docs: fetch HTML, pick the largest semantic content block above 200 characters, strip nav and cookie junk, then convert to Markdown before chunking.
More in the playbook "Retrieval and RAG": <https://wpnews.pro/playbooks/rag-retrieval>

**AI disclosure:** I wrote this article with AI assistance (Claude). The code comes from a small crawler we maintain, and I got every number in the output sections by running it locally on 2026-11-22.

Most RAG pipelines start with "get the docs into the vector store", and that step often turns out worse than it should. Raw HTML is full of navigation menus, cookie banners, "Edit this page" links and footers. If you embed all of that, every chunk carries the same boilerplate, retrieval gets noisy, and you pay for tokens that add nothing.

People often reach for a headless browser first. For a large share of the web, including most documentation sites, blogs and help centers, you don't need one. The HTML the server sends already contains the content. A plain `fetch`, an HTML parser and an HTML-to-Markdown converter will do, and the result is faster and cheaper, with fewer moving parts.

This post walks through the approach with real code: fetch, pick the main content, remove the junk, convert to Markdown, and chunk. I also show the output from three public docs pages, and the cases where this approach fails.

`fetch` for HTTP (with a timeout, retries and a size cap)`cheerio`` turndown`
That's it. No Chromium download, no Playwright, and no 1 GB container image.

The fetcher is ordinary, but a few details matter once you crawl more than one page: a timeout via `AbortController`, retries with backoff on `429` and `5xx` (honoring `Retry-After`), and a byte cap so one huge page can't use up your memory. The crawler also checks `robots.txt` and keeps about 250 ms between requests to the same host.

``` js
const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), timeoutMs);
const res = await fetch(url, {
  redirect: 'follow',
  signal: ctrl.signal,
  headers: { 'user-agent': USER_AGENT, accept: 'text/html,application/xhtml+xml;q=0.9,*/*;q=0.5' },
});
if ((res.status === 429 || res.status >= 500) && attempt < retries) {
  const ra = Number(res.headers.get('retry-after'));
  await sleep(Number.isFinite(ra) && ra > 0 ? Math.min(ra, 30) * 1000 : 1000 * 2 ** attempt);
  continue;
}
```

Then `cheerio.load(body)` gives you a jQuery-like `$`.

Most sites wrap the actual content in something semantic. Try the usual candidates in order. If a selector matches several elements, keep the one with the most text, and only accept it if it has a meaningful amount of text:

``` js
function pickMain($) {
  const candidates = ['article', 'main', '[role="main"]', '#content', '#main',
    '.post-content', '.entry-content', '.article-body', '.content'];
  for (const sel of candidates) {
    const els = $(sel);
    if (!els.length) continue;
    let best = null; let bestLen = 0;
    els.each((_, e) => { const l = $(e).text().trim().length; if (l > bestLen) { best = e; bestLen = l; } });
    if (best && bestLen > 200) return $(best);
  }
  return $('body');
}
```

The 200-character threshold matters. Some pages have an empty `<article>` used as a card wrapper, and without the check you would get an empty result.

Before picking the root, remove elements that are never content. After picking it, remove the layout parts (`nav`, `header`, `footer`, `aside`) that live inside the main column:

``` js
const JUNK = 'script,style,noscript,template,iframe,svg,canvas,form,button,input,select,textarea,dialog,'
  + '[role="navigation"],[role="banner"],[role="contentinfo"],[aria-hidden="true"],[hidden],'
  + '.cookie,.cookies,#cookie-banner,.cookie-banner,.consent,.newsletter,.advert,.ads,.ad,.share,.social,.breadcrumbs,.breadcrumb';

$(JUNK).remove();
const root = pickMain($);
root.find('nav,header,footer,aside').remove();
root.find('.mw-editsection,.hash-link,a.anchor,.headerlink,.toc,#toc,.sr-only,.visually-hidden').remove();
```

One more heuristic catches what the selectors miss, like in-content menus, language lists and tag clouds. If a block is mostly link text, has at least 8 links, and is less than half of the page, drop it:

``` js
const rootLen = root.text().replace(/\s+/g, ' ').trim().length || 1;
root.find('ul,ol,div,section,table').each((_, e) => {
  const el = $(e);
  const t = el.text().replace(/\s+/g, ' ').trim();
  if (t.length < 80) return;
  const lt = el.find('a').text().replace(/\s+/g, ' ').trim().length;
  if (lt / t.length > 0.75 && el.find('a').length >= 8 && t.length < rootLen * 0.5) el.remove();
});
```

The "less than half of the page" condition keeps the heuristic from deleting the content of an index page that really is a list of links.

Finally, make every `href` and `src` absolute with `new URL(href, finalUrl)`. A chunk that says `[see here](../guide)` is useless once it's sitting in a vector store.

Turndown does most of the work. Two custom rules made a real difference for docs:

``` js
const td = new TurndownService({ headingStyle: 'atx', codeBlockStyle: 'fenced', bulletListMarker: '-' });

// Docs sites wrap <pre> in syntax-highlighting spans; use the raw text instead.
td.addRule('pre', {
  filter: (n) => n.nodeName === 'PRE',
  replacement: (_c, n) => `\n\n\`\`\`\n${n.textContent.replace(/\n+$/, '')}\n\`\`\`\n\n`,
});

// Turndown core drops table structure; emit simple GFM tables.
td.addRule('tableCell', { filter: ['th', 'td'], replacement: (c) => ` ${c.replace(/\n+/g, ' ').trim()} |` });
```

(The full version also has rules for table rows that add a `| --- |` separator after the header row.)

If the result has no `#` heading near the top, prepend the page title. That way the first chunk always says what document it came from.

There are many chunking strategies. This one is deliberately simple: split on blank lines, pack paragraphs into a chunk until the next one would exceed the budget, and carry a small overlap into the next chunk.

``` js
const estimateTokens = (s) => Math.ceil(s.length / 4);

function chunk(md, size, overlap) {
  const paras = md.split(/\n{2,}/);
  const chunks = [];
  let cur = '';
  for (const p of paras) {
    if (cur && estimateTokens(cur + '\n\n' + p) > size) {
      chunks.push(cur);
      const tail = overlap ? cur.slice(-overlap * 4) : '';
      cur = tail ? tail + '\n\n' + p : p;
    } else cur = cur ? cur + '\n\n' + p : p;
  }
  if (cur.trim()) chunks.push(cur);
  return chunks.map((text, i) => ({ index: i, text, tokenEstimate: estimateTokens(text) }));
}
```

The token estimate is "4 characters per token". It's close enough for English prose when you're sizing chunks, but it's not your embedding model's tokenizer. If you're close to a hard model limit, count with the real tokenizer.

I ran it on three docs pages and one single-page app, with `chunkSizeTokens: 800` and `chunkOverlapTokens: 50`:

```
{
  "startUrls": [
    { "url": "https://docs.python.org/3/tutorial/classes.html" },
    { "url": "https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching" },
    { "url": "https://nodejs.org/api/fs.html" },
    { "url": "https://app.netlify.com/" }
  ],
  "chunkSizeTokens": 800,
  "chunkOverlapTokens": 50
}
```

The whole run, including Node startup, took 2.3 seconds on a laptop.

| Page | Words | Token estimate | Chunks | Largest chunk | 
|---|---|---|---|---|
| Python tutorial: Classes | 5,476 | 9,775 | 14 | 800 | 
| MDN: HTTP caching | 5,394 | 10,061 | 14 | 800 | 
| Node.js: File system API | 39,815 | 108,659 | 165 | 1,663 | 
| app.netlify.com | 2 | (skipped) | 0 | n/a | 

The start of the MDN page's first chunk, with no nav, sidebar or footer:

```
# HTTP caching

The HTTP cache stores a response associated with a request and reuses the stored response for subsequent requests.

There are several advantages to reusability. First, since there is no need to deliver the request to the origin server, then the closer the client and cache are, the faster the response will be. ...
```

And the item for the Netlify app, which renders everything client-side:

```
{
  "url": "https://app.netlify.com/",
  "statusCode": 200,
  "error": "Only 2 words extracted. The page may need JavaScript rendering.",
  "markdown": "# Netlify"
}
```

That check (`wordCount < minWords`, default 20) is the most useful line in the whole pipeline. Without it, a JavaScript-rendered page returns HTTP 200 and an almost empty document, and that empty document quietly goes into your index.

Looking closely at the output, there are some real flaws you should know about if you copy this:

`fs` page, 16 of 165 chunks had a code fence opened in one chunk and closed in the next. The fix is to tokenize fenced blocks as one unit before splitting.`overlap * 4` characters of the previous chunk, so it often starts mid-word (`proxy cache" implementations that...`) and sometimes inside a code block. Overlapping by whole sentences or paragraphs would be cleaner.`[#](...)` permalink anchors next to each heading, and MDN code blocks keep a stray language label (`http`) on the line above the fence. Neither breaks retrieval, but both are noise. Per-site cleanup rules fix them.`HTTP caching > Freshness > max-age`). That's the next thing I'd add.` pickMain` and the link-density filter work well on docs, blogs and help centers. On unusual layouts they can keep a sidebar or, more rarely, cut a list you wanted. Spot-check a few pages per site before you index thousands.
For server-rendered docs, though, this covers most of what a RAG ingestion step needs, in a few hundred lines and a couple of seconds per batch.

We packaged this exact code (with the crawl mode, include/exclude URL patterns, depth limits and `robots.txt` handling) as an Apify actor: [Website to Markdown for LLM & RAG](https://apify.com/agebreak/website-to-markdown). **Full disclosure: it's ours.** It's free to use until December 6, 2026, and after that it costs about $1 per 1,000 pages converted. Skipped and failed pages aren't charged. It has the same limits described above, including no JavaScript rendering.

If you only need a few pages, the snippets above are most of it. Copy them and adjust the selectors for your site.
