Turning any website into clean Markdown chunks for RAG, without a headless browser A developer published a crawler that converts websites into clean Markdown chunks for RAG pipelines without a headless browser, using only fetch, cheerio and turndown. The approach fetches HTML, selects the main content via semantic selectors with a 200-character threshold, strips boilerplate such as navigation, cookie banners and link-heavy blocks, then converts to Markdown and chunks it. The crawler adds timeouts, backoff retries honoring Retry-After, a byte cap, robots.txt checks and roughly 250 ms spacing between requests to the same host. AI disclosure: I wrote this article with AI assistance Claude . The code comes from a small crawler we maintain, and I got every number in the output sections by running it locally on 2026-11-22. Most RAG pipelines start with "get the docs into the vector store", and that step often turns out worse than it should. Raw HTML is full of navigation menus, cookie banners, "Edit this page" links and footers. If you embed all of that, every chunk carries the same boilerplate, retrieval gets noisy, and you pay for tokens that add nothing. People often reach for a headless browser first. For a large share of the web, including most documentation sites, blogs and help centers, you don't need one. The HTML the server sends already contains the content. A plain fetch , an HTML parser and an HTML-to-Markdown converter will do, and the result is faster and cheaper, with fewer moving parts. This post walks through the approach with real code: fetch, pick the main content, remove the junk, convert to Markdown, and chunk. I also show the output from three public docs pages, and the cases where this approach fails. fetch for HTTP with a timeout, retries and a size cap cheerio turndown That's it. No Chromium download, no Playwright, and no 1 GB container image. The fetcher is ordinary, but a few details matter once you crawl more than one page: a timeout via AbortController , retries with backoff on 429 and 5xx honoring Retry-After , and a byte cap so one huge page can't use up your memory. The crawler also checks robots.txt and keeps about 250 ms between requests to the same host. js const ctrl = new AbortController ; const timer = setTimeout = ctrl.abort , timeoutMs ; const res = await fetch url, { redirect: 'follow', signal: ctrl.signal, headers: { 'user-agent': USER AGENT, accept: 'text/html,application/xhtml+xml;q=0.9, / ;q=0.5' }, } ; if res.status === 429 || res.status = 500 && attempt < retries { const ra = Number res.headers.get 'retry-after' ; await sleep Number.isFinite ra && ra 0 ? Math.min ra, 30 1000 : 1000 2 attempt ; continue; } Then cheerio.load body gives you a jQuery-like $ . Most sites wrap the actual content in something semantic. Try the usual candidates in order. If a selector matches several elements, keep the one with the most text, and only accept it if it has a meaningful amount of text: js function pickMain $ { const candidates = 'article', 'main', ' role="main" ', ' content', ' main', '.post-content', '.entry-content', '.article-body', '.content' ; for const sel of candidates { const els = $ sel ; if els.length continue; let best = null; let bestLen = 0; els.each , e = { const l = $ e .text .trim .length; if l bestLen { best = e; bestLen = l; } } ; if best && bestLen 200 return $ best ; } return $ 'body' ; } The 200-character threshold matters. Some pages have an empty