Turn any website into clean Markdown for your RAG pipeline (with code)

If you've built anything with retrieval-augmented generation (RAG), you know the

unglamorous truth: 80% of the work is getting clean text in. Your embeddings

are only as good as the content you feed them, and raw web pages are a mess of

nav bars, cookie banners, ads, share widgets, and <div>

soup.

This post shows a fast, reliable way to convert any URL — or a whole site — into clean, LLM-ready Markdown, with code you can drop into a pipeline today.

requests

• BeautifulSoup? You can, and for a single static page it's fine:

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com").text
text = BeautifulSoup(html, "html.parser").get_text()

But this falls over fast in the real world:

.get_text()

flattens them.Readability + a headless browser + a Markdown converter solves the quality

problem, but now you're maintaining browser infrastructure. That's the part

worth outsourcing.

Website to Markdown runs the

whole pipeline for you — real Chromium rendering → Mozilla Readability

(main-content extraction) → Markdown with headings, links, tables and code

preserved. It respects robots.txt

, can crawl a site via its sitemap, and

returns one clean record per page.

curl -X POST "https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"] }'

You get back:

[{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "title": "Retrieval-augmented generation",
  "description": "…",
  "wordCount": 2841,
  "markdown": "**Retrieval-augmented generation (RAG)** is a technique that…"
}]

Point it at the root, turn on sitemap discovery, and filter to the section you

care about:

{
  "urls": ["https://docs.your-product.com"],
  "crawl": true,
  "useSitemap": true,
  "maxPages": 300,
  "includeUrlPatterns": ["*/docs/*"],
  "excludeUrlPatterns": ["*/changelog/*"]
}

Here's the whole ingest step in Node — fetch clean Markdown, chunk it, embed it:

const runResp = await fetch(
  "https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=" + process.env.APIFY_TOKEN,
  {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      urls: ["https://docs.your-product.com"],
      crawl: true,
      useSitemap: true,
      maxPages: 50,
    }),
  }
);
const pages = await runResp.json();

// Chunk each page's markdown and embed
for (const page of pages) {
  const chunks = chunkMarkdown(page.markdown, { maxTokens: 500 });
  for (const chunk of chunks) {
    const embedding = await embed(chunk);
    await vectorStore.upsert({
      text: chunk,
      embedding,
      metadata: { url: page.url, title: page.title },
    });
  }
}

Because the output is clean Markdown, your chunker splits on real headings

instead of guessing, and your citations point at real page titles and URLs.

For big crawls (hundreds of pages), use the async endpoint (

POST /v2/acts/…/runs

)

and poll the run, or run it on aSchedule— therun-sync-*

endpoint used

above is best for a handful of pages, since it caps at ~5 minutes.

Half of most companies' knowledge lives in PDFs and .docx

files, not web

pages. The companion PDF & DOCX to Markdown actor handles those

{ "fileUrls": ["https://example.com/whitepaper.pdf"], "outputFormat": "markdown" }

Wrap the run in an Apify Schedule (say, nightly) and re-embed changed pages.

Use the sinceDate

pattern on feeds or a content hash to avoid re-embedding

everything.

It's pay-per-page with no subscription and no startup fee — you're only billed

for pages that extract successfully, and free-tier Apify credits cover a

generous amount of testing. For a few hundred docs pages that's cents, versus

the ongoing cost of running and babysitting your own browser fleet.

*If you try it, I'd genuinely like feedback — what breaks, what's missing. Open an issue on the actor and it gets fixed fast. There's a whole suite of these (screenshots, SEO/Core-Web-Vitals audits, tech-stack detection, feeds, job APIs) at *