Turn any website into clean Markdown for your RAG pipeline (with code)

A developer demonstrates how to convert any URL or entire website into clean Markdown for retrieval-augmented generation (RAG) pipelines using Apify's Website to Markdown tool. The tool uses real Chromium rendering and Mozilla Readability to extract main content, preserving headings, links, tables, and code, and can crawl sites via sitemaps. The post includes code examples for integrating the tool into a Node.js pipeline to fetch, chunk, and embed content into a vector store.

If you've built anything with retrieval-augmented generation RAG , you know the unglamorous truth: 80% of the work is getting clean text in. Your embeddings are only as good as the content you feed them, and raw web pages are a mess of nav bars, cookie banners, ads, share widgets, and <div soup. This post shows a fast, reliable way to convert any URL — or a whole site — into clean, LLM-ready Markdown , with code you can drop into a pipeline today. requests + BeautifulSoup? You can, and for a single static page it's fine: python import requests from bs4 import BeautifulSoup html = requests.get "https://example.com" .text text = BeautifulSoup html, "html.parser" .get text But this falls over fast in the real world: .get text flattens them.Readability + a headless browser + a Markdown converter solves the quality problem, but now you're maintaining browser infrastructure. That's the part worth outsourcing. Website to Markdown https://apify.com/runlayer/website-to-markdown runs the whole pipeline for you — real Chromium rendering → Mozilla Readability main-content extraction → Markdown with headings, links, tables and code preserved. It respects robots.txt , can crawl a site via its sitemap, and returns one clean record per page. curl -X POST "https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=YOUR APIFY TOKEN" \ -H "Content-Type: application/json" \ -d '{ "urls": "https://en.wikipedia.org/wiki/Retrieval-augmented generation" }' You get back: { "url": "https://en.wikipedia.org/wiki/Retrieval-augmented generation", "title": "Retrieval-augmented generation", "description": "…", "wordCount": 2841, "markdown": " Retrieval-augmented generation RAG is a technique that…" } Point it at the root, turn on sitemap discovery, and filter to the section you care about: { "urls": "https://docs.your-product.com" , "crawl": true, "useSitemap": true, "maxPages": 300, "includeUrlPatterns": " /docs/ " , "excludeUrlPatterns": " /changelog/ " } Here's the whole ingest step in Node — fetch clean Markdown, chunk it, embed it: js const runResp = await fetch "https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=" + process.env.APIFY TOKEN, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify { urls: "https://docs.your-product.com" , crawl: true, useSitemap: true, maxPages: 50, } , } ; const pages = await runResp.json ; // Chunk each page's markdown and embed for const page of pages { const chunks = chunkMarkdown page.markdown, { maxTokens: 500 } ; for const chunk of chunks { const embedding = await embed chunk ; await vectorStore.upsert { text: chunk, embedding, metadata: { url: page.url, title: page.title }, } ; } } Because the output is clean Markdown, your chunker splits on real headings instead of guessing, and your citations point at real page titles and URLs. For big crawls hundreds of pages , use the async endpoint POST /v2/acts/…/runs and poll the run, or run it on aSchedule— the run-sync- endpoint used above is best for a handful of pages, since it caps at ~5 minutes. Half of most companies' knowledge lives in PDFs and .docx files, not web pages. The companion PDF & DOCX to Markdown https://apify.com/runlayer/document-to-markdown actor handles those { "fileUrls": "https://example.com/whitepaper.pdf" , "outputFormat": "markdown" } Wrap the run in an Apify Schedule say, nightly and re-embed changed pages. Use the sinceDate pattern on feeds or a content hash to avoid re-embedding everything. It's pay-per-page with no subscription and no startup fee — you're only billed for pages that extract successfully, and free-tier Apify credits cover a generous amount of testing. For a few hundred docs pages that's cents, versus the ongoing cost of running and babysitting your own browser fleet. If you try it, I'd genuinely like feedback — what breaks, what's missing. Open an issue on the actor and it gets fixed fast. There's a whole suite of these screenshots, SEO/Core-Web-Vitals audits, tech-stack detection, feeds, job APIs at