{"slug": "turn-any-website-into-clean-markdown-for-your-rag-pipeline-with-code", "title": "Turn any website into clean Markdown for your RAG pipeline (with code)", "summary": "A developer demonstrates how to convert any URL or entire website into clean Markdown for retrieval-augmented generation (RAG) pipelines using Apify's Website to Markdown tool. The tool uses real Chromium rendering and Mozilla Readability to extract main content, preserving headings, links, tables, and code, and can crawl sites via sitemaps. The post includes code examples for integrating the tool into a Node.js pipeline to fetch, chunk, and embed content into a vector store.", "body_md": "If you've built anything with retrieval-augmented generation (RAG), you know the\n\nunglamorous truth: **80% of the work is getting clean text in.** Your embeddings\n\nare only as good as the content you feed them, and raw web pages are a mess of\n\nnav bars, cookie banners, ads, share widgets, and `<div>`\n\nsoup.\n\nThis post shows a fast, reliable way to convert **any URL — or a whole site —\ninto clean, LLM-ready Markdown**, with code you can drop into a pipeline today.\n\n`requests`\n\n+ BeautifulSoup?\nYou can, and for a single static page it's fine:\n\n``` python\nimport requests\nfrom bs4 import BeautifulSoup\n\nhtml = requests.get(\"https://example.com\").text\ntext = BeautifulSoup(html, \"html.parser\").get_text()\n```\n\nBut this falls over fast in the real world:\n\n`.get_text()`\n\nflattens them.Readability + a headless browser + a Markdown converter solves the quality\n\nproblem, but now you're maintaining browser infrastructure. That's the part\n\nworth outsourcing.\n\n[Website to Markdown](https://apify.com/runlayer/website-to-markdown) runs the\n\nwhole pipeline for you — real Chromium rendering → Mozilla Readability\n\n(main-content extraction) → Markdown with headings, links, tables and code\n\npreserved. It respects `robots.txt`\n\n, can crawl a site via its sitemap, and\n\nreturns one clean record per page.\n\n```\ncurl -X POST \"https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{ \"urls\": [\"https://en.wikipedia.org/wiki/Retrieval-augmented_generation\"] }'\n```\n\nYou get back:\n\n```\n[{\n  \"url\": \"https://en.wikipedia.org/wiki/Retrieval-augmented_generation\",\n  \"title\": \"Retrieval-augmented generation\",\n  \"description\": \"…\",\n  \"wordCount\": 2841,\n  \"markdown\": \"**Retrieval-augmented generation (RAG)** is a technique that…\"\n}]\n```\n\nPoint it at the root, turn on sitemap discovery, and filter to the section you\n\ncare about:\n\n```\n{\n  \"urls\": [\"https://docs.your-product.com\"],\n  \"crawl\": true,\n  \"useSitemap\": true,\n  \"maxPages\": 300,\n  \"includeUrlPatterns\": [\"*/docs/*\"],\n  \"excludeUrlPatterns\": [\"*/changelog/*\"]\n}\n```\n\nHere's the whole ingest step in Node — fetch clean Markdown, chunk it, embed it:\n\n``` js\nconst runResp = await fetch(\n  \"https://api.apify.com/v2/acts/runlayer~website-to-markdown/run-sync-get-dataset-items?token=\" + process.env.APIFY_TOKEN,\n  {\n    method: \"POST\",\n    headers: { \"Content-Type\": \"application/json\" },\n    body: JSON.stringify({\n      urls: [\"https://docs.your-product.com\"],\n      crawl: true,\n      useSitemap: true,\n      maxPages: 50,\n    }),\n  }\n);\nconst pages = await runResp.json();\n\n// Chunk each page's markdown and embed\nfor (const page of pages) {\n  const chunks = chunkMarkdown(page.markdown, { maxTokens: 500 });\n  for (const chunk of chunks) {\n    const embedding = await embed(chunk);\n    await vectorStore.upsert({\n      text: chunk,\n      embedding,\n      metadata: { url: page.url, title: page.title },\n    });\n  }\n}\n```\n\nBecause the output is clean Markdown, your chunker splits on real headings\n\ninstead of guessing, and your citations point at real page titles and URLs.\n\nFor big crawls (hundreds of pages), use the async endpoint (\n\n`POST /v2/acts/…/runs`\n\n)\n\nand poll the run, or run it on aSchedule— the`run-sync-*`\n\nendpoint used\n\nabove is best for a handful of pages, since it caps at ~5 minutes.\n\nHalf of most companies' knowledge lives in PDFs and `.docx`\n\nfiles, not web\n\npages. The companion [PDF & DOCX to\nMarkdown](https://apify.com/runlayer/document-to-markdown) actor handles those\n\n```\n{ \"fileUrls\": [\"https://example.com/whitepaper.pdf\"], \"outputFormat\": \"markdown\" }\n```\n\nWrap the run in an Apify **Schedule** (say, nightly) and re-embed changed pages.\n\nUse the `sinceDate`\n\npattern on feeds or a content hash to avoid re-embedding\n\neverything.\n\nIt's pay-per-page with no subscription and no startup fee — you're only billed\n\nfor pages that extract successfully, and free-tier Apify credits cover a\n\ngenerous amount of testing. For a few hundred docs pages that's cents, versus\n\nthe ongoing cost of running and babysitting your own browser fleet.\n\n*If you try it, I'd genuinely like feedback — what breaks, what's missing. Open\nan issue on the actor and it gets fixed fast. There's a whole suite of these\n(screenshots, SEO/Core-Web-Vitals audits, tech-stack detection, feeds, job APIs)\nat *", "url": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-with-code", "canonical_source": "https://dev.to/quantoracle/turn-any-website-into-clean-markdown-for-your-rag-pipeline-with-code-34oe", "published_at": "2026-07-12 16:18:30+00:00", "updated_at": "2026-07-12 16:45:08.827898+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Apify", "Mozilla Readability", "Chromium", "BeautifulSoup", "Node.js"], "alternates": {"html": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-with-code", "markdown": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-with-code.md", "text": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-with-code.txt", "jsonld": "https://wpnews.pro/news/turn-any-website-into-clean-markdown-for-your-rag-pipeline-with-code.jsonld"}}