{"slug": "turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code", "title": "Turn a docs site into RAG-ready Markdown chunks in 5 minutes (no scraper code)", "summary": "A developer built a hosted tool, the Web Page to Markdown Actor on Apify, that converts documentation sites into Markdown and cuts them into roughly 400-token chunks tagged with their heading path, aimed at RAG pipelines. A walkthrough shows a FastAPI docs sitemap being converted and loaded into a local Chroma vector store in about 25 lines of Python, with a five-page test returning in 2.4 seconds without a browser. The tool costs $1 per 1,000 pages converted, and failed pages are not charged.", "body_md": "You want a chatbot that answers questions about a library's documentation. The hard part is rarely the LLM. It is getting the docs out of HTML and into clean, well-cut chunks: no navigation menus, no cookie banners, no footers repeated on every page, and chunks that keep track of which section they came from.\n\nThis walkthrough takes the FastAPI documentation, converts it to Markdown, cuts it into ~400-token chunks with their heading path, and loads them into a local vector store. Total code: about 25 lines of Python.\n\n`pip install apify-client chromadb`\nWe use the [Web Page to Markdown](https://apify.com/blueberry_rainstorm/web-page-to-markdown) Actor for the conversion. Disclosure: I built it. It costs $1 per 1,000 pages converted, and failed pages are free. Everything below also works with any tool that gives you clean Markdown plus heading-aware chunks; this one just does both in one call.\n\nMost docs sites publish a `sitemap.xml`. Giving the sitemap instead of a list of URLs means new pages are picked up automatically the next time you run it.\n\n``` python\nimport os\nfrom apify_client import ApifyClient\n\nclient = ApifyClient(os.environ[\"APIFY_TOKEN\"])\n\nrun = client.actor(\"blueberry_rainstorm/web-page-to-markdown\").call(run_input={\n    \"urls\": [],                       # only the sitemap this time\n    \"sitemapUrls\": [{\"url\": \"https://fastapi.tiangolo.com/sitemap.xml\"}],\n    \"maxPages\": 50,                   # keep the first test small\n    \"contentMode\": \"main\",            # drop menus, sidebars and footers\n    \"chunkSize\": 400,                 # ~tokens per chunk\n    \"chunkOverlap\": 50,\n})\npages = list(client.dataset(run[\"defaultDatasetId\"]).iterate_items())\nprint(len(pages), \"pages\")\n```\n\nOn a test run with `maxPages: 5`, the five pages came back in 2.4 seconds without a browser, because the FastAPI docs are static HTML. JavaScript-only pages are detected and re-rendered in Chrome automatically.\n\nEach page is one item with metadata, the full Markdown, and the chunks:\n\n```\n{\n  \"url\": \"https://fastapi.tiangolo.com/\",\n  \"status\": \"ok\",\n  \"title\": \"FastAPI - FastAPI\",\n  \"description\": \"FastAPI framework, high performance, easy to learn, fast to code, ready for production\",\n  \"language\": \"en\",\n  \"canonicalUrl\": \"https://fastapi.tiangolo.com/\",\n  \"wordCount\": 2134,\n  \"approxTokens\": 4139,\n  \"renderedWithBrowser\": false,\n  \"markdown\": \"# FastAPI - FastAPI\\n\\n*FastAPI framework, high performance, easy to learn...\",\n  \"chunks\": [\n    { \"index\": 0, \"headingPath\": \"FastAPI - FastAPI\", \"text\": \"...\", \"approxTokens\": 400 }\n  ]\n}\n```\n\nTwo fields matter most for RAG:\n\n`headingPath`` Concurrency and async / await - FastAPI > In a hurry?`). Store it as metadata, and prepend it to the chunk text before embedding. A chunk that starts \"Use `async def` when...\" is much easier to retrieve when the embedding also knows it sits under \"Concurrency and async / await\".`approxTokens`\nChroma runs locally and embeds with a small default model, so there is nothing else to sign up for.\n\n``` python\nimport chromadb\n\ndb = chromadb.PersistentClient(path=\"./fastapi-index\")\ncol = db.get_or_create_collection(\"fastapi-docs\")\n\nids, docs, metas = [], [], []\nfor page in pages:\n    if page[\"status\"] != \"ok\":\n        continue                       # failed pages are listed, not charged\n    for chunk in page.get(\"chunks\") or []:\n        ids.append(f'{page[\"url\"]}#{chunk[\"index\"]}')\n        docs.append(f'{chunk[\"headingPath\"]}\\n\\n{chunk[\"text\"]}')\n        metas.append({\"url\": page[\"url\"], \"title\": page.get(\"title\") or \"\",\n                      \"section\": chunk[\"headingPath\"]})\n\ncol.upsert(ids=ids, documents=docs, metadatas=metas)\nprint(col.count(), \"chunks indexed\")\n```\n\nUsing the page URL plus the chunk index as the id makes re-runs safe: `upsert` replaces chunks instead of duplicating them.\n\n```\nhits = col.query(query_texts=[\"When should I use async def instead of def?\"], n_results=3)\nfor meta in hits[\"metadatas\"][0]:\n    print(meta[\"section\"], \"->\", meta[\"url\"])\n```\n\nPass the top chunks, with their URLs, to the LLM of your choice and ask it to cite the URL. Because each chunk carries its source page and section, the answer can link back to the exact part of the docs.\n\n`\"contentMode\": \"main\"` is right for docs, blogs and articles. For homepages, product pages and listings, use `\"full\"` so nothing important is cut.`removeSelectors`, for example `.related-posts, #comments, .edit-this-page`.` maxConcurrency` modest on small sites.\n50 pages at $1 per 1,000 pages is $0.05. The Chroma part runs on your machine for free.\n\nIf you try this on another docs site and something comes out messy (tables, code blocks, odd navigation), tell me which site in the Actor's Issues tab and I will look at it.", "url": "https://wpnews.pro/news/turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code", "canonical_source": "https://dev.to/rabii_slimane_22e39df3bdd/turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code-59j", "published_at": "2026-11-12 23:06:57+00:00", "updated_at": "2026-11-12 23:19:36.253534+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools", "ai-products"], "entities": ["Apify", "FastAPI", "Chroma", "Web Page to Markdown"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code", "markdown": "https://wpnews.pro/news/turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code.md", "text": "https://wpnews.pro/news/turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code.txt", "jsonld": "https://wpnews.pro/news/turn-a-docs-site-into-rag-ready-markdown-chunks-in-5-minutes-no-scraper-code.jsonld"}}