Turn a docs site into RAG-ready Markdown chunks in 5 minutes (no scraper code)

A developer built a hosted tool, the Web Page to Markdown Actor on Apify, that converts documentation sites into Markdown and cuts them into roughly 400-token chunks tagged with their heading path, aimed at RAG pipelines. A walkthrough shows a FastAPI docs sitemap being converted and loaded into a local Chroma vector store in about 25 lines of Python, with a five-page test returning in 2.4 seconds without a browser. The tool costs $1 per 1,000 pages converted, and failed pages are not charged.

You want a chatbot that answers questions about a library's documentation. The hard part is rarely the LLM. It is getting the docs out of HTML and into clean, well-cut chunks: no navigation menus, no cookie banners, no footers repeated on every page, and chunks that keep track of which section they came from. This walkthrough takes the FastAPI documentation, converts it to Markdown, cuts it into ~400-token chunks with their heading path, and loads them into a local vector store. Total code: about 25 lines of Python. pip install apify-client chromadb We use the Web Page to Markdown https://apify.com/blueberry rainstorm/web-page-to-markdown Actor for the conversion. Disclosure: I built it. It costs $1 per 1,000 pages converted, and failed pages are free. Everything below also works with any tool that gives you clean Markdown plus heading-aware chunks; this one just does both in one call. Most docs sites publish a sitemap.xml . Giving the sitemap instead of a list of URLs means new pages are picked up automatically the next time you run it. python import os from apify client import ApifyClient client = ApifyClient os.environ "APIFY TOKEN" run = client.actor "blueberry rainstorm/web-page-to-markdown" .call run input={ "urls": , only the sitemap this time "sitemapUrls": {"url": "https://fastapi.tiangolo.com/sitemap.xml"} , "maxPages": 50, keep the first test small "contentMode": "main", drop menus, sidebars and footers "chunkSize": 400, ~tokens per chunk "chunkOverlap": 50, } pages = list client.dataset run "defaultDatasetId" .iterate items print len pages , "pages" On a test run with maxPages: 5 , the five pages came back in 2.4 seconds without a browser, because the FastAPI docs are static HTML. JavaScript-only pages are detected and re-rendered in Chrome automatically. Each page is one item with metadata, the full Markdown, and the chunks: { "url": "https://fastapi.tiangolo.com/", "status": "ok", "title": "FastAPI - FastAPI", "description": "FastAPI framework, high performance, easy to learn, fast to code, ready for production", "language": "en", "canonicalUrl": "https://fastapi.tiangolo.com/", "wordCount": 2134, "approxTokens": 4139, "renderedWithBrowser": false, "markdown": " FastAPI - FastAPI\n\n FastAPI framework, high performance, easy to learn...", "chunks": { "index": 0, "headingPath": "FastAPI - FastAPI", "text": "...", "approxTokens": 400 } } Two fields matter most for RAG: headingPath Concurrency and async / await - FastAPI In a hurry? . Store it as metadata, and prepend it to the chunk text before embedding. A chunk that starts "Use async def when..." is much easier to retrieve when the embedding also knows it sits under "Concurrency and async / await". approxTokens Chroma runs locally and embeds with a small default model, so there is nothing else to sign up for. python import chromadb db = chromadb.PersistentClient path="./fastapi-index" col = db.get or create collection "fastapi-docs" ids, docs, metas = , , for page in pages: if page "status" = "ok": continue failed pages are listed, not charged for chunk in page.get "chunks" or : ids.append f'{page "url" } {chunk "index" }' docs.append f'{chunk "headingPath" }\n\n{chunk "text" }' metas.append {"url": page "url" , "title": page.get "title" or "", "section": chunk "headingPath" } col.upsert ids=ids, documents=docs, metadatas=metas print col.count , "chunks indexed" Using the page URL plus the chunk index as the id makes re-runs safe: upsert replaces chunks instead of duplicating them. hits = col.query query texts= "When should I use async def instead of def?" , n results=3 for meta in hits "metadatas" 0 : print meta "section" , "- ", meta "url" Pass the top chunks, with their URLs, to the LLM of your choice and ask it to cite the URL. Because each chunk carries its source page and section, the answer can link back to the exact part of the docs. "contentMode": "main" is right for docs, blogs and articles. For homepages, product pages and listings, use "full" so nothing important is cut. removeSelectors , for example .related-posts, comments, .edit-this-page . maxConcurrency modest on small sites. 50 pages at $1 per 1,000 pages is $0.05. The Chroma part runs on your machine for free. If you try this on another docs site and something comes out messy tables, code blocks, odd navigation , tell me which site in the Actor's Issues tab and I will look at it.