How to Build a RAG Knowledge Base from Any Documentation Site in 5 Minutes

You want to feed documentation into your RAG pipeline, but web scraping gives you a mess of navigation, sidebars, cookie banners, and broken formatting mixed with actual content. You spend hours cleaning up HTML before you can even start building your knowledge base.

I built an automated extraction + chunking pipeline that converts any documentation site into clean, structured markdown ready for your vector store.

Using the RAG Docs Extractor on Apify, you can crawl any docs site and get chunked output with a single API call:

{
  "startUrl": "https://fastapi.tiangolo.com/",
  "maxPages": 100,
  "chunkByHeading": true
}

Each chunk in the output looks like:

{
  "url": "https://fastapi.tiangolo.com/tutorial/first-steps/",
  "title": "First Steps - FastAPI",
  "heading": "Create a FastAPI instance",
  "content": "## Create a FastAPI instance\n\nThe simplest FastAPI file could look like this...\n\n```

python\nfrom fastapi import FastAPI\n\napp = FastAPI()\n

```",
  "token_count": 245
}

Notice the token_count

field — it uses cl100k_base encoding (GPT-4 / modern embedding models), so you know exactly how many tokens each chunk costs before embedding.

With LangChain and ChromaDB:

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain.schema import Document
import json

with open("dataset.json") as f:
    chunks = json.load(f)

docs = [
    Document(
        page_content=chunk["content"],
        metadata={
            "url": chunk["url"],
            "title": chunk["title"],
            "heading": chunk.get("heading", ""),
            "token_count": chunk["token_count"],
        }
    )
    for chunk in chunks
]

vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
print(f"Indexed {len(docs)} chunks")

No re-tokenization needed — the token counts are already computed.

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA

llm = ChatOpenAI(model="gpt-4")
qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)

result = qa.invoke("How do I add authentication to a FastAPI app?")
print(result["result"])

If you just need to convert individual pages to markdown (no chunking), use Website to Markdown instead:

{
  "startUrl": "https://docs.python.org/3/library/asyncio.html",
  "maxPages": 1
}

Output is clean markdown with token counts. Good for when you want to control your own chunking strategy or feed single pages into an LLM context window.

Under the hood, the extractor:

<nav>

, <footer>

, .sidebar

, .cookie-banner

, <script>

, <style>

, and 20+ other noise selectors<article>

, <main>

, .markdown-body

, .prose

, etc.The result is clean, structured content that's ready for any RAG pipeline.

Both are open on the Apify Store with pay-per-result pricing. No subscription needed.