{"slug": "building-a-rag-chatbot-with-fastapi-and-chromadb-that-runs-locally-no-api-key", "title": "Building a RAG Chatbot with FastAPI and ChromaDB (that runs locally, no API key)", "summary": "A developer open-sourced a RAG chatbot starter template built with FastAPI and ChromaDB that runs entirely locally without an API key. The project demonstrates retrieval-augmented generation for document Q&A, using the same embedding model for indexing and querying to avoid common bugs. The template is available on GitHub under an MIT license.", "body_md": "Everyone wants a chatbot that can answer questions from *their own* documents — a company handbook, a contract, a research paper. The technique behind this is called **RAG (Retrieval-Augmented Generation)**.\n\nI build production RAG and LLM systems for a living, and I kept re-wiring the same foundation on every project. So I cleaned it up into a small, open-source starter. This post walks through how RAG actually works, the design decisions that matter, and how you can run the whole thing for free — no API key required.\n\nAt the end there's a link to the full open-source repo you can clone and run in minutes.\n\nAn LLM like GPT or Claude only knows what it was trained on. Ask it about a document sitting on your laptop and it will guess — or hallucinate.\n\nRAG fixes this with a simple idea: **retrieve the relevant information first, then let the LLM answer using only that.** The result is answers grounded in your actual documents, with citations you can trace back to a specific page.\n\n**Path 1 — Indexing (when a document comes in):**\n\n**Path 2 — Querying (when a user asks something):**\n\nAfter building a few of these, here's a stack that's easy to start with and holds up in production:\n\nOne detail people miss: you can test the whole pipeline **without any API key**, either in a retrieval-only mode that returns the matched chunks, or fully local with Ollama so nothing leaves your machine.\n\nHere's roughly what the query side looks like — embed the question, search, hand the chunks to the LLM:\n\n``` python\ndef answer_question(question: str, top_k: int = 3):\n    # 1. Embed the question with the same model used at indexing time\n    query_embedding = embed(question)\n\n    # 2. Retrieve the most similar chunks from the vector store\n    chunks = vector_store.search(query_embedding, top_k=top_k)\n\n    # 3. Build a prompt grounded in those chunks, then ask the LLM\n    context = \"\\n\\n\".join(c[\"text\"] for c in chunks)\n    answer = llm.generate(question=question, context=context)\n\n    return answer, chunks  # chunks double as citations\n```\n\nThe key constraint: **use the same embedding model for the question and the documents.** Vectors from different models aren't comparable, and this is a surprisingly common bug.\n\nFrom experience, whether a RAG system answers well comes down to a few things people underestimate:\n\nI packaged all of this into a starter template, open-sourced under MIT. Clone it, run one Docker command, and you have a working document Q&A chatbot with source citations. It's multilingual, and runs free with no API key.\n\n👉 **GitHub:** [https://github.com/panutpl/rag-chatbot-template-starter](https://github.com/panutpl/rag-chatbot-template-starter)\n\n```\ngit clone https://github.com/panutpl/rag-chatbot-template-starter\ncd rag-chatbot-template-starter\ncp .env.example .env\ndocker compose up --build\n```\n\nThen open `http://localhost:8000/docs`\n\n, upload a PDF, and start asking questions.\n\nIf you're building RAG or just getting started, I'd love feedback — especially on chunking and retrieval strategies, which I'm still tuning myself. Drop a comment about what's working for you.", "url": "https://wpnews.pro/news/building-a-rag-chatbot-with-fastapi-and-chromadb-that-runs-locally-no-api-key", "canonical_source": "https://dev.to/deaw_ai/building-a-rag-chatbot-with-fastapi-and-chromadb-that-runs-locally-no-api-key-36aj", "published_at": "2026-07-23 13:57:14+00:00", "updated_at": "2026-07-23 14:34:06.076091+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools"], "entities": ["FastAPI", "ChromaDB", "Ollama", "GitHub", "panutpl"], "alternates": {"html": "https://wpnews.pro/news/building-a-rag-chatbot-with-fastapi-and-chromadb-that-runs-locally-no-api-key", "markdown": "https://wpnews.pro/news/building-a-rag-chatbot-with-fastapi-and-chromadb-that-runs-locally-no-api-key.md", "text": "https://wpnews.pro/news/building-a-rag-chatbot-with-fastapi-and-chromadb-that-runs-locally-no-api-key.txt", "jsonld": "https://wpnews.pro/news/building-a-rag-chatbot-with-fastapi-and-chromadb-that-runs-locally-no-api-key.jsonld"}}