I Built a Local RAG Assistant with Ollama, ChromaDB and LangChain. Here's What I Learned

A developer built a fully local RAG assistant using Ollama, ChromaDB, and LangChain to help technicians query PDF manuals without sending data to the cloud. The system ingests 2,111 pages into 9,669 chunks and runs entirely via Docker Compose, ensuring sensitive financial data never leaves local infrastructure.

During my internship at a software services company, I noticed a recurring problem: technicians spent hours searching through hundreds of pages of PDF manuals to answer client questions. The company deploys a suite of management tools accounting, HR, logistics mainly to public institutions managing projects funded by international donors. I kept thinking: this is exactly the kind of problem LLMs were made for. So for my Master's project in AI & Big Data, I built an intelligent assistant to handle it. The constraint that shaped every technical decision: no data could leave the local infrastructure . These institutions handle sensitive financial data it cannot be sent to OpenAI, Anthropic, or any cloud provider. This forced me into the world of fully local LLMs. Here's what I built, what broke, and what I learned. The company had accumulated massive technical documentation PDF manuals, configuration guides, FAQ files but no efficient way to query it. When a technician faced an issue with the accounting module or the HR module, they searched manually. Slow, inconsistent, and dependent on individual expertise. Classic LLMs can't help here: they don't know the private internal documentation. And without grounding in official docs, they hallucinate procedures dangerous when dealing with accounting operations for donor-funded projects. The solution: RAG Retrieval-Augmented Generation make the LLM answer only from the official documentation, injected at query time. Before the code, the concept in one sentence: instead of asking the LLM to "know" everything, you retrieve the relevant passages from your documents and inject them into the prompt as context. The pipeline has two distinct phases: Phase 1 : Ingestion done once PDF documents ↓ Chunking split into 300-character blocks ↓ Embedding convert each block to a vector ↓ ChromaDB store all vectors Phase 2 : Query at each question User question ↓ Embed the question same model ↓ ChromaDB similarity search → top 3 chunks ↓ Inject chunks + question into LLM prompt ↓ Llama 3 generates a grounded answer The key insight: the LLM never "knows" your documents. It reads them fresh at each query, from the context you provide. No fine-tuning needed. No retraining when docs are updated. Everything runs locally via Docker Compose. Four services, each with a clear role: | Service | Role | Port | |---|---|---| Ollama | Runs Llama 3 locally | 11434 | ChromaDB | Vector database | 8001 | FastAPI | RAG pipeline + REST API | 8000 | Streamlit | User interface | 8501 | docker-compose.yml simplified services: ollama: image: ollama/ollama:latest ports: "11434:11434" volumes: "./ollama models:/root/.ollama" environment: - OLLAMA KEEP ALIVE=24h chromadb: image: chromadb/chroma:latest ports: "8001:8000" volumes: "./chroma db:/chroma/chroma" python-api: build: ./src ports: "8000:8000" depends on: ollama, chromadb environment: - OLLAMA BASE URL=http://ollama:11434 - CHROMA HOST=chromadb - CHROMA PORT=8000 I ingested 10 documents accounting references SYSCOHADA, SYCEBNL , a general accounting code, and internal user manuals. That's 2,111 pages split into 9,669 chunks . python src/ingestion.py import os import chromadb from langchain community.document loaders import PyPDFLoader from langchain text splitters import RecursiveCharacterTextSplitter from langchain chroma import Chroma from langchain community.embeddings import HuggingFaceEmbeddings DATA DIR = "/app/data/documents" COLLECTION NAME = "documentation" def load documents : documents = for fichier in os.listdir DATA DIR : if fichier.endswith ".pdf" : try: loader = PyPDFLoader os.path.join DATA DIR, fichier docs = loader.load Tag each chunk with its source document for doc in docs: doc.metadata "module" = fichier.replace ".pdf", "" documents.extend docs except Exception as e: print f"Skipped {fichier}: {e}" Don't crash on encrypted PDFs return documents def store embeddings chunks : embeddings = HuggingFaceEmbeddings model name="all-MiniLM-L6-v2", model kwargs={"device": "cpu"} Explicit HTTP connection — not localhost, the Docker service name client = chromadb.HttpClient host="chromadb", port=8000 Chroma.from documents documents=chunks, embedding=embeddings, collection name=COLLECTION NAME, client=client print f"Done — {len chunks } vectors stored" Two things worth noting: try/except is not optional one encrypted PDF it happened would have crashed the entire ingestion without it. Install cryptography if you hit an AES error: pip install cryptography python src/rag chain.py import chromadb from langchain community.embeddings import HuggingFaceEmbeddings from langchain community.llms import Ollama from langchain chroma import Chroma from langchain core.prompts import ChatPromptTemplate from langchain core.runnables import RunnablePassthrough from langchain core.output parsers import StrOutputParser def create rag chain : Must be the same model used during ingestion embeddings = HuggingFaceEmbeddings model name="all-MiniLM-L6-v2" client = chromadb.HttpClient host="chromadb", port=8000 vectorstore = Chroma client=client, collection name="documentation", embedding function=embeddings Retrieve top 3 most relevant chunks retriever = vectorstore.as retriever search type="similarity", search kwargs={"k": 3} llm = Ollama base url="http://ollama:11434", model="llama3", temperature=0.3, num predict=250 Cap response length — critical on CPU template = """You are a technical support assistant. Answer ONLY from the provided context. Cite your source. If the answer is not in the context, say so clearly. Context: {context} Question: {question} Answer:""" prompt = ChatPromptTemplate.from template template def format docs docs : return "\n\n".join f" Source: {d.metadata.get 'module', '?' } \n{d.page content}" for d in docs The full pipeline in 4 steps return {"context": retriever | format docs, "question": RunnablePassthrough } | prompt | llm | StrOutputParser The chain is a pipeline: retriever → prompt → llm → parser . LangChain's | operator makes this clean and readable. Each step passes its output to the next. FastAPI exposes the chain as a REST endpoint: python src/main.py from fastapi import FastAPI, HTTPException from pydantic import BaseModel from rag chain import create rag chain app = FastAPI title="Local RAG Assistant" chain = create rag chain class QueryRequest BaseModel : question: str @app.post "/query" def query request: QueryRequest : try: answer = chain.invoke request.question return {"question": request.question, "answer": answer} except Exception as e: raise HTTPException status code=500, detail=str e Streamlit makes it usable in a browser in 10 lines: python src/app.py import streamlit as st import requests st.title "Technical Support Assistant" question = st.text input "Your question:" if st.button "Search" and question: with st.spinner "Searching documentation... 1-2 min on CPU " : r = requests.post "http://python-api:8000/query", json={"question": question}, timeout=600 st.write r.json "answer" This is the part nobody writes about. Three real problems I hit. I was connecting with the old client settings syntax: WRONG silently stores data locally instead of sending to the server Chroma.from documents documents=chunks, client settings=Chroma.Settings chroma server host="chromadb" The ingestion ran without errors, showed 9669 chunks stored but ChromaDB had 0 vectors. Querying returned nothing. The fix explicit HttpClient : CORRECT explicit HTTP connection to the ChromaDB Docker service client = chromadb.HttpClient host="chromadb", port=8000 Chroma.from documents documents=chunks, client=client, ... Also: the v1 heartbeat endpoint /api/v1/heartbeat is deprecated. Use /api/v2/heartbeat to check if ChromaDB is alive. python Broke silently after a LangChain update from langchain.prompts import ChatPromptTemplate ❌ ModuleNotFoundError from langchain.schema.runnable import RunnablePassthrough ❌ ModuleNotFoundError Everything stable lives in langchain core now from langchain core.prompts import ChatPromptTemplate ✅ from langchain core.runnables import RunnablePassthrough ✅ LangChain has been refactoring aggressively across versions. If you hit ModuleNotFoundError , check langchain core first it's the stable base layer they're committing to not break. Llama 3 8B parameters on CPU generates roughly 2-5 tokens per second. A 200-word answer takes 2+ minutes. My 120-second timeout was too short. Three things helped significantly: k from 5 to 3 chunks shorter prompt = faster time-to-first-token num predict=250 to cap response length without it, the model generates forever .wslconfig on Windows to avoid swapAfter setup, the assistant correctly answers questions grounded in the official documentation: And critically when the answer isn't in the documentation, it says so explicitly instead of hallucinating. That was the whole point. On RAG: The embedding model used during ingestion must be identical to the one used at query time. A different model produces incompatible vector spaces your similarity search returns garbage without any error message. Chunk size matters more than expected. 500 characters worked less well than 300 for technical documents shorter chunks produce more precise retrieval. Metadata on chunks is essential for source citation. Tag every chunk with its document name at ingestion time. On local LLMs: Running on CPU is slow but viable for non-real-time use cases like internal support tools. num predict is your most important parameter for performance. Without it, the model generates until it decides to stop sometimes never. Docker container networking does not use localhost . Use the service name defined in docker-compose.yml e.g., chromadb , ollama as the hostname when connecting between containers. This project is available on my github : https://github.com/josaphatstar/Assistant-Intelligent-RAG https://github.com/josaphatstar/Assistant-Intelligent-RAG This RAG pipeline handles straightforward questions well. But it has a core limitation: it always does the same thing regardless of the question type. Whether you ask a procedural question or report a system crash, it searches the docs and answers. No reasoning about what strategy fits best. In the next article, I'll show how I evolved this into an Agentic AI architecture with LangGraph where the system first decides which agent to call documentation search, error diagnosis, or human escalation ticket , then acts accordingly. I'm a Master's student in AI & Big Data, and this project came out of a real internship constraint no cloud, no API keys, just local infrastructure and open-source tools. I'm curious: have you built a local RAG pipeline under similar constraints? What stack did you use, and what broke first? Drop it in the comments I'd genuinely like to compare notes.