{"slug": "i-built-a-local-rag-assistant-with-ollama-chromadb-and-langchain-here-s-what-i", "title": "I Built a Local RAG Assistant with Ollama, ChromaDB and LangChain. Here's What I Learned", "summary": "A developer built a fully local RAG assistant using Ollama, ChromaDB, and LangChain to help technicians query PDF manuals without sending data to the cloud. The system ingests 2,111 pages into 9,669 chunks and runs entirely via Docker Compose, ensuring sensitive financial data never leaves local infrastructure.", "body_md": "During my internship at a software services company, I noticed a recurring problem: technicians spent hours searching through hundreds of pages of PDF manuals to answer client questions. The company deploys a suite of management tools accounting, HR, logistics mainly to public institutions managing projects funded by international donors.\n\nI kept thinking: this is exactly the kind of problem LLMs were made for. So for my Master's project in AI & Big Data, I built an intelligent assistant to handle it.\n\nThe constraint that shaped every technical decision: **no data could leave the local infrastructure**. These institutions handle sensitive financial data it cannot be sent to OpenAI, Anthropic, or any cloud provider.\n\nThis forced me into the world of fully local LLMs. Here's what I built, what broke, and what I learned.\n\nThe company had accumulated massive technical documentation PDF manuals, configuration guides, FAQ files but no efficient way to query it. When a technician faced an issue with the accounting module or the HR module, they searched manually. Slow, inconsistent, and dependent on individual expertise.\n\nClassic LLMs can't help here: they don't know the private internal documentation. And without grounding in official docs, they hallucinate procedures dangerous when dealing with accounting operations for donor-funded projects.\n\nThe solution: **RAG (Retrieval-Augmented Generation)** make the LLM answer *only* from the official documentation, injected at query time.\n\nBefore the code, the concept in one sentence: instead of asking the LLM to \"know\" everything, you retrieve the relevant passages from your documents and inject them into the prompt as context.\n\nThe pipeline has two distinct phases:\n\n**Phase 1 : Ingestion (done once)**\n\n```\nPDF documents\n    ↓\nChunking (split into 300-character blocks)\n    ↓\nEmbedding (convert each block to a vector)\n    ↓\nChromaDB (store all vectors)\n```\n\n**Phase 2 : Query (at each question)**\n\n```\nUser question\n    ↓\nEmbed the question (same model)\n    ↓\nChromaDB similarity search → top 3 chunks\n    ↓\nInject chunks + question into LLM prompt\n    ↓\nLlama 3 generates a grounded answer\n```\n\nThe key insight: the LLM never \"knows\" your documents. It reads them fresh at each query, from the context you provide. No fine-tuning needed. No retraining when docs are updated.\n\nEverything runs locally via Docker Compose. Four services, each with a clear role:\n\n| Service | Role | Port |\n|---|---|---|\nOllama |\nRuns Llama 3 locally | 11434 |\nChromaDB |\nVector database | 8001 |\nFastAPI |\nRAG pipeline + REST API | 8000 |\nStreamlit |\nUser interface | 8501 |\n\n```\n# docker-compose.yml (simplified)\nservices:\n  ollama:\n    image: ollama/ollama:latest\n    ports: [\"11434:11434\"]\n    volumes: [\"./ollama_models:/root/.ollama\"]\n    environment:\n      - OLLAMA_KEEP_ALIVE=24h\n\n  chromadb:\n    image: chromadb/chroma:latest\n    ports: [\"8001:8000\"]\n    volumes: [\"./chroma_db:/chroma/chroma\"]\n\n  python-api:\n    build: ./src\n    ports: [\"8000:8000\"]\n    depends_on: [ollama, chromadb]\n    environment:\n      - OLLAMA_BASE_URL=http://ollama:11434\n      - CHROMA_HOST=chromadb\n      - CHROMA_PORT=8000\n```\n\nI ingested 10 documents accounting references (SYSCOHADA, SYCEBNL), a general accounting code, and internal user manuals. That's **2,111 pages** split into **9,669 chunks**.\n\n``` python\n# src/ingestion.py\n\nimport os\nimport chromadb\nfrom langchain_community.document_loaders import PyPDFLoader\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\nfrom langchain_chroma import Chroma\nfrom langchain_community.embeddings import HuggingFaceEmbeddings\n\nDATA_DIR = \"/app/data/documents\"\nCOLLECTION_NAME = \"documentation\"\n\ndef load_documents():\n    documents = []\n    for fichier in os.listdir(DATA_DIR):\n        if fichier.endswith(\".pdf\"):\n            try:\n                loader = PyPDFLoader(os.path.join(DATA_DIR, fichier))\n                docs = loader.load()\n                # Tag each chunk with its source document\n                for doc in docs:\n                    doc.metadata[\"module\"] = fichier.replace(\".pdf\", \"\")\n                documents.extend(docs)\n            except Exception as e:\n                print(f\"Skipped {fichier}: {e}\")  # Don't crash on encrypted PDFs\n    return documents\n\ndef store_embeddings(chunks):\n    embeddings = HuggingFaceEmbeddings(\n        model_name=\"all-MiniLM-L6-v2\",\n        model_kwargs={\"device\": \"cpu\"}\n    )\n    # Explicit HTTP connection — not localhost, the Docker service name\n    client = chromadb.HttpClient(host=\"chromadb\", port=8000)\n    Chroma.from_documents(\n        documents=chunks,\n        embedding=embeddings,\n        collection_name=COLLECTION_NAME,\n        client=client\n    )\n    print(f\"Done — {len(chunks)} vectors stored\")\n```\n\nTwo things worth noting:\n\n`try/except`\n\nis not optional one encrypted PDF (it happened) would have crashed the entire ingestion without it. Install `cryptography`\n\nif you hit an AES error: `pip install cryptography`\n\n``` python\n# src/rag_chain.py\n\nimport chromadb\nfrom langchain_community.embeddings import HuggingFaceEmbeddings\nfrom langchain_community.llms import Ollama\nfrom langchain_chroma import Chroma\nfrom langchain_core.prompts import ChatPromptTemplate\nfrom langchain_core.runnables import RunnablePassthrough\nfrom langchain_core.output_parsers import StrOutputParser\n\ndef create_rag_chain():\n    # Must be the same model used during ingestion\n    embeddings = HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\")\n\n    client = chromadb.HttpClient(host=\"chromadb\", port=8000)\n    vectorstore = Chroma(\n        client=client,\n        collection_name=\"documentation\",\n        embedding_function=embeddings\n    )\n\n    # Retrieve top 3 most relevant chunks\n    retriever = vectorstore.as_retriever(\n        search_type=\"similarity\",\n        search_kwargs={\"k\": 3}\n    )\n\n    llm = Ollama(\n        base_url=\"http://ollama:11434\",\n        model=\"llama3\",\n        temperature=0.3,\n        num_predict=250   # Cap response length — critical on CPU\n    )\n\n    template = \"\"\"You are a technical support assistant.\nAnswer ONLY from the provided context. Cite your source.\nIf the answer is not in the context, say so clearly.\n\nContext: {context}\nQuestion: {question}\nAnswer:\"\"\"\n\n    prompt = ChatPromptTemplate.from_template(template)\n\n    def format_docs(docs):\n        return \"\\n\\n\".join([\n            f\"[Source: {d.metadata.get('module', '?')}]\\n{d.page_content}\"\n            for d in docs\n        ])\n\n    # The full pipeline in 4 steps\n    return (\n        {\"context\": retriever | format_docs, \"question\": RunnablePassthrough()}\n        | prompt\n        | llm\n        | StrOutputParser()\n    )\n```\n\nThe chain is a pipeline: `retriever → prompt → llm → parser`\n\n. LangChain's `|`\n\noperator makes this clean and readable. Each step passes its output to the next.\n\nFastAPI exposes the chain as a REST endpoint:\n\n``` python\n# src/main.py\n\nfrom fastapi import FastAPI, HTTPException\nfrom pydantic import BaseModel\nfrom rag_chain import create_rag_chain\n\napp = FastAPI(title=\"Local RAG Assistant\")\nchain = create_rag_chain()\n\nclass QueryRequest(BaseModel):\n    question: str\n\n@app.post(\"/query\")\ndef query(request: QueryRequest):\n    try:\n        answer = chain.invoke(request.question)\n        return {\"question\": request.question, \"answer\": answer}\n    except Exception as e:\n        raise HTTPException(status_code=500, detail=str(e))\n```\n\nStreamlit makes it usable in a browser in 10 lines:\n\n``` python\n# src/app.py\n\nimport streamlit as st\nimport requests\n\nst.title(\"Technical Support Assistant\")\nquestion = st.text_input(\"Your question:\")\n\nif st.button(\"Search\") and question:\n    with st.spinner(\"Searching documentation... (1-2 min on CPU)\"):\n        r = requests.post(\n            \"http://python-api:8000/query\",\n            json={\"question\": question},\n            timeout=600\n        )\n        st.write(r.json()[\"answer\"])\n```\n\nThis is the part nobody writes about. Three real problems I hit.\n\nI was connecting with the old `client_settings`\n\nsyntax:\n\n```\n# WRONG silently stores data locally instead of sending to the server\nChroma.from_documents(\n    documents=chunks,\n    client_settings=Chroma.Settings(chroma_server_host=\"chromadb\")\n)\n```\n\nThe ingestion ran without errors, showed 9669 chunks stored but ChromaDB had 0 vectors. Querying returned nothing.\n\nThe fix explicit `HttpClient`\n\n:\n\n```\n# CORRECT explicit HTTP connection to the ChromaDB Docker service\nclient = chromadb.HttpClient(host=\"chromadb\", port=8000)\nChroma.from_documents(documents=chunks, client=client, ...)\n```\n\nAlso: the v1 heartbeat endpoint (`/api/v1/heartbeat`\n\n) is deprecated. Use `/api/v2/heartbeat`\n\nto check if ChromaDB is alive.\n\n``` python\n# Broke silently after a LangChain update\nfrom langchain.prompts import ChatPromptTemplate           # ❌ ModuleNotFoundError\nfrom langchain.schema.runnable import RunnablePassthrough  # ❌ ModuleNotFoundError\n\n# Everything stable lives in langchain_core now\nfrom langchain_core.prompts import ChatPromptTemplate      # ✅\nfrom langchain_core.runnables import RunnablePassthrough   # ✅\n```\n\nLangChain has been refactoring aggressively across versions. If you hit `ModuleNotFoundError`\n\n, check `langchain_core`\n\nfirst it's the stable base layer they're committing to not break.\n\nLlama 3 (8B parameters) on CPU generates roughly 2-5 tokens per second. A 200-word answer takes 2+ minutes. My 120-second timeout was too short.\n\nThree things helped significantly:\n\n`k`\n\nfrom 5 to 3 chunks shorter prompt = faster time-to-first-token`num_predict=250`\n\nto cap response length without it, the model generates forever`.wslconfig`\n\non Windows to avoid swapAfter setup, the assistant correctly answers questions grounded in the official documentation:\n\nAnd critically when the answer isn't in the documentation, it says so explicitly instead of hallucinating. That was the whole point.\n\n**On RAG:**\n\nThe embedding model used during ingestion must be **identical** to the one used at query time. A different model produces incompatible vector spaces your similarity search returns garbage without any error message.\n\nChunk size matters more than expected. 500 characters worked less well than 300 for technical documents shorter chunks produce more precise retrieval.\n\nMetadata on chunks is essential for source citation. Tag every chunk with its document name at ingestion time.\n\n**On local LLMs:**\n\nRunning on CPU is slow but viable for non-real-time use cases like internal support tools.\n\n`num_predict`\n\nis your most important parameter for performance. Without it, the model generates until it decides to stop sometimes never.\n\nDocker container networking does not use `localhost`\n\n. Use the service name defined in `docker-compose.yml`\n\n(e.g., `chromadb`\n\n, `ollama`\n\n) as the hostname when connecting between containers.\n\nThis project is available on my github : [https://github.com/josaphatstar/Assistant-Intelligent-RAG](https://github.com/josaphatstar/Assistant-Intelligent-RAG)\n\nThis RAG pipeline handles straightforward questions well. But it has a core limitation: **it always does the same thing** regardless of the question type. Whether you ask a procedural question or report a system crash, it searches the docs and answers. No reasoning about what strategy fits best.\n\nIn the next article, I'll show how I evolved this into an **Agentic AI architecture** with LangGraph where the system first *decides* which agent to call (documentation search, error diagnosis, or human escalation ticket), then acts accordingly.\n\nI'm a Master's student in AI & Big Data, and this project came out of a real internship constraint no cloud, no API keys, just local infrastructure and open-source tools. I'm curious: have you built a local RAG pipeline under similar constraints? What stack did you use, and what broke first? Drop it in the comments I'd genuinely like to compare notes.", "url": "https://wpnews.pro/news/i-built-a-local-rag-assistant-with-ollama-chromadb-and-langchain-here-s-what-i", "canonical_source": "https://dev.to/josaphatstar/i-built-a-local-rag-assistant-with-ollama-chromadb-and-langchain-heres-what-i-learned-5a2e", "published_at": "2026-07-25 14:36:54+00:00", "updated_at": "2026-07-25 15:01:09.907542+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-agents", "ai-infrastructure"], "entities": ["Ollama", "ChromaDB", "LangChain", "Llama 3", "FastAPI", "Streamlit", "Docker Compose", "HuggingFaceEmbeddings"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-local-rag-assistant-with-ollama-chromadb-and-langchain-here-s-what-i", "markdown": "https://wpnews.pro/news/i-built-a-local-rag-assistant-with-ollama-chromadb-and-langchain-here-s-what-i.md", "text": "https://wpnews.pro/news/i-built-a-local-rag-assistant-with-ollama-chromadb-and-langchain-here-s-what-i.txt", "jsonld": "https://wpnews.pro/news/i-built-a-local-rag-assistant-with-ollama-chromadb-and-langchain-here-s-what-i.jsonld"}}