# Building a Zero-Cost Local RAG Pipeline using Python and Ollama

> Source: <https://dev.to/heena_khan_22b6ad208a0810/building-a-zero-cost-local-rag-pipeline-using-python-and-ollama-4gh2>
> Published: 2026-11-04 20:41:24+00:00

Retrieval-Augmented Generation (RAG) lets you query your custom private documents using large language models (LLMs) without sending sensitive data to third-party APIs.

In this tutorial, we will build an entirely local, zero-cost RAG pipeline using **Python**, **Ollama**, and **ChromaDB**.

`ollama pull mistral`)
Install ChromaDB (vector database) and the Ollama Python client:

```
pip install chromadb ollama
```

We will store our knowledge base documents as vector embeddings in ChromaDB:

``` python
import chromadb
import ollama

# Initialize local persistent vector store
client = chromadb.PersistentClient(path="./rag_db")
collection = client.get_or_create_collection(name="local_knowledge")

# Sample document knowledge base
documents = [
    "Our enterprise platform policy requires JWT rotation every 24 hours.",
    "System maintenance occurs every Sunday from 2 AM to 4 AM UTC.",
    "Database read replicas are deployed across US-East and EU-Central regions."
]

# Generate embeddings locally via Ollama and store in ChromaDB
for i, doc in enumerate(documents):
    response = ollama.embeddings(model="mistral", prompt=doc)
    collection.add(
        ids=[f"doc_{i}"],
        embeddings=[response["embedding"]],
        documents=[doc]
    )

print("Knowledge base successfully embedded and indexed locally!")
```

Now, retrieve context relevant to a user prompt and generate an accurate response:

``` python
def query_rag(user_query: str):
    # 1. Embed the incoming question
    query_embedding = ollama.embeddings(model="mistral", prompt=user_query)["embedding"]

    # 2. Query top 2 matching document chunks
    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=2
    )
    retrieved_context = " ".join(results["documents"][0])

    # 3. Pass context and question to the local LLM
    prompt = f"""
    Answer the question based ONLY on the context below:

    Context: {retrieved_context}

    Question: {user_query}
    """

    response = ollama.generate(model="mistral", prompt=prompt)
    return response["response"]

# Example execution
answer = query_rag("When does system maintenance happen?")
print("AI Response:\n", answer)
```

By pairing **Ollama** with **ChromaDB**, you get a production-ready, local RAG pipeline with **zero API costs** and **100% data privacy**.