# Build a Fully Local RAG API with FastAPI, ChromaDB and LFM3 on Ollama > Source: > Published: 2026-11-03 14:01:05+00:00 Follow me on 🔗[Linkedin](https://www.linkedin.com/in/prajwal-waykos/), 🐙[GitHub](https://github.com/praj-17) You need Ollama v0.40.13 or newer (that is the release where LFM3 started running on Linux and Windows, Apple silicon got it in v0.40.12). Then run these commands and you have a private RAG API on your own machine. ``` ollama pull lfm3ollama pull nomic-embed-textpip install fastapi uvicorn chromadb ollamauvicorn app:app --port 8000# then open http://localhost:8000/docs and try /ingest and /ask ``` The full app.py is further down. On my CPU only test machine the whole stack used about 1.9 GB of RAM, and a warm answer took 4 to 17 seconds. *“My laptop fan has opinions about large language models. Today it finally stayed quiet.”* Hey there, fellow tech-enthusiast! Here’s a slightly embarrassing confession. My most read post on Medium is still the RAG tutorial I wrote back in December 2024, the one with FastAPI, Haystack and ChromaDB. It has close to 2K views and most of them come from Google. Two years later, people are still landing on it. So I reread it. It was about 4,100 words long, needed ten pip packages, and had an Ollama URL in the code that I had proudly marked CONFIDENTIAL. Past me was very serious about security and not very serious about word counts. Then on October 27, 2026, Liquid AI released LFM3, and their blog said the 1.7B text model fits in about 1 GB at 4-bit precision. That’s smaller than some of my Chrome sessions. I took it as a sign to rebuild the tutorial the way I wish it had been, simple and fully local. LFM3 is the new family of open weight models from [Liquid AI](https://www.liquid.ai/blog/lfm3). The first release has six models around the 2B size, covering text, vision, audio, omni and two decision models. We only need the text one, LFM3–1.7B. A few things stood out to me in the release. It has a 256K token context window. It uses what Liquid calls a tri-hybrid design, 24 layers made of 13 short convolution layers, 5 sliding window attention layers and 6 global attention layers. Liquid also says it beats their older LFM2.5–2.6B with 35% fewer parameters. But the line that sold me was in the Ollama model page. It says the model is not great at knowledge heavy tasks on its own and recommends using retrieval or search for those. In other words, the model is basically asking for RAG. A small model that knows it is small is my kind of model. Ollama added it in [v0.40.12](https://github.com/ollama/ollama/releases/tag/v0.40.12) for Apple hardware, and [v0.40.13](https://github.com/ollama/ollama/releases/tag/v0.40.13) brought Linux and Windows. The tag is simply lfm3 (also listed as lfm3:1.7b). There are only two jobs. /ingest takes some text, cuts it into chunks, turns every chunk into a vector with nomic-embed-text, and saves it in ChromaDB. /ask turns your question into a vector, pulls the three closest chunks, and asks LFM3 to answer using only those. I dropped Haystack this time. Not because it’s bad, it’s great for bigger pipelines, but for a beginner the framework hides exactly the part you are trying to learn. Save this as src/app.py. It's 97 lines including comments. ``` """A fully local RAG API: FastAPI + ChromaDB + LFM3 on Ollama. No API keys."""import timeimport chromadbimport ollamafrom fastapi import FastAPIfrom pydantic import BaseModelCHAT_MODEL = "lfm3" # Liquid AI LFM3-1.7B, about 1 GB at 4-bitEMBED_MODEL = "nomic-embed-text" # small local embedding modelapp = FastAPI(title="Local RAG with LFM3")# Persistent storage: your vectors survive restarts in ./chroma_dbdb = chromadb.PersistentClient(path="./chroma_db")docs = db.get_or_create_collection("docs", metadata={"hnsw:space": "cosine"})class IngestRequest(BaseModel): text: str source: str = "manual" chunk_size: int = 500 # characters per chunkclass AskRequest(BaseModel): question: str top_k: int = 3def embed(texts: list[str]) -> list[list[float]]: """Turn text into vectors using the local embedding model.""" return ollama.embed(model=EMBED_MODEL, input=texts)["embeddings"]def split_into_chunks(text: str, size: int) -> list[str]: """Split on paragraphs, then pack paragraphs into chunks of about `size` chars.""" chunks, current = [], "" for paragraph in text.split("\n\n"): if current and len(current) + len(paragraph) > size: chunks.append(current.strip()) current = "" current += paragraph + "\n\n" if current.strip(): chunks.append(current.strip()) return chunks@app.post("/ingest")def ingest(req: IngestRequest): start = time.perf_counter() chunks = split_into_chunks(req.text, req.chunk_size) docs.upsert( ids=[f"{req.source}-{i}" for i in range(len(chunks))], documents=chunks, embeddings=embed(chunks), metadatas=[{"source": req.source} for _ in chunks], ) return { "chunks_added": len(chunks), "total_chunks": docs.count(), "seconds": round(time.perf_counter() - start, 2), }@app.post("/ask")def ask(req: AskRequest): start = time.perf_counter() # 1. Retrieve: find the chunks closest to the question hits = docs.query(query_embeddings=embed([req.question]), n_results=req.top_k) context = "\n\n---\n\n".join(hits["documents"][0]) retrieve_time = time.perf_counter() - start # 2. Generate: let LFM3 answer using only that context prompt = ( "Answer the question using only the context below. " "If the answer is not in the context, say you don't know.\n\n" f"Context:\n{context}\n\nQuestion: {req.question}" ) reply = ollama.chat( model=CHAT_MODEL, messages=[{"role": "user", "content": prompt}], options={"temperature": 0.1}, ) # Small models sometimes "think out loud" first, keep only the final answer answer = reply["message"]["content"].split("")[-1].strip() return { "answer": answer, "sources": sorted({m["source"] for m in hits["metadatas"][0]}), "retrieve_seconds": round(retrieve_time, 2), "total_seconds": round(time.perf_counter() - start, 2), }@app.get("/health")def health(): return {"status": "ok", "chunks": docs.count(), "model": CHAT_MODEL} ``` The PersistentClient is the bit I was most happy about. Your vectors are saved in a chroma_db folder, so restarting the server doesn't wipe your knowledge base. I tested this by killing the server and starting it again, and /health still reported all 3 chunks. I wanted to test it with something LFM3 could not possibly know from training. So I fed it a short notes file about LFM3’s own release, including the Ollama version numbers from last week. Yes, I made the model read about itself. It felt a bit like showing someone their own Wikipedia page. ``` python try_it.py ../docs/lfm3_notes.md ``` Here’s the real output from my run. ``` INGEST {'chunks_added': 3, 'total_chunks': 3, 'seconds': 1.03}Q: How much memory does LFM3-1.7B need?A: LFM3-1.7B-Base is a model that fits in about 1 GB of memory at 4-bit precision, according to Liquid AI's specifications. retrieve 0.04s, total 17.5sQ: Which Ollama version added Linux support for LFM3?A: Version v0.40.13 added Linux and Windows support for LFM3. retrieve 0.21s, total 4.5sQ: How many layers are global attention?A: ... Therefore, the number of layers that are global attention is **6**. retrieve 0.23s, total 7.11sQ: Who won the 2026 cricket world cup?A: I don't have any information about the winner of the 2026 cricket world cup in the provided context. retrieve 0.2s, total 4.56s ``` Three correct answers, and one very polite “I don’t know”. As an Indian, I respect a model that refuses to guess about cricket. That’s how fights start. I ran this on a Linux box with 8 vCPUs and no GPU, so think of these as “worst case laptop” numbers. Raw generation speed was about 31 tokens per second. A Mac with Apple silicon or any laptop with a decent GPU should feel snappier. Retrieval is fast, generation is the slow part. Finding the right chunks took less than a third of a second every time. Almost all the waiting was the model writing. So if you want to speed things up, look at the model and the prompt before you blame ChromaDB. The first question is always slow. That 17.5 seconds was mostly Ollama loading the model into memory. After that it stays loaded for a few minutes. If you demo this to a friend, ask a warm up question first. Trust me. Small models like to think out loud. When I created the model from the GGUF file, LFM3 sometimes wrote its reasoning before the answer, ending with a tag. That's why there's a tiny split("") line in the code. It also explains why my warm answers ranged from 4 to 17 seconds. Some questions got a longer think than others. “Only use the context” really matters. The cricket question would have been a lovely hallucination without that instruction. With it, the model just admitted it didn’t know. For a RAG app, an honest “I don’t know” is worth more than a confident wrong answer. Small models are close, not perfect. Look at the first answer again. It said “LFM3–1.7B-Base” when the notes talk about LFM3–1.7B. The number was right and the label was slightly off. That’s fine for a personal notes bot, but always show sources so the reader can check. My API returns them for exactly this reason. Chunk size is a knob, not a law. I used paragraphs packed into roughly 500 characters. Too small and each chunk loses its meaning. Too big and you stuff the prompt with noise. Start at 500 and play with it on your own documents. Local doesn’t mean free of trade offs. You pay with a one time download of about 1.4 GB and some patience on slow hardware. In return you get zero API keys, zero per question bills, and documents that never leave your machine. If you want to keep going, add a /ingest-file endpoint that reads PDFs, or bring back the URL crawling from my 2024 post. You could also try top_k values of 2 to 5 and see how the answers change. And since LFM3 is built for tool calling, a small agent on top of this is a very tempting weekend project. Two years ago, building RAG felt like assembling furniture with a manual in another language. This time it was one file, two models, and a laptop that stayed calm the whole time. The best part is that a beginner can now read every line and understand what’s happening. *Remember, the only limit is your imagination (and maybe your RAM)!* **What’s the first document you’d feed to your own local RAG? Tell me in the comments!** **Happy coding! If you have any questions or suggestions, feel free to reach out or leave a comment.** [Build a Fully Local RAG API with FastAPI, ChromaDB and LFM3 on Ollama](https://pub.towardsai.net/build-a-fully-local-rag-api-with-fastapi-chromadb-and-lfm3-on-ollama-181480164a6c) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.