Build a Fully Local RAG API with FastAPI, ChromaDB and LFM3 on Ollama Liquid AI released LFM3 on October 27, 2026, an open-weight model family whose first release includes six models around the 2B size, and the 1.7B text model fits in about 1 GB at 4-bit precision with a 256K token context window. Ollama added LFM3 in v0.40.12 for Apple hardware and v0.40.13 for Linux and Windows, and a tutorial shows a 97-line FastAPI, ChromaDB and nomic-embed-text RAG API that used about 1.9 GB of RAM on a CPU-only machine with warm answers taking 4 to 17 seconds. Liquid says LFM3-1.7B beats its older LFM2.5-2.6B with 35% fewer parameters, and its tri-hybrid design uses 24 layers: 13 short convolution layers, 5 sliding window attention layers and 6 global attention layers. Follow me on 🔗 Linkedin https://www.linkedin.com/in/prajwal-waykos/ , 🐙 GitHub https://github.com/praj-17 You need Ollama v0.40.13 or newer that is the release where LFM3 started running on Linux and Windows, Apple silicon got it in v0.40.12 . Then run these commands and you have a private RAG API on your own machine. ollama pull lfm3ollama pull nomic-embed-textpip install fastapi uvicorn chromadb ollamauvicorn app:app --port 8000 then open http://localhost:8000/docs and try /ingest and /ask The full app.py is further down. On my CPU only test machine the whole stack used about 1.9 GB of RAM, and a warm answer took 4 to 17 seconds. “My laptop fan has opinions about large language models. Today it finally stayed quiet.” Hey there, fellow tech-enthusiast Here’s a slightly embarrassing confession. My most read post on Medium is still the RAG tutorial I wrote back in December 2024, the one with FastAPI, Haystack and ChromaDB. It has close to 2K views and most of them come from Google. Two years later, people are still landing on it. So I reread it. It was about 4,100 words long, needed ten pip packages, and had an Ollama URL in the code that I had proudly marked CONFIDENTIAL. Past me was very serious about security and not very serious about word counts. Then on October 27, 2026, Liquid AI released LFM3, and their blog said the 1.7B text model fits in about 1 GB at 4-bit precision. That’s smaller than some of my Chrome sessions. I took it as a sign to rebuild the tutorial the way I wish it had been, simple and fully local. LFM3 is the new family of open weight models from Liquid AI https://www.liquid.ai/blog/lfm3 . The first release has six models around the 2B size, covering text, vision, audio, omni and two decision models. We only need the text one, LFM3–1.7B. A few things stood out to me in the release. It has a 256K token context window. It uses what Liquid calls a tri-hybrid design, 24 layers made of 13 short convolution layers, 5 sliding window attention layers and 6 global attention layers. Liquid also says it beats their older LFM2.5–2.6B with 35% fewer parameters. But the line that sold me was in the Ollama model page. It says the model is not great at knowledge heavy tasks on its own and recommends using retrieval or search for those. In other words, the model is basically asking for RAG. A small model that knows it is small is my kind of model. Ollama added it in v0.40.12 https://github.com/ollama/ollama/releases/tag/v0.40.12 for Apple hardware, and v0.40.13 https://github.com/ollama/ollama/releases/tag/v0.40.13 brought Linux and Windows. The tag is simply lfm3 also listed as lfm3:1.7b . There are only two jobs. /ingest takes some text, cuts it into chunks, turns every chunk into a vector with nomic-embed-text, and saves it in ChromaDB. /ask turns your question into a vector, pulls the three closest chunks, and asks LFM3 to answer using only those. I dropped Haystack this time. Not because it’s bad, it’s great for bigger pipelines, but for a beginner the framework hides exactly the part you are trying to learn. Save this as src/app.py. It's 97 lines including comments. """A fully local RAG API: FastAPI + ChromaDB + LFM3 on Ollama. No API keys."""import timeimport chromadbimport ollamafrom fastapi import FastAPIfrom pydantic import BaseModelCHAT MODEL = "lfm3" Liquid AI LFM3-1.7B, about 1 GB at 4-bitEMBED MODEL = "nomic-embed-text" small local embedding modelapp = FastAPI title="Local RAG with LFM3" Persistent storage: your vectors survive restarts in ./chroma dbdb = chromadb.PersistentClient path="./chroma db" docs = db.get or create collection "docs", metadata={"hnsw:space": "cosine"} class IngestRequest BaseModel : text: str source: str = "manual" chunk size: int = 500 characters per chunkclass AskRequest BaseModel : question: str top k: int = 3def embed texts: list str - list list float : """Turn text into vectors using the local embedding model.""" return ollama.embed model=EMBED MODEL, input=texts "embeddings" def split into chunks text: str, size: int - list str : """Split on paragraphs, then pack paragraphs into chunks of about size chars.""" chunks, current = , "" for paragraph in text.split "\n\n" : if current and len current + len paragraph size: chunks.append current.strip current = "" current += paragraph + "\n\n" if current.strip : chunks.append current.strip return chunks@app.post "/ingest" def ingest req: IngestRequest : start = time.perf counter chunks = split into chunks req.text, req.chunk size docs.upsert ids= f"{req.source}-{i}" for i in range len chunks , documents=chunks, embeddings=embed chunks , metadatas= {"source": req.source} for in chunks , return { "chunks added": len chunks , "total chunks": docs.count , "seconds": round time.perf counter - start, 2 , }@app.post "/ask" def ask req: AskRequest : start = time.perf counter 1. Retrieve: find the chunks closest to the question hits = docs.query query embeddings=embed req.question , n results=req.top k context = "\n\n---\n\n".join hits "documents" 0 retrieve time = time.perf counter - start 2. Generate: let LFM3 answer using only that context prompt = "Answer the question using only the context below. " "If the answer is not in the context, say you don't know.\n\n" f"Context:\n{context}\n\nQuestion: {req.question}" reply = ollama.chat model=CHAT MODEL, messages= {"role": "user", "content": prompt} , options={"temperature": 0.1}, Small models sometimes "think out loud" first, keep only the final answer answer = reply "message" "content" .split "