Your RAG Pipeline Doesn't Need a Vector Database

A developer has published a walkthrough showing that RAG pipelines for corpora under roughly one million chunks can run entirely offline using SQLite with the sqlite-vec extension and local embeddings from Ollama's nomic-embed-text model, eliminating hosted vector databases and embedding APIs. The approach keeps data inside a single file, runs in about 50ms per query on a laptop, and avoids sending sensitive documents to external services, though the author notes text-embedding-3-large still scores a few points higher on MTEB benchmarks.

You're building a RAG system over internal HR docs, medical records, or a client's legal contracts. You reach for Pinecone or pgvector on a managed Postgres, wire up OpenAI embeddings, and ship. Six weeks later legal asks where the data lives, your per-query cost is $0.004 and climbing, and your "air-gapped" deployment story has a hole in it the size of an HTTPS connection to api.openai.com . The problem isn't RAG. The problem is that most RAG tutorials assume you need a managed vector store and a hosted embedding API. For corpora under ~1M chunks on a single machine, you don't. SQLite with the sqlite-vec extension, plus a local embedding model via Ollama, gives you a fully offline pipeline that fits in a single file, runs in ~50ms per query on a laptop, and leaks nothing. This article walks through the actual code: chunking, embedding, storage, retrieval, and the parts that will bite you. Three concrete reasons, in order of how often they actually matter: text-embedding-3-small is a data transfer to OpenAI. If your contract says "data never leaves the customer VPC," you're already in violation. Local embeddings make this a non-issue. nomic-embed-text via Ollama is free after the one-time model download. The trade-off is real: text-embedding-3-large 3072 dims still beats nomic-embed-text 768 dims on MTEB by a few points. On domain-specific corpora, that gap often shrinks. Test on your data before assuming. Ollama serves the embedding model locally on :11434 ollama pull nomic-embed-text sqlite-vec is a loadable extension, not a fork of SQLite pip install sqlite-vec ollama schema.sql : -- vec0 is a virtual table; the embedding column has fixed dimensionality -- because sqlite-vec stores vectors in a packed binary format, not JSON. CREATE VIRTUAL TABLE IF NOT EXISTS chunks USING vec0 embedding float 768 , +text TEXT, +source TEXT, +chunk index INTEGER ; -- Metadata filtering needs a regular index alongside the virtual table. CREATE INDEX IF NOT EXISTS idx source ON chunks source ; The + prefix marks auxiliary columns — they're stored but not indexed for vector search. You can filter on them in WHERE clauses. Chunking is where most RAG pipelines quietly fail. Fixed 512-token windows split sentences, split code blocks, and split table rows. Overlap helps but doubles your storage. python import re def chunk text text: str, target chars: int = 1800, overlap: int = 200 - list str : """Split on paragraph boundaries first, then sentences, then hard-wrap. Why not just split on tokens? Because token counts don't align with semantic boundaries — a 512-token window will happily cut a numbered list in half. Char-based targets are crude but the boundary logic below keeps units intact. """ paragraphs = re.split r"\n\s \n", text.strip chunks, buf = , "" for para in paragraphs: If adding this paragraph overshoots, flush and start fresh. Only carry overlap when the buffer is already substantial. if len buf + len para target chars and buf: chunks.append buf.strip buf = buf -overlap: if overlap else "" buf += para + "\n\n" if buf.strip : chunks.append buf.strip Any chunk still oversized gets sentence-split as a last resort. final = for c in chunks: if len c <= target chars 2: final.append c else: sentences = re.split r" ?<= . ? \s+", c sub = "" for s in sentences: if len sub + len s target chars and sub: final.append sub.strip sub = "" sub += s + " " if sub.strip : final.append sub.strip return final Two things to tune: target chars should match your embedding model's context. nomic-embed-text handles 8192 tokens, but retrieval quality degrades on long inputs — 1500–2000 chars is the sweet spot in my testing. And overlap should be roughly one sentence, not 20% of the chunk. php import sqlite3, sqlite vec, ollama, struct def embed texts: list str - list list float : Batch through Ollama's /api/embed; one HTTP call per batch, not per text. resp = ollama.embed model="nomic-embed-text", input=texts return resp "embeddings" def ingest db path: str, source: str, text: str : conn = sqlite3.connect db path conn.enable load extension True sqlite vec.load conn conn.enable load extension False chunks = chunk text text Batch size 32 keeps memory bounded and Ollama's queue happy. for i in range 0, len chunks , 32 : batch = chunks i:i+32 vectors = embed batch conn.executemany "INSERT INTO chunks embedding, text, source, chunk index " "VALUES ?, ?, ?, ? ", struct.pack f"{len v }f", v , t, source, i + j for j, t, v in enumerate zip batch, vectors , conn.commit conn.close The struct.pack step is not optional. sqlite-vec expects raw little-endian float32 bytes, not a JSON array or a Python list. Passing a list silently produces garbage results or an error depending on version — always pack. python def search db path: str, query: str, k: int = 5, source filter: str | None = None : conn = sqlite3.connect db path conn.enable load extension True sqlite vec.load conn conn.enable load extension False qvec = embed query 0 qbytes = struct.pack f"{len qvec }f", qvec KNN in sqlite-vec uses the MATCH operator with k = ? . The distance is L2 by default; see below for cosine. sql = """ SELECT text, source, chunk index, distance FROM chunks WHERE embedding MATCH ? AND k = ? """ params = qbytes, k if source filter: Post-filter is fine at small k, but see the gotcha below. sql = sql.replace "k = ?", "k = ? AND source = ?" params.append source filter return conn.execute sql, params .fetchall Call it: for text, src, idx, dist in search "docs.db", "what is the PTO carryover policy?", k=5 : print f" {dist:.3f} {src} {idx}: {text :120 }..." php def answer db path: str, question: str - str: hits = search db path, question, k=5 context = "\n\n---\n\n".join h 0 for h in hits prompt = "Answer using ONLY the context below. If the answer isn't present, " "say so. Cite chunk indices.\n\n" f"Context:\n{context}\n\nQuestion: {question}" resp = ollama.chat model="llama3.1:8b", messages= {"role": "user", "content": prompt} , return resp "message" "content" llama3.1:8b at Q4 K M runs at ~40 tok/s on an M2 Pro and is good enough for extractive QA. Step up to qwen2.5:14b or llama3.3:70b if you have the VRAM and the questions require real reasoning across chunks. Cosine vs L2. sqlite-vec defaults to L2 distance. If you want cosine, normalize your vectors before insertion and before query — then L2 and cosine rank identically. nomic-embed-text does not return normalized vectors. Post-filtering kills recall. WHERE embedding MATCH ? AND k = 5 AND source = 'hr.pdf' retrieves 5 nearest overall, then filters. If your corpus is 90% legal and 10% HR, you'll often get zero HR hits. Either pre-filter with a separate query or over-fetch k = 50 and truncate. There's an open issue on this in the sqlite-vec repo; the workaround is a two-stage query. sqlite-vec is pre-1.0. The API has changed between 0.0.x and 0.1.x. Pin your version. It also doesn't yet support ANN indexes — every query is a brute-force scan. At 100K vectors × 768 dims that's ~30ms. At 1M it's ~300ms and climbing. Batch embedding can OOM Ollama. Sending 500 texts in one call will spike memory. 32–64 is safe on most machines. No incremental re-embedding. Change your chunker and you re-embed everything. Store the chunker version in a metadata table so you can detect drift. pgvector with HNSW, Qdrant, or LanceDB. text-embedding-3-large beats your local model by 15 points on your eval set, the privacy win isn't worth the retrieval loss — negotiate a BAA or self-host a bigger model instead. For everything else — internal wikis, personal knowledge bases, single-tenant document Q&A, air-gapped deployments — a 200-line Python file and a .db you can scp is the right amount of infrastructure. Reference: sqlite-vec docs https://alexgarcia.xyz/sqlite-vec/ , Ollama embeddings API https://github.com/ollama/ollama/blob/main/docs/api.md generate-embeddings , MTEB leaderboard https://huggingface.co/spaces/mteb/leaderboard .