{"slug": "your-rag-pipeline-doesn-t-need-a-vector-database", "title": "Your RAG Pipeline Doesn't Need a Vector Database", "summary": "A developer has published a walkthrough showing that RAG pipelines for corpora under roughly one million chunks can run entirely offline using SQLite with the sqlite-vec extension and local embeddings from Ollama's nomic-embed-text model, eliminating hosted vector databases and embedding APIs. The approach keeps data inside a single file, runs in about 50ms per query on a laptop, and avoids sending sensitive documents to external services, though the author notes text-embedding-3-large still scores a few points higher on MTEB benchmarks.", "body_md": "You're building a RAG system over internal HR docs, medical records, or a client's legal contracts. You reach for Pinecone or `pgvector` on a managed Postgres, wire up OpenAI embeddings, and ship. Six weeks later legal asks where the data lives, your per-query cost is $0.004 and climbing, and your \"air-gapped\" deployment story has a hole in it the size of an HTTPS connection to `api.openai.com`.\n\nThe problem isn't RAG. The problem is that most RAG tutorials assume you need a managed vector store and a hosted embedding API. For corpora under ~1M chunks on a single machine, you don't. SQLite with the `sqlite-vec` extension, plus a local embedding model via Ollama, gives you a fully offline pipeline that fits in a single file, runs in ~50ms per query on a laptop, and leaks nothing.\n\nThis article walks through the actual code: chunking, embedding, storage, retrieval, and the parts that will bite you.\n\nThree concrete reasons, in order of how often they actually matter:\n\n`text-embedding-3-small` is a data transfer to OpenAI. If your contract says \"data never leaves the customer VPC,\" you're already in violation. Local embeddings make this a non-issue.`nomic-embed-text` via Ollama is free after the one-time model download.\nThe trade-off is real: `text-embedding-3-large` (3072 dims) still beats `nomic-embed-text` (768 dims) on MTEB by a few points. On domain-specific corpora, that gap often shrinks. Test on your data before assuming.\n\n```\n# Ollama serves the embedding model locally on :11434\nollama pull nomic-embed-text\n\n# sqlite-vec is a loadable extension, not a fork of SQLite\npip install sqlite-vec ollama\n```\n\n`schema.sql`:\n\n```\n-- vec0 is a virtual table; the embedding column has fixed dimensionality\n-- because sqlite-vec stores vectors in a packed binary format, not JSON.\nCREATE VIRTUAL TABLE IF NOT EXISTS chunks USING vec0(\n    embedding float[768],\n    +text TEXT,\n    +source TEXT,\n    +chunk_index INTEGER\n);\n\n-- Metadata filtering needs a regular index alongside the virtual table.\nCREATE INDEX IF NOT EXISTS idx_source ON chunks(source);\n```\n\nThe `+` prefix marks auxiliary columns — they're stored but not indexed for vector search. You can filter on them in `WHERE` clauses.\n\nChunking is where most RAG pipelines quietly fail. Fixed 512-token windows split sentences, split code blocks, and split table rows. Overlap helps but doubles your storage.\n\n``` python\nimport re\n\ndef chunk_text(text: str, target_chars: int = 1800, overlap: int = 200) -> list[str]:\n    \"\"\"Split on paragraph boundaries first, then sentences, then hard-wrap.\n\n    Why not just split on tokens? Because token counts don't align with\n    semantic boundaries — a 512-token window will happily cut a numbered\n    list in half. Char-based targets are crude but the boundary logic\n    below keeps units intact.\n    \"\"\"\n    paragraphs = re.split(r\"\\n\\s*\\n\", text.strip())\n    chunks, buf = [], \"\"\n\n    for para in paragraphs:\n        # If adding this paragraph overshoots, flush and start fresh.\n        # Only carry overlap when the buffer is already substantial.\n        if len(buf) + len(para) > target_chars and buf:\n            chunks.append(buf.strip())\n            buf = buf[-overlap:] if overlap else \"\"\n        buf += para + \"\\n\\n\"\n\n    if buf.strip():\n        chunks.append(buf.strip())\n\n    # Any chunk still oversized gets sentence-split as a last resort.\n    final = []\n    for c in chunks:\n        if len(c) <= target_chars * 2:\n            final.append(c)\n        else:\n            sentences = re.split(r\"(?<=[.!?])\\s+\", c)\n            sub = \"\"\n            for s in sentences:\n                if len(sub) + len(s) > target_chars and sub:\n                    final.append(sub.strip())\n                    sub = \"\"\n                sub += s + \" \"\n            if sub.strip():\n                final.append(sub.strip())\n    return final\n```\n\nTwo things to tune: `target_chars` should match your embedding model's context. `nomic-embed-text` handles 8192 tokens, but retrieval quality degrades on long inputs — 1500–2000 chars is the sweet spot in my testing. And overlap should be roughly one sentence, not 20% of the chunk.\n\n``` php\nimport sqlite3, sqlite_vec, ollama, struct\n\ndef embed(texts: list[str]) -> list[list[float]]:\n    # Batch through Ollama's /api/embed; one HTTP call per batch, not per text.\n    resp = ollama.embed(model=\"nomic-embed-text\", input=texts)\n    return resp[\"embeddings\"]\n\ndef ingest(db_path: str, source: str, text: str):\n    conn = sqlite3.connect(db_path)\n    conn.enable_load_extension(True)\n    sqlite_vec.load(conn)\n    conn.enable_load_extension(False)\n\n    chunks = chunk_text(text)\n    # Batch size 32 keeps memory bounded and Ollama's queue happy.\n    for i in range(0, len(chunks), 32):\n        batch = chunks[i:i+32]\n        vectors = embed(batch)\n        conn.executemany(\n            \"INSERT INTO chunks(embedding, text, source, chunk_index) \"\n            \"VALUES (?, ?, ?, ?)\",\n            [\n                (struct.pack(f\"{len(v)}f\", *v), t, source, i + j)\n                for j, (t, v) in enumerate(zip(batch, vectors))\n            ],\n        )\n    conn.commit()\n    conn.close()\n```\n\nThe `struct.pack` step is not optional. `sqlite-vec` expects raw little-endian float32 bytes, not a JSON array or a Python list. Passing a list silently produces garbage results or an error depending on version — always pack.\n\n``` python\ndef search(db_path: str, query: str, k: int = 5, source_filter: str | None = None):\n    conn = sqlite3.connect(db_path)\n    conn.enable_load_extension(True)\n    sqlite_vec.load(conn)\n    conn.enable_load_extension(False)\n\n    qvec = embed([query])[0]\n    qbytes = struct.pack(f\"{len(qvec)}f\", *qvec)\n\n    # KNN in sqlite-vec uses the `MATCH` operator with `k = ?`.\n    # The distance is L2 by default; see below for cosine.\n    sql = \"\"\"\n        SELECT text, source, chunk_index, distance\n        FROM chunks\n        WHERE embedding MATCH ? AND k = ?\n    \"\"\"\n    params = [qbytes, k]\n\n    if source_filter:\n        # Post-filter is fine at small k, but see the gotcha below.\n        sql = sql.replace(\"k = ?\", \"k = ? AND source = ?\")\n        params.append(source_filter)\n\n    return conn.execute(sql, params).fetchall()\n```\n\nCall it:\n\n```\nfor text, src, idx, dist in search(\"docs.db\", \"what is the PTO carryover policy?\", k=5):\n    print(f\"[{dist:.3f}] {src}#{idx}: {text[:120]}...\")\nphp\ndef answer(db_path: str, question: str) -> str:\n    hits = search(db_path, question, k=5)\n    context = \"\\n\\n---\\n\\n\".join(h[0] for h in hits)\n    prompt = (\n        \"Answer using ONLY the context below. If the answer isn't present, \"\n        \"say so. Cite chunk indices.\\n\\n\"\n        f\"Context:\\n{context}\\n\\nQuestion: {question}\"\n    )\n    resp = ollama.chat(\n        model=\"llama3.1:8b\",\n        messages=[{\"role\": \"user\", \"content\": prompt}],\n    )\n    return resp[\"message\"][\"content\"]\n```\n\n`llama3.1:8b` at Q4_K_M runs at ~40 tok/s on an M2 Pro and is good enough for extractive QA. Step up to `qwen2.5:14b` or `llama3.3:70b` if you have the VRAM and the questions require real reasoning across chunks.\n\n**Cosine vs L2.** `sqlite-vec` defaults to L2 distance. If you want cosine, normalize your vectors before insertion and before query — then L2 and cosine rank identically. `nomic-embed-text` does not return normalized vectors.\n\n**Post-filtering kills recall.** `WHERE embedding MATCH ? AND k = 5 AND source = 'hr.pdf'` retrieves 5 nearest overall, *then* filters. If your corpus is 90% legal and 10% HR, you'll often get zero HR hits. Either pre-filter with a separate query or over-fetch (`k = 50`) and truncate. There's an open issue on this in the `sqlite-vec` repo; the workaround is a two-stage query.\n\n**sqlite-vec is pre-1.0.** The API has changed between 0.0.x and 0.1.x. Pin your version. It also doesn't yet support ANN indexes — every query is a brute-force scan. At 100K vectors × 768 dims that's ~30ms. At 1M it's ~300ms and climbing.\n\n**Batch embedding can OOM Ollama.** Sending 500 texts in one call will spike memory. 32–64 is safe on most machines.\n\n**No incremental re-embedding.** Change your chunker and you re-embed everything. Store the chunker version in a metadata table so you can detect drift.\n\n`pgvector` with HNSW, Qdrant, or LanceDB.`text-embedding-3-large` beats your local model by 15 points on your eval set, the privacy win isn't worth the retrieval loss — negotiate a BAA or self-host a bigger model instead.\nFor everything else — internal wikis, personal knowledge bases, single-tenant document Q&A, air-gapped deployments — a 200-line Python file and a `.db` you can `scp` is the right amount of infrastructure.\n\nReference: [`sqlite-vec` docs](https://alexgarcia.xyz/sqlite-vec/), [Ollama embeddings API](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-embeddings), [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard).", "url": "https://wpnews.pro/news/your-rag-pipeline-doesn-t-need-a-vector-database", "canonical_source": "https://dev.to/mdyer94/your-rag-pipeline-doesnt-need-a-vector-database-66m", "published_at": "2026-09-22 10:00:29+00:00", "updated_at": "2026-09-22 10:23:20.743773+00:00", "lang": "en", "topics": ["ai-tools", "ai-infrastructure", "developer-tools", "natural-language-processing"], "entities": ["SQLite", "sqlite-vec", "Ollama", "nomic-embed-text", "OpenAI", "Pinecone", "pgvector", "text-embedding-3-large"], "alternates": {"html": "https://wpnews.pro/news/your-rag-pipeline-doesn-t-need-a-vector-database", "markdown": "https://wpnews.pro/news/your-rag-pipeline-doesn-t-need-a-vector-database.md", "text": "https://wpnews.pro/news/your-rag-pipeline-doesn-t-need-a-vector-database.txt", "jsonld": "https://wpnews.pro/news/your-rag-pipeline-doesn-t-need-a-vector-database.jsonld"}}