{"slug": "i-ditched-cloud-vector-databases-for-sqlite-fts5-and-my-rag-pipeline-got-10x", "title": "I Ditched Cloud Vector Databases for SQLite FTS5 — and My RAG Pipeline Got 10x Better", "summary": "A developer replaced a cloud vector database with SQLite FTS5 and dense embeddings in a hybrid RAG pipeline, reporting a 10x improvement in retrieval accuracy for exact keywords, error codes, and version numbers. The approach combines BM25 lexical search with cosine similarity using Reciprocal Rank Fusion, cutting infrastructure costs and complexity.", "body_md": "A few months ago, I was debugging a RAG pipeline for an internal engineering repo. A developer typed:\n\n\"What is the timeout limit in`config_v2.py`\n\nfor worker pool #4?\"\n\nThe system, backed by a popular managed vector database and state-of-the-art cosine embeddings, confidently retrieved five paragraphs about worker thread best practices, microservice resiliency, and thread pool scaling patterns.\n\nIt completely missed the single-line comment in `config_v2.py`\n\nwhere `WORKER_POOL_4_TIMEOUT = 45`\n\nwas defined.\n\nWhy? Because mathematically, a variable name and an exact integer don't have high semantic similarity to an abstract question about architecture.\n\nThat night, I audited the infrastructure bill. We were paying hundreds of dollars a month for a cluster of vector instances storing less than 80 megabytes of text and code. We had added network hops, cold starts, API rate limits, and an extra layer of operational complexity — only to get worse retrieval accuracy on the things engineers care about most: **exact keywords, error codes, version numbers, and file paths.**\n\nI decided to tear it down and rebuild it from first principles.\n\nNo cloud databases. No heavyweight containers. Just **SQLite FTS5, dense embeddings, and Reciprocal Rank Fusion (RRF)**.\n\nHere is why this hybrid architecture consistently beats pure semantic search, and how you can implement it in less than 60 lines of clean Python.\n\nDense vector search is great at understanding intent. If a user asks *\"How do I restart the server?\"*, semantic search will easily find a document explaining *\"Rebooting your instance.\"*\n\nBut in production systems, users don't just ask philosophical questions. They search for:\n\n`ERR_CONN_REFUSED_0x82`\n\n`SKU-4982-A`\n\n`def compute_rrf_rank(...)`\n\n`$14,250 quarterly budget`\n\nDense embeddings smash these unique tokens into a dense latent space, smoothing out the sharp edges that give exact tokens their identity. The result? **Semantic hallucination in retrieval.**\n\nOn the other hand, classic **BM25 lexical search** (the foundation of Lucene and Elasticsearch) excels precisely where vector search fails: exact token matching, term frequency, and inverse document frequency.\n\nThe solution isn't picking one over the other. It's combining them.\n\n```\n                  ┌──────────────────────┐\n                  │      User Query      │\n                  └──────────┬───────────┘\n                             │\n            ┌────────────────┴────────────────┐\n            ▼                                 ▼\n   ┌──────────────────┐              ┌──────────────────┐\n   │   SQLite FTS5    │              │ Dense Embeddings │\n   │ (BM25 Sparse)    │              │ (Cosine Vector)  │\n   └────────┬─────────┘              └────────┬─────────┘\n            │                                 │\n     Top-K Ranked List                 Top-K Ranked List\n            │                                 │\n            └────────────────┬────────────────┘\n                             ▼\n               ┌───────────────────────────┐\n               │  Reciprocal Rank Fusion   │\n               │         (k = 60)          │\n               └─────────────┬─────────────┘\n                             ▼\n                  Final Hybrid Top Hits\n```\n\nMost engineers forget that **SQLite already ships with one of the fastest full-text search engines on the planet (FTS5)** right inside the standard library.\n\n`unicode61`\n\n, trigram, prefix matching).`.db`\n\nfile that you can commit to Git or mount anywhere.When you pair SQLite FTS5 with a local embedding model (or a fast API like Google's `gemini-embedding-001`\n\nor Cohere Embed), you have a complete, self-contained search engine.\n\nWhen you run both BM25 and Cosine Similarity, you get two sets of scores:\n\n`8.45`\n\n, `14.20`\n\n, `2.10`\n\n).`-1.0`\n\nand `1.0`\n\n(or `0.0`\n\nto `1.0`\n\n).Trying to normalize and sum these raw scores is a trap. If one query produces a massive BM25 outlier, it completely drowns out the semantic engine.\n\nThis is where **Reciprocal Rank Fusion (RRF)** comes in.\n\nInstead of looking at arbitrary score numbers, RRF looks at the **rank order** of results from each engine:\n\n$$RRF(d) = \\sum_{m \\in M} \\frac{1}{k + r_m(d)}$$\n\nWhere:\n\nIf a document is ranked #1 in BM25 and #2 in semantic search, it gets a massive boost. If it only appears in one engine at rank #20, its score decays smoothly. It is deterministic, immune to score scale mismatches, and requires zero hyperparameter tuning.\n\nHere is a minimal, complete implementation using standard Python and SQLite. You can copy and run this directly:\n\n``` python\nimport sqlite3\nimport math\nfrom typing import List, Dict, Tuple\n\nclass LocalHybridSearch:\n    def __init__(self, db_path: str = \":memory:\"):\n        self.conn = sqlite3.connect(db_path)\n        self.cursor = self.conn.cursor()\n        self._setup_db()\n        self.vectors = {}  # doc_id -> list[float]\n\n    def _setup_db(self):\n        self.cursor.execute(\"\"\"\n            CREATE VIRTUAL TABLE IF NOT EXISTS docs_fts USING fts5(\n                doc_id UNINDEXED,\n                title,\n                content,\n                tokenize='unicode61 remove_diacritics 2'\n            );\n        \"\"\")\n        self.conn.commit()\n\n    def add_document(self, doc_id: str, title: str, content: str, embedding: List[float]):\n        self.cursor.execute(\n            \"INSERT INTO docs_fts(doc_id, title, content) VALUES (?, ?, ?)\",\n            (doc_id, title, content)\n        )\n        self.conn.commit()\n        self.vectors[doc_id] = embedding\n\n    def _bm25_search(self, query: str, top_k: int = 20) -> List[Tuple[str, float]]:\n        # Clean query tokens for FTS5 syntax\n        clean_tokens = [f'\"{w}\"' for w in query.replace('\"', '').split() if w.strip()]\n        if not clean_tokens:\n            return []\n        match_expr = \" OR \".join(clean_tokens)\n\n        self.cursor.execute(\"\"\"\n            SELECT doc_id, bm25(docs_fts) as score\n            FROM docs_fts\n            WHERE docs_fts MATCH ?\n            ORDER BY score ASC LIMIT ?\n        \"\"\", (match_expr, top_k))\n\n        # SQLite bm25() returns lower/negative values for better matches\n        return [(row[0], abs(float(row[1]))) for row in self.cursor.fetchall()]\n\n    def _dense_search(self, query_vec: List[float], top_k: int = 20) -> List[Tuple[str, float]]:\n        def cosine_sim(a: List[float], b: List[float]) -> float:\n            dot = sum(x * y for x, y in zip(a, b))\n            norm_a = math.sqrt(sum(x * x for x in a))\n            norm_b = math.sqrt(sum(y * y for y in b))\n            return dot / (norm_a * norm_b) if norm_a and norm_b else 0.0\n\n        scores = [(doc_id, cosine_sim(query_vec, vec)) for doc_id, vec in self.vectors.items()]\n        scores.sort(key=lambda x: x[1], reverse=True)\n        return scores[:top_k]\n\n    def search(self, query: str, query_vec: List[float], top_k: int = 5, k_rrf: int = 60) -> List[Dict]:\n        bm25_results = self._bm25_search(query, top_k=20)\n        dense_results = self._dense_search(query_vec, top_k=20)\n\n        rrf_scores = {}\n        for rank, (doc_id, _) in enumerate(bm25_results, start=1):\n            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + (1.0 / (k_rrf + rank))\n\n        for rank, (doc_id, _) in enumerate(dense_results, start=1):\n            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + (1.0 / (k_rrf + rank))\n\n        sorted_hits = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)[:top_k]\n        return [{\"doc_id\": doc_id, \"rrf_score\": round(score, 4)} for doc_id, score in sorted_hits]\n```\n\n`hybrid-rag-action`\n\nI packaged this exact pattern into an open-source tool called [Hybrid RAG GitHub Action](https://github.com/Cagrik34/hybrid-rag-action).\n\nSeeing this architecture get officially applied, verified, and published on the **GitHub Marketplace (Microsoft/GitHub ecosystem)** was a genuinely proud and humbling milestone for me.\n\nWhenever a new issue or pull request is opened in a repository, the action:\n\nBecause it has zero external database dependencies, the whole pipeline runs directly inside GitHub Actions runners in **under 3 seconds** with zero infrastructure overhead.\n\nBeyond this live integration, the same architecture is also being submitted and shared as a reference recipe across **10+ major open-source AI ecosystems and repositories**, including the **Google Gemini Cookbook**, **Meta Llama Cookbook**, and **Stanford DSPy**.\n\nLet's be realistic. You *do* need Milvus, Qdrant, or Pinecone if:\n\nBut if your corpus is under **1 million chunks** (which accounts for ~90% of internal company knowledge bases, codebases, and documentation sites), spinning up a cloud vector cluster is massive over-engineering.\n\nSQLite FTS5 + in-memory or SQLite-backed dense vector similarity gives you:\n\nThe AI industry spent the last two years convincing engineers that everything needs to be a vector.\n\nVectors are great, but language is nuanced. Sometimes the best way to find `ERR_404_NULL_POINTER`\n\nis not through a 1536-dimensional cosine angle — it's through good old-fashioned inverted index token matching.\n\nGive SQLite FTS5 + RRF a try in your next RAG project. You might find you don't need another SaaS subscription after all.\n\nI’d love to hear your thoughts and experiences with hybrid retrieval in the comments below.\n\n*Open Source & Project Links:*", "url": "https://wpnews.pro/news/i-ditched-cloud-vector-databases-for-sqlite-fts5-and-my-rag-pipeline-got-10x", "canonical_source": "https://dev.to/cagrik34/i-ditched-cloud-vector-databases-for-sqlite-fts5-and-my-rag-pipeline-got-10x-better-759", "published_at": "2026-08-28 13:54:35+00:00", "updated_at": "2026-08-28 14:20:33.603443+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools"], "entities": ["SQLite", "FTS5", "BM25", "Reciprocal Rank Fusion", "Google", "Cohere"], "alternates": {"html": "https://wpnews.pro/news/i-ditched-cloud-vector-databases-for-sqlite-fts5-and-my-rag-pipeline-got-10x", "markdown": "https://wpnews.pro/news/i-ditched-cloud-vector-databases-for-sqlite-fts5-and-my-rag-pipeline-got-10x.md", "text": "https://wpnews.pro/news/i-ditched-cloud-vector-databases-for-sqlite-fts5-and-my-rag-pipeline-got-10x.txt", "jsonld": "https://wpnews.pro/news/i-ditched-cloud-vector-databases-for-sqlite-fts5-and-my-rag-pipeline-got-10x.jsonld"}}