{"slug": "anatomy-of-a-full-rag-application-every-concept-one-self-hosted-stack", "title": "Anatomy of a Full RAG Application: Every Concept, One Self-Hosted Stack", "summary": "A developer built myRAG, a fully self-hosted RAG stack combining FastAPI, React, Qdrant, PostgreSQL, and Neo4j. The system uses hybrid search with dense and sparse embeddings, cross-encoder reranking, and knowledge graph triples to improve retrieval accuracy for complex queries. The project demonstrates a complete pipeline from document ingestion to streaming LLM responses.", "body_md": "\"Chat with your documents\" sounds simple. Then you build it, and you discover a good RAG system is really eight systems wearing a trench coat.\n\nI recently finished **myRAG** — a fully self-hosted RAG stack: FastAPI backend, React frontend, and three storage engines (Qdrant, PostgreSQL, Neo4j), all orchestrated with Docker Compose. This post walks through every stage of the pipeline and the *concept* behind it, with real code from the project.\n\nFor a link to the codebase, scroll to bottom!\n\n```\nDocument ─► Docling ─► Chunker ─┬─► Dense embed (OpenRouter) ──┐\n                                ├─► Sparse BM25 (fastembed) ───┼─► Qdrant (named vectors)\n                                └─► LLM triple extraction ─────┼─► Neo4j (knowledge graph)\n                                                               └─► PostgreSQL (metadata, doc_uuid)\n\nQuestion ─► hybrid search (RRF) ─► rerank ─► + graph facts ─► + memory ─► token budget ─► LLM ─► SSE stream\n```\n\nSix containers: the app, the React/nginx frontend, `docling-serve`\n\n, Qdrant, Postgres, and Neo4j.\n\nEverything starts with converting PDFs/DOCX/HTML into text an LLM can use. I run [Docling](https://github.com/docling-project/docling) as a separate service that returns clean **Markdown**, preserving headings and tables.\n\nWhy Markdown? Because structure survives. A table flattened into a character soup is unsearchable no matter how good your embedding model is. Garbage in, garbage out — this stage silently determines your ceiling.\n\nChunks are the retrieval unit. Too large and the embedding becomes a blurry average of many topics; too small and the chunk loses the context needed to be understood alone.\n\n```\nchunking:\n  strategy: recursive\n  chunk_size: 512\n  chunk_overlap: 64\n```\n\nRecursive splitting respects paragraph and sentence boundaries, and the **overlap** is essential: without it, answers that straddle a chunk boundary fall into the gap and are never retrieved whole.\n\nMost tutorials embed once and call it done. I index every chunk **two ways**:\n\n`fastembed`\n\n, no API cost) — lexical similarity. Exact part numbers, names, acronyms — the things embedding models fumble.Qdrant stores both on the same point as *named vectors*, with BM25's IDF computed server-side:\n\n```\nself.client.create_collection(\n    collection_name=...,\n    vectors_config={\"dense\": VectorParams(size=4096, distance=Distance.COSINE)},\n    sparse_vectors_config={\"bm25\": models.SparseVectorParams(modifier=models.Modifier.IDF)},\n)\n```\n\nOne subtlety: BM25 weights documents and queries differently, so ingestion uses `embed()`\n\n(term weighting) while queries use `query_embed()`\n\n(term presence).\n\nAt query time, both searches run and Qdrant fuses them with **RRF** — which merges *ranked lists* instead of trying to normalize incomparable score scales:\n\n```\nresult = self.client.query_points(\n    collection_name=...,\n    prefetch=[\n        models.Prefetch(query=dense_vector, using=\"dense\", limit=20),\n        models.Prefetch(query=sparse_query, using=\"bm25\", limit=20),\n    ],\n    query=models.FusionQuery(fusion=models.Fusion.RRF),\n    limit=10,\n)\n```\n\nRRF scores each document by `Σ 1/(k + rank)`\n\nacross lists. A chunk ranked highly by *either* method surfaces; one ranked highly by *both* wins. No score calibration, no tuning — and it works disturbingly well.\n\nVector search compares a query against chunks that were embedded *without knowing the question*. A **cross-encoder reranker** (Cohere rerank, via OpenRouter) reads query and chunk *together* and produces a much sharper relevance score.\n\nThe pattern is a funnel: hybrid-retrieve 10 candidates cheaply, rerank down to the best 5. It's the same two-stage architecture search engines have used for decades — recall first, precision second.\n\nPure vector RAG struggles with relational questions — *\"Who reports to the person who founded X?\"* spans facts that live in different chunks.\n\nSo during ingestion, an LLM extracts **(subject, relation, object)** triples from every chunk into Neo4j:\n\n```\nYou are a knowledge-graph extraction engine. From the text below, extract factual\nrelationships as a JSON array of triples... Only extract relationships explicitly\nstated in the text. Return ONLY the JSON array, no prose.\n```\n\nAt query time the flow is: extract entities from the question → match them against graph nodes via Neo4j's fulltext index → pull their 1-hop neighborhood → inject the triples into the prompt as structured facts:\n\n``` php\nKnowledge graph facts:\n- Acme Corp --acquired--> Widget Inc\n- Widget Inc --founded_by--> Jane Doe\n```\n\nCall it GraphRAG-lite: the vector store answers *\"what's relevant,\"* the graph answers *\"how things relate.\"* Every relationship is tagged with `doc_uuid`\n\n, so deleting a document prunes its facts (and any orphaned entities) cleanly.\n\nMulti-turn chat needs history, but the context window is finite. Two mechanisms:\n\n**Rolling summarization** — after N turns, older exchanges are compressed into a running summary by a small LLM. Long conversations cost a paragraph, not pages.\n\n**Token budgeting** — before generation, the prompt is assembled against a hard cap with explicit priorities:\n\n```\n# Priority (always kept): system prompt, summary, graph facts, the question.\n# Then newest history, then chunks best-first; lowest-ranked chunks drop first.\nwhile history_msgs and fixed + est(history_msgs) > budget:\n    history_msgs.pop(0)          # oldest history goes first\n\nfor chunk in chunks:             # rerank order: best first\n    if used + est(chunk) > avail and len(kept) >= min_chunks:\n        break\n    kept.append(chunk)\n```\n\nThe mental model: every token of history you keep is a token of evidence you can't include. Making that trade-off explicit — instead of letting the API truncate arbitrarily — noticeably improves long conversations.\n\nIngestion is embarrassingly parallel — embedding API calls, local BM25 encoding, and per-chunk graph extraction are all independent I/O. A single shared thread pool handles the fan-out:\n\n``` python\n# executor.py — one process-wide pool, order-preserving map\ndef map_parallel(fn, items):\n    items = list(items)\n    if len(items) <= 1:\n        return [fn(i) for i in items]\n    return list(get_pool().map(fn, items))\n```\n\nThree applications:\n\n```\n# 1. Dense embedding: sub-batches of 32 texts, requests in flight concurrently\nbatches = [texts[i:i+32] for i in range(0, len(texts), 32)]\nvectors = [v for batch in map_parallel(self._embed_request, batches) for v in batch]\n\n# 2. Sparse BM25 runs concurrently with the dense API round-trips\nsparse_future = get_pool().submit(self.sparse_embedder.embed_batch, texts)\nvectors = self.embedder.embed_batch(texts)\nsparse_vectors = sparse_future.result()\n\n# 3. Graph extraction: one LLM call per chunk, previously sequential — now fanned out\nfutures = [get_pool().submit(self.graph_extractor.extract, t) for t in texts]\nfor i, future in enumerate(as_completed(futures)):\n    ...  # upsert triples + yield progress events as each completes\n```\n\nAt query time, the graph fact lookup is submitted before retrieval starts, so it overlaps with retrieval + reranking instead of running after them. Graph extraction was the slowest ingestion phase by far — parallelizing it is the difference between 90 seconds and 15 for a mid-sized document. No async rewrite needed; threads are plenty for I/O-bound work.\n\n`doc_uuid`\n\nties each document across Postgres (primary key), Qdrant (point payload), and Neo4j (relationship property). Deletes cascade through all three stores.`parsing → chunking → embedding → storing → graph → done`\n\nlive. One gotcha: `EventSource`\n\ncan't send headers, so the client streams via `fetch`\n\n+ `ReadableStream`\n\nto pass the auth token.`/api/*`\n\nendpoint requires a bearer token, checked with `secrets.compare_digest`\n\n(constant-time). The React build bakes the key in at build time from an env var — no login screen; docker-compose sources frontend and backend from the same `.env`\n\nso they can't drift.`config.yaml`\n\nfor every tunable, Pydantic Settings overlays secrets from `.env`\n\n, zero hardcoded values.RAG isn't one technique. It's a pipeline of small, well-understood ideas:\n\nParse cleanly → chunk thoughtfully → index twice → fuse ranks → rerank deeply → add structure with a graph → budget your tokens → parallelize the waits.\n\nNone of these steps is hard alone. The engineering is in making them agree with each other — sharing one UUID, one config, one thread pool, and one honest token budget.\n\nAccess the full codebase here: [https://github.com/sumannath/myRAG](https://github.com/sumannath/myRAG)\n\nQuestions about any layer? The hybrid search and knowledge-graph stages delivered the biggest quality jumps for me, and I'm happy to go deeper on either in the comments.", "url": "https://wpnews.pro/news/anatomy-of-a-full-rag-application-every-concept-one-self-hosted-stack", "canonical_source": "https://dev.to/sumanpro/i-built-a-production-grade-rag-application-from-scratch-heres-every-concept-that-goes-into-one-3a22", "published_at": "2026-07-12 13:35:29+00:00", "updated_at": "2026-07-12 14:16:13.990987+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["myRAG", "FastAPI", "React", "Qdrant", "PostgreSQL", "Neo4j", "Docling", "OpenRouter"], "alternates": {"html": "https://wpnews.pro/news/anatomy-of-a-full-rag-application-every-concept-one-self-hosted-stack", "markdown": "https://wpnews.pro/news/anatomy-of-a-full-rag-application-every-concept-one-self-hosted-stack.md", "text": "https://wpnews.pro/news/anatomy-of-a-full-rag-application-every-concept-one-self-hosted-stack.txt", "jsonld": "https://wpnews.pro/news/anatomy-of-a-full-rag-application-every-concept-one-self-hosted-stack.jsonld"}}