{"slug": "the-llm-was-the-easy-part-building-a-hybrid-rag-api", "title": "The LLM Was the Easy Part: Building a Hybrid RAG API", "summary": "A developer built a hybrid RAG API for PDF question-answering, combining dense and sparse vector search with reciprocal rank fusion and a cross-encoder reranker. The system processes PDFs asynchronously, caches answers, and preserves source metadata to provide grounded responses.", "body_md": "A basic Retrieval-Augmented Generation (RAG) demo is surprisingly small:\n\nBut when I turned that flow into an API, the LLM call became the least interesting part.\n\nI needed to process PDFs without blocking requests, combine semantic and keyword search, rerank noisy results, preserve source metadata, cache answers, and secure the API.\n\nSo I built a PDF question-answering backend with:\n\nThis article focuses on the most interesting part: the path from a user’s question to a grounded answer.\n\nThe application has two main workflows.\n\nWhen a client uploads a PDF, the API:\n\nWhen a question arrives, the API:\n\nHere is the complete query flow:\n\n```\n                       ┌─────────────────┐\n                       │  User question  │\n                       └────────┬────────┘\n                                │\n                   ┌────────────┴────────────┐\n                   │                         │\n                   ▼                         ▼\n          ┌────────────────┐       ┌────────────────┐\n          │ Dense embedding│       │ Sparse vector  │\n          └───────┬────────┘       └───────┬────────┘\n                  │                        │\n                  └───────────┬────────────┘\n                              ▼\n                    ┌───────────────────┐\n                    │ Qdrant + RRF      │\n                    │ 20 candidates     │\n                    └─────────┬─────────┘\n                              ▼\n                    ┌───────────────────┐\n                    │ Cross-encoder     │\n                    │ Top 5 chunks      │\n                    └─────────┬─────────┘\n                              ▼\n                    ┌───────────────────┐\n                    │ Grounded prompt   │\n                    └─────────┬─────────┘\n                              ▼\n                    ┌───────────────────┐\n                    │ LLM answer        │\n                    └───────────────────┘\n```\n\nDense embeddings are good at retrieving text by meaning.\n\nFor example, a semantic search system may recognize that these sentences are related:\n\n```\n\"How are API credentials invalidated?\"\n\n\"How can I revoke an access key?\"\n```\n\nThe wording is different, but the intent is similar.\n\nTechnical documents also contain exact lexical signals:\n\nA semantic model may not always preserve the importance of an identifier such as:\n\n```\nERR_AUTH_0042\n```\n\nSparse retrieval helps with those exact words and identifiers.\n\nInstead of choosing between semantic and lexical retrieval, I store both representations for every chunk:\n\n```\nPointStruct(\n    id=point_id,\n    vector={\n        \"dense\": dense_vector,\n        \"sparse\": SparseVector(\n            indices=sparse_vector[\"indices\"],\n            values=sparse_vector[\"values\"],\n        ),\n    },\n    payload={\n        \"text\": chunk_text,\n        \"source\": filename,\n        \"document_id\": str(document_id),\n        \"page_number\": page_number,\n        \"chunk_index\": chunk_index,\n    },\n)\n```\n\nEach Qdrant point contains:\n\nKeeping provenance next to the vectors makes it possible to return useful sources with each answer.\n\nDense and sparse searches produce different score scales.\n\nAdding their raw scores directly would require normalization and tuning. Instead, I use reciprocal rank fusion, or RRF.\n\nRRF focuses on where a result appears in each ranked list rather than directly comparing the original scores.\n\nThe hybrid query looks like this:\n\n```\nresponse = await qdrant_client.query_points(\n    collection_name=\"embeddings\",\n    prefetch=[\n        Prefetch(\n            query=dense_query_vector,\n            using=\"dense\",\n            limit=limit * 4,\n        ),\n        Prefetch(\n            query=SparseVector(\n                indices=sparse_query[\"indices\"],\n                values=sparse_query[\"values\"],\n            ),\n            using=\"sparse\",\n            limit=limit * 4,\n        ),\n    ],\n    query=FusionQuery(fusion=Fusion.RRF),\n    limit=limit,\n    with_payload=True,\n)\n```\n\nQdrant executes the dense and sparse searches and then fuses their rankings.\n\nThis allows a chunk to rank well because it:\n\nHybrid retrieval is not automatically better for every dataset. Its value depends on the documents, query patterns, embedding models, and search configuration. It still needs evaluation against real questions.\n\nInitial retrieval needs to be fast enough to search the full collection.\n\nIt does not always need to produce the final ordering.\n\nMy pipeline retrieves 20 candidates and sends them to a cross-encoder:\n\n```\npairs = [\n    (query, candidate[\"text\"])\n    for candidate in candidates\n]\n\nscores = reranker.predict(pairs)\n```\n\nThe candidates are sorted using those scores:\n\n```\nreranked = sorted(\n    zip(candidates, scores),\n    key=lambda item: item<span class=\"footnote-wrapper\">[1](1)</span>,\n    reverse=True,\n)\n\ntop_chunks = [\n    candidate\n    for candidate, score in reranked[:5]\n]\n```\n\nUnlike independent vector embeddings, a cross-encoder examines the question and candidate together.\n\nThat can produce a more precise relevance score, but it is also more computationally expensive. This is why I use it only after the initial retrieval stage.\n\nThe pipeline narrows the context like a funnel:\n\n```\nHybrid retrieval       ████████████████████  20 candidates\nCross-encoder output   █████                  5 chunks\nLLM context            █████                  5 chunks\n```\n\nThese bars show the candidate counts configured in the code. They are not benchmark results or accuracy measurements.\n\nAfter reranking, the five best chunks are joined into a context block.\n\nThe prompt tells the model to use only that context:\n\n```\nsystem_prompt = (\n    \"Answer the question using only the provided context. \"\n    \"If the answer is not present in the context, say that \"\n    \"the available documents do not contain enough information.\"\n)\n\nuser_prompt = f\"\"\"\nContext:\n{context}\n\nQuestion:\n{question}\n\"\"\"\n```\n\nThis instruction establishes a clear contract:\n\nA prompt cannot guarantee factual correctness. If retrieval returns irrelevant chunks, the generator still receives poor evidence.\n\nThat is why I think of RAG quality as a chain:\n\n```\nDocument quality\n      ×\nChunk quality\n      ×\nRetrieval quality\n      ×\nReranking quality\n      ×\nGeneration quality\n      =\nFinal answer quality\n```\n\nA strong LLM cannot fully compensate for a weak retrieval pipeline.\n\nPDF ingestion includes several expensive operations:\n\nI did not want the upload request to remain open during that work.\n\nThe endpoint creates the document record and schedules processing as a FastAPI background task:\n\n```\nbackground_tasks.add_task(\n    process_document,\n    document_id,\n    pdf_bytes,\n    file.filename,\n)\n\nreturn {\n    \"document_id\": str(document_id),\n    \"filename\": file.filename,\n    \"processing_status\": \"processing\",\n}\n```\n\nThe client receives a response immediately and can check the status later:\n\n```\nGET /documents/{document_id}\n```\n\nThe document moves through states such as:\n\n```\nprocessing → completed\n           ↘ failed\n```\n\nThis is enough for a prototype, but an in-process background task is not a durable job queue.\n\nIf the API process stops, accepted work may be interrupted.\n\nFor a more dependable version, I would move ingestion to a dedicated worker system with:\n\nCompleted answers are cached in Redis for 24 hours.\n\nThe current cache key is based on the question:\n\n```\ndigest = hashlib.sha256(\n    question.encode(\"utf-8\")\n).hexdigest()\n\ncache_key = f\"rag:{digest}\"\n```\n\nThis is simple, but incomplete.\n\nThe same question can produce a different answer when any of these change:\n\nA safer cache key would include those dependencies:\n\n```\ncache_input = {\n    \"question\": normalized_question,\n    \"corpus_revision\": corpus_revision,\n    \"filters\": filters,\n    \"embedding_version\": embedding_version,\n    \"reranker_version\": reranker_version,\n    \"prompt_version\": prompt_version,\n    \"generation_model\": generation_model,\n}\n\nserialized = json.dumps(\n    cache_input,\n    sort_keys=True,\n)\n\ndigest = hashlib.sha256(\n    serialized.encode(\"utf-8\")\n).hexdigest()\n```\n\nCaching is not just a performance optimization. A stale cache can return an answer that no longer reflects the current knowledge base.\n\nThe retrieval pipeline is only one part of the service.\n\nThe API also includes:\n\nThe full system looks more like a backend platform than a single AI function:\n\n```\n                         ┌─────────────┐\n                         │ API client  │\n                         └──────┬──────┘\n                                │\n                         ┌──────▼──────┐\n                         │  FastAPI    │\n                         └──────┬──────┘\n              ┌─────────────────┼─────────────────┐\n              │                 │                 │\n       ┌──────▼──────┐   ┌──────▼──────┐  ┌──────▼──────┐\n       │ PostgreSQL  │   │    Redis    │  │   Qdrant   │\n       │ Status/Auth │   │    Cache    │  │  Retrieval │\n       └─────────────┘   └─────────────┘  └──────┬──────┘\n                                                 │\n                                      ┌──────────▼──────────┐\n                                      │ Reranker and LLM    │\n                                      └─────────────────────┘\n```\n\nThe LLM may be the most visible component, but most reliability problems live around it.\n\nThe next version would focus on measurement and failure recovery.\n\nI would replace in-process background tasks with a proper worker queue.\n\nStable point IDs would make retries safer and reduce duplicate chunks.\n\nQdrant and PostgreSQL cannot share a transaction. A reconciliation process should detect and repair partial ingestion.\n\nCache keys should include corpus, model, filter, and prompt versions.\n\nI would build a small evaluation dataset containing:\n\nThen I would compare:\n\nUseful retrieval metrics would include:\n\nI would also measure latency for each pipeline stage.\n\nUntil those experiments exist, I would avoid claiming that one configuration is faster or more accurate than another.\n\nThe most useful lesson from this project was that RAG is not one model call.\n\nIt is a chain of systems:\n\n```\nIngestion\n   → Chunking\n   → Embedding\n   → Retrieval\n   → Fusion\n   → Reranking\n   → Prompt construction\n   → Generation\n   → Caching\n   → Evaluation\n```\n\nMy current pipeline uses dense and sparse retrieval to find a broad candidate set, reciprocal rank fusion to combine the rankings, and a cross-encoder to select the final context.\n\nThe LLM comes last.\n\nThat is exactly why the LLM was the easy part.\n\nIf you are building something similar, I would be interested to hear how you handle:\n\nSource code: [https://github.com/abuhurayraniloy/RAGEval](https://github.com/abuhurayraniloy/RAGEval)\n\nIf this walkthrough was useful, consider leaving a comment or reaction.", "url": "https://wpnews.pro/news/the-llm-was-the-easy-part-building-a-hybrid-rag-api", "canonical_source": "https://dev.to/niloy_/the-llm-was-the-easy-part-building-a-hybrid-rag-api-26eh", "published_at": "2026-07-16 17:04:10+00:00", "updated_at": "2026-07-16 18:05:31.878047+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Qdrant", "RRF"], "alternates": {"html": "https://wpnews.pro/news/the-llm-was-the-easy-part-building-a-hybrid-rag-api", "markdown": "https://wpnews.pro/news/the-llm-was-the-easy-part-building-a-hybrid-rag-api.md", "text": "https://wpnews.pro/news/the-llm-was-the-easy-part-building-a-hybrid-rag-api.txt", "jsonld": "https://wpnews.pro/news/the-llm-was-the-easy-part-building-a-hybrid-rag-api.jsonld"}}