The LLM Was the Easy Part: Building a Hybrid RAG API

A developer built a hybrid RAG API for PDF question-answering, combining dense and sparse vector search with reciprocal rank fusion and a cross-encoder reranker. The system processes PDFs asynchronously, caches answers, and preserves source metadata to provide grounded responses.

A basic Retrieval-Augmented Generation RAG demo is surprisingly small: But when I turned that flow into an API, the LLM call became the least interesting part. I needed to process PDFs without blocking requests, combine semantic and keyword search, rerank noisy results, preserve source metadata, cache answers, and secure the API. So I built a PDF question-answering backend with: This article focuses on the most interesting part: the path from a user’s question to a grounded answer. The application has two main workflows. When a client uploads a PDF, the API: When a question arrives, the API: Here is the complete query flow: ┌─────────────────┐ │ User question │ └────────┬────────┘ │ ┌────────────┴────────────┐ │ │ ▼ ▼ ┌────────────────┐ ┌────────────────┐ │ Dense embedding│ │ Sparse vector │ └───────┬────────┘ └───────┬────────┘ │ │ └───────────┬────────────┘ ▼ ┌───────────────────┐ │ Qdrant + RRF │ │ 20 candidates │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ Cross-encoder │ │ Top 5 chunks │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ Grounded prompt │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ LLM answer │ └───────────────────┘ Dense embeddings are good at retrieving text by meaning. For example, a semantic search system may recognize that these sentences are related: "How are API credentials invalidated?" "How can I revoke an access key?" The wording is different, but the intent is similar. Technical documents also contain exact lexical signals: A semantic model may not always preserve the importance of an identifier such as: ERR AUTH 0042 Sparse retrieval helps with those exact words and identifiers. Instead of choosing between semantic and lexical retrieval, I store both representations for every chunk: PointStruct id=point id, vector={ "dense": dense vector, "sparse": SparseVector indices=sparse vector "indices" , values=sparse vector "values" , , }, payload={ "text": chunk text, "source": filename, "document id": str document id , "page number": page number, "chunk index": chunk index, }, Each Qdrant point contains: Keeping provenance next to the vectors makes it possible to return useful sources with each answer. Dense and sparse searches produce different score scales. Adding their raw scores directly would require normalization and tuning. Instead, I use reciprocal rank fusion, or RRF. RRF focuses on where a result appears in each ranked list rather than directly comparing the original scores. The hybrid query looks like this: response = await qdrant client.query points collection name="embeddings", prefetch= Prefetch query=dense query vector, using="dense", limit=limit 4, , Prefetch query=SparseVector indices=sparse query "indices" , values=sparse query "values" , , using="sparse", limit=limit 4, , , query=FusionQuery fusion=Fusion.RRF , limit=limit, with payload=True, Qdrant executes the dense and sparse searches and then fuses their rankings. This allows a chunk to rank well because it: Hybrid retrieval is not automatically better for every dataset. Its value depends on the documents, query patterns, embedding models, and search configuration. It still needs evaluation against real questions. Initial retrieval needs to be fast enough to search the full collection. It does not always need to produce the final ordering. My pipeline retrieves 20 candidates and sends them to a cross-encoder: pairs = query, candidate "text" for candidate in candidates scores = reranker.predict pairs The candidates are sorted using those scores: reranked = sorted zip candidates, scores , key=lambda item: item<span class="footnote-wrapper" 1 1 </span , reverse=True, top chunks = candidate for candidate, score in reranked :5 Unlike independent vector embeddings, a cross-encoder examines the question and candidate together. That can produce a more precise relevance score, but it is also more computationally expensive. This is why I use it only after the initial retrieval stage. The pipeline narrows the context like a funnel: Hybrid retrieval ████████████████████ 20 candidates Cross-encoder output █████ 5 chunks LLM context █████ 5 chunks These bars show the candidate counts configured in the code. They are not benchmark results or accuracy measurements. After reranking, the five best chunks are joined into a context block. The prompt tells the model to use only that context: system prompt = "Answer the question using only the provided context. " "If the answer is not present in the context, say that " "the available documents do not contain enough information." user prompt = f""" Context: {context} Question: {question} """ This instruction establishes a clear contract: A prompt cannot guarantee factual correctness. If retrieval returns irrelevant chunks, the generator still receives poor evidence. That is why I think of RAG quality as a chain: Document quality × Chunk quality × Retrieval quality × Reranking quality × Generation quality = Final answer quality A strong LLM cannot fully compensate for a weak retrieval pipeline. PDF ingestion includes several expensive operations: I did not want the upload request to remain open during that work. The endpoint creates the document record and schedules processing as a FastAPI background task: background tasks.add task process document, document id, pdf bytes, file.filename, return { "document id": str document id , "filename": file.filename, "processing status": "processing", } The client receives a response immediately and can check the status later: GET /documents/{document id} The document moves through states such as: processing → completed ↘ failed This is enough for a prototype, but an in-process background task is not a durable job queue. If the API process stops, accepted work may be interrupted. For a more dependable version, I would move ingestion to a dedicated worker system with: Completed answers are cached in Redis for 24 hours. The current cache key is based on the question: digest = hashlib.sha256 question.encode "utf-8" .hexdigest cache key = f"rag:{digest}" This is simple, but incomplete. The same question can produce a different answer when any of these change: A safer cache key would include those dependencies: cache input = { "question": normalized question, "corpus revision": corpus revision, "filters": filters, "embedding version": embedding version, "reranker version": reranker version, "prompt version": prompt version, "generation model": generation model, } serialized = json.dumps cache input, sort keys=True, digest = hashlib.sha256 serialized.encode "utf-8" .hexdigest Caching is not just a performance optimization. A stale cache can return an answer that no longer reflects the current knowledge base. The retrieval pipeline is only one part of the service. The API also includes: The full system looks more like a backend platform than a single AI function: ┌─────────────┐ │ API client │ └──────┬──────┘ │ ┌──────▼──────┐ │ FastAPI │ └──────┬──────┘ ┌─────────────────┼─────────────────┐ │ │ │ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ │ PostgreSQL │ │ Redis │ │ Qdrant │ │ Status/Auth │ │ Cache │ │ Retrieval │ └─────────────┘ └─────────────┘ └──────┬──────┘ │ ┌──────────▼──────────┐ │ Reranker and LLM │ └─────────────────────┘ The LLM may be the most visible component, but most reliability problems live around it. The next version would focus on measurement and failure recovery. I would replace in-process background tasks with a proper worker queue. Stable point IDs would make retries safer and reduce duplicate chunks. Qdrant and PostgreSQL cannot share a transaction. A reconciliation process should detect and repair partial ingestion. Cache keys should include corpus, model, filter, and prompt versions. I would build a small evaluation dataset containing: Then I would compare: Useful retrieval metrics would include: I would also measure latency for each pipeline stage. Until those experiments exist, I would avoid claiming that one configuration is faster or more accurate than another. The most useful lesson from this project was that RAG is not one model call. It is a chain of systems: Ingestion → Chunking → Embedding → Retrieval → Fusion → Reranking → Prompt construction → Generation → Caching → Evaluation My current pipeline uses dense and sparse retrieval to find a broad candidate set, reciprocal rank fusion to combine the rankings, and a cross-encoder to select the final context. The LLM comes last. That is exactly why the LLM was the easy part. If you are building something similar, I would be interested to hear how you handle: Source code: https://github.com/abuhurayraniloy/RAGEval https://github.com/abuhurayraniloy/RAGEval If this walkthrough was useful, consider leaving a comment or reaction.