# The LLM Was the Easy Part: Building a Hybrid RAG API > Source: > Published: 2026-07-16 17:04:10+00:00 A basic Retrieval-Augmented Generation (RAG) demo is surprisingly small: But when I turned that flow into an API, the LLM call became the least interesting part. I needed to process PDFs without blocking requests, combine semantic and keyword search, rerank noisy results, preserve source metadata, cache answers, and secure the API. So I built a PDF question-answering backend with: This article focuses on the most interesting part: the path from a user’s question to a grounded answer. The application has two main workflows. When a client uploads a PDF, the API: When a question arrives, the API: Here is the complete query flow: ``` ┌─────────────────┐ │ User question │ └────────┬────────┘ │ ┌────────────┴────────────┐ │ │ ▼ ▼ ┌────────────────┐ ┌────────────────┐ │ Dense embedding│ │ Sparse vector │ └───────┬────────┘ └───────┬────────┘ │ │ └───────────┬────────────┘ ▼ ┌───────────────────┐ │ Qdrant + RRF │ │ 20 candidates │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ Cross-encoder │ │ Top 5 chunks │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ Grounded prompt │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ LLM answer │ └───────────────────┘ ``` Dense embeddings are good at retrieving text by meaning. For example, a semantic search system may recognize that these sentences are related: ``` "How are API credentials invalidated?" "How can I revoke an access key?" ``` The wording is different, but the intent is similar. Technical documents also contain exact lexical signals: A semantic model may not always preserve the importance of an identifier such as: ``` ERR_AUTH_0042 ``` Sparse retrieval helps with those exact words and identifiers. Instead of choosing between semantic and lexical retrieval, I store both representations for every chunk: ``` PointStruct( id=point_id, vector={ "dense": dense_vector, "sparse": SparseVector( indices=sparse_vector["indices"], values=sparse_vector["values"], ), }, payload={ "text": chunk_text, "source": filename, "document_id": str(document_id), "page_number": page_number, "chunk_index": chunk_index, }, ) ``` Each Qdrant point contains: Keeping provenance next to the vectors makes it possible to return useful sources with each answer. Dense and sparse searches produce different score scales. Adding their raw scores directly would require normalization and tuning. Instead, I use reciprocal rank fusion, or RRF. RRF focuses on where a result appears in each ranked list rather than directly comparing the original scores. The hybrid query looks like this: ``` response = await qdrant_client.query_points( collection_name="embeddings", prefetch=[ Prefetch( query=dense_query_vector, using="dense", limit=limit * 4, ), Prefetch( query=SparseVector( indices=sparse_query["indices"], values=sparse_query["values"], ), using="sparse", limit=limit * 4, ), ], query=FusionQuery(fusion=Fusion.RRF), limit=limit, with_payload=True, ) ``` Qdrant executes the dense and sparse searches and then fuses their rankings. This allows a chunk to rank well because it: Hybrid retrieval is not automatically better for every dataset. Its value depends on the documents, query patterns, embedding models, and search configuration. It still needs evaluation against real questions. Initial retrieval needs to be fast enough to search the full collection. It does not always need to produce the final ordering. My pipeline retrieves 20 candidates and sends them to a cross-encoder: ``` pairs = [ (query, candidate["text"]) for candidate in candidates ] scores = reranker.predict(pairs) ``` The candidates are sorted using those scores: ``` reranked = sorted( zip(candidates, scores), key=lambda item: item[1](1), reverse=True, ) top_chunks = [ candidate for candidate, score in reranked[:5] ] ``` Unlike independent vector embeddings, a cross-encoder examines the question and candidate together. That can produce a more precise relevance score, but it is also more computationally expensive. This is why I use it only after the initial retrieval stage. The pipeline narrows the context like a funnel: ``` Hybrid retrieval ████████████████████ 20 candidates Cross-encoder output █████ 5 chunks LLM context █████ 5 chunks ``` These bars show the candidate counts configured in the code. They are not benchmark results or accuracy measurements. After reranking, the five best chunks are joined into a context block. The prompt tells the model to use only that context: ``` system_prompt = ( "Answer the question using only the provided context. " "If the answer is not present in the context, say that " "the available documents do not contain enough information." ) user_prompt = f""" Context: {context} Question: {question} """ ``` This instruction establishes a clear contract: A prompt cannot guarantee factual correctness. If retrieval returns irrelevant chunks, the generator still receives poor evidence. That is why I think of RAG quality as a chain: ``` Document quality × Chunk quality × Retrieval quality × Reranking quality × Generation quality = Final answer quality ``` A strong LLM cannot fully compensate for a weak retrieval pipeline. PDF ingestion includes several expensive operations: I did not want the upload request to remain open during that work. The endpoint creates the document record and schedules processing as a FastAPI background task: ``` background_tasks.add_task( process_document, document_id, pdf_bytes, file.filename, ) return { "document_id": str(document_id), "filename": file.filename, "processing_status": "processing", } ``` The client receives a response immediately and can check the status later: ``` GET /documents/{document_id} ``` The document moves through states such as: ``` processing → completed ↘ failed ``` This is enough for a prototype, but an in-process background task is not a durable job queue. If the API process stops, accepted work may be interrupted. For a more dependable version, I would move ingestion to a dedicated worker system with: Completed answers are cached in Redis for 24 hours. The current cache key is based on the question: ``` digest = hashlib.sha256( question.encode("utf-8") ).hexdigest() cache_key = f"rag:{digest}" ``` This is simple, but incomplete. The same question can produce a different answer when any of these change: A safer cache key would include those dependencies: ``` cache_input = { "question": normalized_question, "corpus_revision": corpus_revision, "filters": filters, "embedding_version": embedding_version, "reranker_version": reranker_version, "prompt_version": prompt_version, "generation_model": generation_model, } serialized = json.dumps( cache_input, sort_keys=True, ) digest = hashlib.sha256( serialized.encode("utf-8") ).hexdigest() ``` Caching is not just a performance optimization. A stale cache can return an answer that no longer reflects the current knowledge base. The retrieval pipeline is only one part of the service. The API also includes: The full system looks more like a backend platform than a single AI function: ``` ┌─────────────┐ │ API client │ └──────┬──────┘ │ ┌──────▼──────┐ │ FastAPI │ └──────┬──────┘ ┌─────────────────┼─────────────────┐ │ │ │ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ │ PostgreSQL │ │ Redis │ │ Qdrant │ │ Status/Auth │ │ Cache │ │ Retrieval │ └─────────────┘ └─────────────┘ └──────┬──────┘ │ ┌──────────▼──────────┐ │ Reranker and LLM │ └─────────────────────┘ ``` The LLM may be the most visible component, but most reliability problems live around it. The next version would focus on measurement and failure recovery. I would replace in-process background tasks with a proper worker queue. Stable point IDs would make retries safer and reduce duplicate chunks. Qdrant and PostgreSQL cannot share a transaction. A reconciliation process should detect and repair partial ingestion. Cache keys should include corpus, model, filter, and prompt versions. I would build a small evaluation dataset containing: Then I would compare: Useful retrieval metrics would include: I would also measure latency for each pipeline stage. Until those experiments exist, I would avoid claiming that one configuration is faster or more accurate than another. The most useful lesson from this project was that RAG is not one model call. It is a chain of systems: ``` Ingestion → Chunking → Embedding → Retrieval → Fusion → Reranking → Prompt construction → Generation → Caching → Evaluation ``` My current pipeline uses dense and sparse retrieval to find a broad candidate set, reciprocal rank fusion to combine the rankings, and a cross-encoder to select the final context. The LLM comes last. That is exactly why the LLM was the easy part. If you are building something similar, I would be interested to hear how you handle: Source code: [https://github.com/abuhurayraniloy/RAGEval](https://github.com/abuhurayraniloy/RAGEval) If this walkthrough was useful, consider leaving a comment or reaction.