How Qdrant Reduced RAG Token Costs by 67% with Native ColBERT Reranking

Qdrant reduced retrieval-augmented generation token costs by 67% using native ColBERT reranking, which performs token-to-token matrix comparison inside the database in a single query call, eliminating the need for external reranking APIs. The approach was demonstrated on legal AI use cases, where sentence isolation cut unnecessary token consumption from redundant boilerplate text.

Jumping into today’s article, I’m going to take you down a quick comparison of how token bills are taking up most of your time and why every engineering team is sweating bullets over RAG costs. Even giants like Uber are actively looking at ways to slash their token consumption. Across industries, token consumption is a cause for stress. There was all this hype about replacing or augmenting software engineers with AI, but teams are discovering that running unoptimized LLM pipelines can quickly cost much more than the developers they were supposed to save money on So let’s talk about how to fix this. To show you how, I will focus on a sector which everyone has their eyes on right now: Legal AI . If you’ve ever watched the TV show “Suits” , you know the drill. Harvey Specter or Mike Ross gets handed a case, and suddenly there’s a dramatic scene of paralegals wheeling in fifty boxes of Master Services Agreements MSAs and corporate filings for “discovery.” Now, picture an attorney three weeks into that discovery process who’s landed on a contract dispute. They have one very specific compliance question: ”What is the cap on indemnification for third-party intellectual property infringement claims?” It’s a reasonable, direct question. A one-sentence answer exists somewhere inside a forty-page contract. But what do they get back from a standard RAG system? They get the whole page. Sometimes three whole pages. Why? Because the retriever pulled the top three chunks, and each chunk is 500 to 1,000 tokens of dense, boilerplate legalese recitals, definition sections, payment terms, and governing law clauses that have absolutely nothing to do with IP breaches. The one sentence they actually need is buried deep in the middle. This isn’t just annoying for the attorney who has to read it. It is incredibly expensive at scale. If you run this across thousands of corporate filings during a major litigation audit, you are paying your LLM provider to process mountains of redundant boilerplate. I wanted to know: How much of that text is actually necessary? Could a retrieval system isolate the exact sentence instead of the whole page? And could we do it without using a heavy, expensive second service to handle the reranking? If you look up standard two-stage retrieval architectures, the blueprint almost always looks like this: You run a fast, first-pass dense vector search to get the top 100 documents. Then, you make a second network call to an external reranking API like Cohere, SageMaker, or a custom Hugging Face container to sort the candidates. That works, but it has drawbacks: This is why I chose Qdrant . Since version 1.10, Qdrant has supported native multi-vector fields with a MAX SIM comparator. This is the exact mathematical operator behind ColBERT-style late interaction. Instead of exporting candidates to a separate model or service, Qdrant does the token-to-token matrix comparison inside the database, in a single query call . Normally, to get this level of accuracy, you would have to pay for a second reranking service like Cohere and wait for a second network hop. With Qdran t, we get that exact same late-interaction reranking accuracy completely native to the database, while using sentence isolation to cut token costs by 67%. By coupling this with Qdrant’s memory management, we get a highly optimized database profile: What we actually built: One database call, zero external reranking APIs Here’s the plan for our optimized pipeline: Note: Pricing calculated at $3.00 per million input tokens e.g., Claude 3.5 Sonnet . Here’s the Qdrant configuration script that runs our standard dense embeddings and ColBERT multi-vectors side by side: python from qdrant client import QdrantClient, models ADDED: Load FastEmbed models locally on CPUfrom fastembed import TextEmbedding, LateInteractionTextEmbeddingCOLLECTION NAME = "legal discovery"DENSE DIM = 384 BAAI/bge-small-en-v1.5COLBERT DIM = 128 colbert-ir/colbertv2.0 ADDED: Instantiate the vector modelsdense model = TextEmbedding "BAAI/bge-small-en-v1.5" colbert model = LateInteractionTextEmbedding "colbert-ir/colbertv2.0" client = QdrantClient "<http://localhost:6333 " client.create collection collection name=COLLECTION NAME, vectors config={ "dense": models.VectorParams size=DENSE DIM, distance=models.Distance.COSINE, quantization config=models.BinaryQuantization binary=models.BinaryQuantizationConfig always ram=True , , , "colbert": models.VectorParams size=COLBERT DIM, distance=models.Distance.COSINE, multivector config=models.MultiVectorConfig comparator=models.MultiVectorComparator.MAX SIM , on disk=True, hnsw config=models.HnswConfigDiff m=0 , , }, Notice the optimization in the ColBERT vector parameters: hnsw config=models.HnswConfigDiff m=0 . Because we only use ColBERT to rescore the pre-filtered candidate pool, we don’t need to build an HNSW index graph for ColBERT. This saves massive index-building time and storage space. With Qdrant’s universal Query API, we can execute the Stage 1 prefetch and the Stage 2 rescore in a single round-trip database operation. This is where the unified API shines. Instead of querying the database, parsing the IDs, and making a second network hop to a reranker, we orchestrate the entire prefetch-and-rescore flow in a single database round-trip: ADDED: Generate query embeddings ColBERT uses query embed to add prefix padding dense query = next dense model.query embed query .tolist colbert query = next colbert model.query embed query .tolist Run the two-stage query in one network round-tripresults = client.query points collection name=COLLECTION NAME, prefetch=models.Prefetch query=dense query, using="dense", limit=prefetch limit, params=models.SearchParams quantization=models.QuantizationSearchParams rescore=False , , , query=colbert query, using="colbert", limit=top k, with payload=True, Observe the rescore=False parameter inside the prefetch block. When you use Binary Quantization, Qdrant normally retrieves candidates using quantized vectors and then automatically loads the full-precision float32 dense vectors to rescore them. Because we are doing a more precise ColBERT token-level rescore in Stage 2 anyway, we turn off the dense rescoring. This avoids fetching full-precision dense vectors from disk entirely, shaving off database processing overhead. Qdrant’s MAX SIM rescoring gives you a single aggregate score per chunk. It doesn’t break down which specific tokens drove the score. To extract only the relevant text, we run a final, local step: we split the winning chunks into sentences and calculate the ColBERT late-interaction scores for each sentence locally. By computing late-interaction matrix scores locally on only the final candidate chunks where the text is small , we avoid database scaling bottlenecks while achieving surgical sentence-level precision on the CPU: python import reimport numpy as np ADDED: Basic sentence splitter regexSENTENCE SPLIT = re.compile r" ?<= .; \s+ ?= A-Z " def max sim query vecs: np.ndarray, doc vecs: np.ndarray - float: Compute token-to-token similarity matrix sims = query vecs @ doc vecs.T num query tokens, num doc tokens Sum the maximum similarity scores along the document axis return float sims.max axis=1 .sum def isolate sentences chunk text: str, query vecs: np.ndarray, colbert model, top n: int = 1 : ADDED: Split chunk text into candidate sentences sentences = s.strip for s in SENTENCE SPLIT.split chunk text if len s.strip 15 if not sentences: return chunk text, 0.0 Embed each sentence locally using ColBERT sentence vecs = list colbert model.embed sentences scored = sentences i , max sim query vecs, sentence vecs i for i in range len sentences scored.sort key=lambda pair: pair 1 , reverse=True return scored :top n This local calculation is extremely fast because it is only executed on the top 3 or 5 retrieved chunks. Once we isolate the top sentence s , we strip out the rest of the chunk and format the prompt payload: php def build optimized prompt query: str, chunk texts: list str , colbert model - str: query vecs = next colbert model.query embed query context parts = for i, text in enumerate chunk texts : top sentences = isolate sentences text, query vecs, colbert model, top n=1 isolated text = " ".join s for s, in top sentences context parts.append f" Source Chunk {i+1} : {isolated text}" context str = "\n\n".join context parts return f"Context:\n{context str}\n\nQuestion: {query}\nAnswer:" When optimizing token payloads, it is tempting to look at token reduction as the only metric that matters. If you chunk your documents into massive blocks — say, mashing 10 to 12 different legal sections into a single 2,000-word chunk — and then run sentence-level isolation, you can theoretically claim a significant 85%+ token reduction. But in a production legal RAG system, this approach degrades retrieval accuracy. When chunks are too large, the semantic signal of any specific clause like a third-party IP indemnification cap gets drowned out by the surrounding boilerplate in the dense vector space. Running queries against muddy, multi-clause chunks often returns unrelated sections like warranties because the dense retriever cannot isolate the target clause’s signal. To avoid this, we implement a strict one-legal-clause-per-chunk boundary strategy. We ensure every chunk contains exactly one distinct legal section, keeping the clause lengths realistic. Running the benchmarks under this production-grade chunking scheme yields a fairly accurate and verified 67.1% token reduction in the LLM prompt. By prioritizing clean, section-level boundaries, you ensure the retrieval actually finds the right clause. I manually checked this against the indemnification and IP-ownership queries and confirmed the results, even though the headline token savings dropped from a theoretical 87.3% to a real-world 67.1% . I expected the two-stage pipeline to search faster than plain dense search because of binary quantization. It doesn’t. Across repeated runs, two-stage search is consistently 2x to 4x slower than plain dense search, depending on the machine and cache warming. In raw numbers, we are looking at 39.5ms vs. 20.4ms . This is completely logical: while dense-only search just does a single embedding lookup, the two-stage pipeline has to run both query encoders dense and ColBERT and execute a token comparison matrix inside Qdrant. But look at the latency sweep when we raise the prefetch limit standard dense-only search reference: 7.4 ms avg : Raising the prefetch limit from 10 candidates all the way to 500 barely moves the needle; latency stays in the same general band. The bigger driver of total latency is the inclusion of the two-stage pipeline itself rather than the size of the prefetch window. More importantly, end-to-end RAG latency is dominated by LLM generation time, not database queries. Consequently, saving 400+ tokens per query cuts downstream LLM processing and generation time by hundreds of milliseconds, easily making the overall end-to-end application significantly faster. Let’s look at the financial impact of a 67.1% token reduction using standard input pricing of $3.00 per million input tokens : At a litigation-discovery scale of 500,000 queries, that translates to $600+ saved per case audit , not to mention the massive performance gains from avoiding the LLM’s “lost-in-the-middle” effect. If you are building this for your team, keep these takeaways in mind: By using Qdrant’s native multi-vector comparator and local sentence-level scoring, you can build a highly precise, cost-efficient RAG system that extracts the exact clause your users need without inflating your monthly API bills. Feel free to experiment with it yourself here https://github.com/Cappybara12/legal-rag . This implementation builds upon recent advances in neural information retrieval and vector search. The late-interaction retrieval paradigm introduced by ColBERT enables token-level semantic matching without the computational overhead of traditional cross-encoders, while ColBERTv2 further improves retrieval quality and efficiency. Qdrant’s native multivector support and binary quantization capabilities make it possible to deploy these techniques in production with a single database query, reducing infrastructure complexity and memory overhead. Readers interested in the underlying concepts and implementation details can explore the following resources for a deeper understanding of late-interaction retrieval, multivector search, and production-scale vector databases. How Qdrant Reduced RAG Token Costs by 67% with Native ColBERT Reranking https://pub.towardsai.net/how-qdrant-reduced-rag-token-costs-by-67-with-native-colbert-reranking-98b4b4d4d553 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.