Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking

Researchers fine-tuned LLaMA 3 (8B) as a drop-in reranker for RAG pipelines, using supervised fine-tuning and 4-bit quantization to replace traditional cross-encoders. The model achieved up to 21% improvement in answer correctness and 19% in answer similarity on a domain-specific QA benchmark while reducing inference costs. This approach demonstrates that instruction-tuned LLMs can serve as efficient rerankers without the quadratic complexity of cross-encoders.

arXiv:2607.11933v1 Announce Type: new Abstract: Cross-encoders achieve high reranking accuracy in Retrieval-Augmented Generation RAG pipelines but impose quadratic inference costs that limit real-time deployment. We address this by fine-tuning LLaMA 3 8B as a drop-in reranker using a two-stage pipeline: supervised fine-tuning on a custom query-document relevance dataset via the Unsloth framework with LoRA adapters, followed by 4-bit quantization for efficient inference. The resulting model replaces the cross-encoder in a dual-retriever RAG pipeline combining BM25 and dense vector search. Evaluated on a domain-specific question-answering benchmark using the RAGAS framework, our fine-tuned LLaMA 3 reranker achieves gains of 14% in answer relevancy, 16% in context precision, 19% in answer similarity, and 21% in answer correctness over the cross-encoder baseline, while reducing inference overhead through 4-bit quantization. These results demonstrate that instruction-tuned LLMs can be adapted into accurate, efficient rerankers without the quadratic complexity of traditional cross-encoders.