I Benchmarked 19 Retrieval Pipelines Head-to-Head and the Results Were Surprisingly Honest

A developer built raven-retrieval, an open-source framework that benchmarks 19 retrieval pipelines on BEIR datasets with Bonferroni-corrected significance tests. The key finding is that top pipelines like HyDE and Naive Dense are statistically indistinguishable on single-hop tasks, while hybrid methods win on multi-hop datasets, disproving claims of universal superiority.

I got tired of retrieval papers claiming their pipeline is "state-of-the-art" without significance tests, without telling you what dataset they ran on, and without reproducible code. So I built raven-retrieval https://github.com/subhansh-dev/raven-retrieval -- 19 retrieval pipelines, real BEIR datasets, Bonferroni-corrected bootstrap significance tests, all the numbers whether they look good or not. MIT licensed, open source. The core question: does ColBERT-style late interaction scoring at every level of a RAPTOR hierarchical tree actually improve retrieval over simpler approaches? To answer that properly you need the same datasets, same chunking, same metrics, same significance tests for every pipeline. Otherwise you're comparing apples to oranges. The framework implements 19 pipelines and benchmarks them all on BEIR datasets SciFact, HotpotQA on equal footing. No cherry-picking. Core: Naive Dense RAG, Hybrid RAG BM25+Dense+RRF , ColBERT Late Interaction, RAPTOR+Late Interaction novel combination . Research 2024-2025 : HyDE, SPLADE, SPLADE+Dense, BM25+Rocchio PRF, Contextual Retrieval, Contextual Hybrid, Late Chunking, RAPTOR+Late Traversal, Agentic Multi-Hop, Reflection Retriever, Graph Retrieval, Two-Stage Dense+Reranker, Approximate Late Interaction, Contextual Dense, Contextual BM25. Composition modules: Cross-Encoder Reranker, Two-Stage Retriever, Document Graph, Residual Compressor ~30x storage reduction . The novel one is RAPTOR + Late Interaction : build a RAPTOR hierarchical summary tree, then apply ColBERT MaxSim scoring at every node leaf chunks AND summary nodes . Nobody has published on this combination before. ColBERT's scoring: for each query token, find the most similar document token cosine , then sum those maximums across all query tokens. Every query token gets a vote for its best-matching document token. This captures fine-grained token-level relevance that single-vector cosine misses entirely. Implemented in pure numpy, no torch needed for scoring. Run on Google Colab T4 GPU, seed=42, two datasets. | Pipeline | nDCG@10 | Per-Query | |---|---|---| | HyDE | 0.712 | 19ms | | Naive Dense | 0.696 | 32ms | | Contextual Hybrid | 0.682 | 93ms | | Hybrid RAG | 0.667 | 87ms | | BM25+PRF | 0.529 | 43ms | Key finding: Top 4 are statistically indistinguishable all pairwise p 0.05 after Bonferroni . BM25+PRF is significantly worse p ~ 0.000 . The "winner" HyDE is only 2 points above Dense -- within noise. | Pipeline | nDCG@10 | Per-Query | |---|---|---| | Hybrid RAG | 0.925 | 22ms | | Contextual Hybrid | 0.923 | 22ms | | Naive Dense | 0.906 | 12ms | | HyDE | 0.892 | 26ms | | BM25+PRF | 0.869 | 8ms | The ranking flips completely. HyDE drops from 1st to 4th. Hybrid RAG wins because BM25+Dense fusion catches different reasoning hops. A single hypothetical document can't bridge two separate reasoning steps. The most important finding: No single pipeline dominates across task types. Single-hop favors semantic HyDE . Multi-hop favors hybrid. Anyone claiming universal superiority hasn tested on enough datasets. Before v0.3, 13 bugs 6 critical silently corrupted results: These don crash your program. They just make your results wrong in ways you can see from final numbers alone. ColbertContrastiveEncoder with InfoNCE but no trained checkpoint in default benchmark yet. git clone https://github.com/subhansh-dev/raven-retrieval.git cd raven-retrieval pip install -e ". full " python run enhanced benchmark.py --dataset scifact --top-k 10 --seed 42 CLI entry points: raven-benchmark , raven-report . Low-RAM? Use --max-docs . Want trained ColBERT? Use --colbert-checkpoint . All 55 tests pass. Retrieval benchmarks have a credibility problem. Papers claim improvements without significance tests, without multiple datasets, without reproducible code. Raven-retrieval fixes all of that. The honest answer -- that top pipelines are tied, that rankings flip between datasets, that BM25+PRF is bad on scientific text -- is more useful than cherry-picked claims of universal superiority.