# I Benchmarked 19 Retrieval Pipelines Head-to-Head and the Results Were Surprisingly Honest

> Source: <https://dev.to/subhansh/i-benchmarked-19-retrieval-pipelines-head-to-head-and-the-results-were-surprisingly-honest-3efe>
> Published: 2026-07-23 15:09:01+00:00

I got tired of retrieval papers claiming their pipeline is "state-of-the-art" without significance tests, without telling you what dataset they ran on, and without reproducible code. So I built [raven-retrieval](https://github.com/subhansh-dev/raven-retrieval) -- 19 retrieval pipelines, real BEIR datasets, Bonferroni-corrected bootstrap significance tests, all the numbers whether they look good or not. MIT licensed, open source.

The core question: does ColBERT-style late interaction scoring at every level of a RAPTOR hierarchical tree actually improve retrieval over simpler approaches? To answer that properly you need the same datasets, same chunking, same metrics, same significance tests for every pipeline. Otherwise you're comparing apples to oranges.

The framework implements 19 pipelines and benchmarks them all on BEIR datasets (SciFact, HotpotQA) on equal footing. No cherry-picking.

**Core:** Naive Dense RAG, Hybrid RAG (BM25+Dense+RRF), ColBERT Late Interaction, RAPTOR+Late Interaction (novel combination).

**Research (2024-2025):** HyDE, SPLADE, SPLADE+Dense, BM25+Rocchio PRF, Contextual Retrieval, Contextual Hybrid, Late Chunking, RAPTOR+Late Traversal, Agentic Multi-Hop, Reflection Retriever, Graph Retrieval, Two-Stage Dense+Reranker, Approximate Late Interaction, Contextual Dense, Contextual BM25.

**Composition modules:** Cross-Encoder Reranker, Two-Stage Retriever, Document Graph, Residual Compressor (~30x storage reduction).

The novel one is **RAPTOR + Late Interaction**: build a RAPTOR hierarchical summary tree, then apply ColBERT MaxSim scoring at every node (leaf chunks AND summary nodes). Nobody has published on this combination before.

ColBERT's scoring: for each query token, find the most similar document token (cosine), then sum those maximums across all query tokens. Every query token gets a vote for its best-matching document token. This captures fine-grained token-level relevance that single-vector cosine misses entirely. Implemented in pure numpy, no torch needed for scoring.

Run on Google Colab T4 GPU, seed=42, two datasets.

| Pipeline | nDCG@10 | Per-Query |
|---|---|---|
| HyDE | 0.712 | 19ms |
| Naive Dense | 0.696 | 32ms |
| Contextual Hybrid | 0.682 | 93ms |
| Hybrid RAG | 0.667 | 87ms |
| BM25+PRF | 0.529 | 43ms |

**Key finding:** Top 4 are **statistically indistinguishable** (all pairwise p > 0.05 after Bonferroni). BM25+PRF is significantly worse (p ~ 0.000). The "winner" HyDE is only 2 points above Dense -- within noise.

| Pipeline | nDCG@10 | Per-Query |
|---|---|---|
| Hybrid RAG | 0.925 | 22ms |
| Contextual Hybrid | 0.923 | 22ms |
| Naive Dense | 0.906 | 12ms |
| HyDE | 0.892 | 26ms |
| BM25+PRF | 0.869 | 8ms |

**The ranking flips completely.** HyDE drops from 1st to 4th. Hybrid RAG wins because BM25+Dense fusion catches different reasoning hops. A single hypothetical document can't bridge two separate reasoning steps.

**The most important finding:** No single pipeline dominates across task types. Single-hop favors semantic (HyDE). Multi-hop favors hybrid. Anyone claiming universal superiority hasn tested on enough datasets.

Before v0.3, 13 bugs (6 critical) silently corrupted results:

These don crash your program. They just make your results wrong in ways you can see from final numbers alone.

`ColbertContrastiveEncoder`

with InfoNCE) but no trained checkpoint in default benchmark yet.

```
git clone https://github.com/subhansh-dev/raven-retrieval.git
cd raven-retrieval
pip install -e ".[full]"
python run_enhanced_benchmark.py --dataset scifact --top-k 10 --seed 42
```

CLI entry points: `raven-benchmark`

, `raven-report`

. Low-RAM? Use `--max-docs`

. Want trained ColBERT? Use `--colbert-checkpoint`

. All 55 tests pass.

Retrieval benchmarks have a credibility problem. Papers claim improvements without significance tests, without multiple datasets, without reproducible code. Raven-retrieval fixes all of that. The honest answer -- that top pipelines are tied, that rankings flip between datasets, that BM25+PRF is bad on scientific text -- is more useful than cherry-picked claims of universal superiority.