Open-source vector search and database supplier Qdrant has produced a 10 billion record dataset for vector retrieval benchmarking, because existing datasets are too small.
Qdrant-FineWeb-10B is a public dataset for vector retrieval benchmarking built on 10 billion documents and 120,000 ground truth queries, together with the Supernova toolkit that computed the dataset’s ground truth. Qdrant says vector search evaluation and testing requires that the vectors should come from real data. The dataset should be large enough that size-dependent behavior becomes visible, and correct query answers have to be known in advance. The list of right answers is the ground truth, and computing it exactly means comparing every query against every vector in the dataset, with no index and no approximation.
"Accurate benchmarking is one of the hardest things we get asked about, both in our open-source community and in customer discussions," said André Zayarni, CEO and Co-Founder of Qdrant. "Teams need real data at the size they actually run, and they need to know what the correct answers are. Assembling both yourself is expensive. We did it once and made it public.”
Qdrant says “FineWeb-10B is the largest open-source vector database benchmark that we know of. It holds 24.47 TB of vectors and 28.66 TB of source text and metadata. That second number is the important one. It is larger than the vectors, and it is the part that most large benchmarks discard.”
Qdrant-FineWeb-10B's ground truth covers 100,000 queries drawn from MS MARCO, a public set of real search queries, plus 10,000 sparse and 10,000 filtered queries. Running 120,000 queries against 10 billion documents is over a quadrillion distance computations.
There are two other datasets. PubMed-BGE-M3 covers scientific literature with three representations of the same corpus. As the dense, sparse, and multi-vector data all come from one model over one text set, you can compare hybrid retrieval methods with the other variables held constant. Coyo-VE places text and images in one shared representation, with 5.81 TB of payload against only 117 GB of vectors.
Qdrant says that, with Qdrant-FineWeb-10B, you can measure recall, the share of the true nearest results a system actually returned. This is so for any number (k) of nearest neighbors you ask the search to return, up to 1,000 on public data. This works on public data at 10 billion documents, across dense vectors, which encode meaning, sparse vectors, which encode terms, and metadata filters.
The source dataset or corpus is FineWeb, published by Hugging Face and derived from Common Crawl. Qdrant generated the embeddings on infrastructure provided by Vultr, running every embedding through gte-multilingual-base, an open embedding model from Alibaba, to produce one dense and one sparse vector.
Qdrant then computed the ground truth queries with Supernova, using SkyPilot to distribute the exhaustive search across a fleet by rank. Qdrant-FineWeb-10B ships embeddings, payload fields, and ground truth, not the source text.
Qdrant-FineWeb-10B is publicly available on Hugging Face and Supernova on Github. Supernova generates embeddings, computes exact ground truth, and loads and queries a cluster, so you can build the same kind of benchmark on your own corpus. Qdrant says that, as the ground truth runs to depth 1,000, you can measure recall@k for any k up to 1,000 by truncating the list, with no recomputation.
A coming Qdrant blog; Internet-scale Vector Search: Benchmarking Qdrant at 10B Vectors, provides a lot of background information. Bootnote
Qdrant has previously published vector search benchmark results using a dbpedia-openai-1M-1536-angular dataset with 1 million vectors. It also uses a deep-image-96-angular with 10million vectors. The latest dataset is 1,000 times larger.