{"slug": "fineweb-10b-slice-benchmark-qdrant-vs-milvus-vs-elasticsearch-all-scored-against", "title": "FineWeb-10B slice benchmark — Qdrant vs Milvus vs Elasticsearch, all scored against one shared exact ground truth and swept across each engine's search-effort knob for a recall/latency curve", "summary": "A developer published a shell-script benchmark harness that compares Qdrant, Milvus and Elasticsearch on a FineWeb-10B slice, scoring all three engines against a single shared exact brute-force ground truth and sweeping each engine's search-effort knob (hnsw_ef, ef, num_candidates) to produce recall/latency curves rather than single-point results. The script is environment-configured, writes only to a ROOT directory, and was verified end to end on the Qdrant path, with the Milvus and Elasticsearch sweep schemas flagged as less tested and a fast-failing probe subcommand provided to name offending config fields.", "body_md": "|  | #!/usr/bin/env bash | \n|  | # ============================================================================= | \n|  | # FineWeb-10B slice benchmark — Qdrant vs Milvus vs Elasticsearch | \n|  | # | \n|  | # One exact ground truth (brute force over YOUR slice), reused by every engine, | \n|  | # so all three are scored against identical correct answers. Each engine is then | \n|  | # swept across its search-effort knob to produce a recall/latency CURVE rather | \n|  | # than a single point. | \n|  | # | \n|  | # ./bench.sh all # everything, Qdrant only | \n|  | # ENGINES=\"qdrant milvus elastic\" ./bench.sh all | \n|  | # ./bench.sh probe milvus # validate one config, fast | \n|  | # ./bench.sh report | \n|  | # | \n|  | # REQUIRES, on PATH | \n|  | # nova with `bf` and `sweep` subcommands | \n|  | # nova-load built with --features elastic,milvus for the non-Qdrant paths | \n|  | # docker with a compose file in the working directory (see SERVICES) | \n|  | # hf HuggingFace CLI, for the dataset | \n|  | # python with pyarrow and pandas | \n|  | # | \n|  | # CONFIGURE entirely through the environment — nothing here is tied to a | \n|  | # particular machine, and ROOT is the only directory the script writes to: | \n|  | # | \n|  | # ROOT where everything lands (default $HOME/qfw-bench) | \n|  | # ENGINES which to run (default \"qdrant\") | \n|  | # SHARDS dataset shards to download (default 1) | \n|  | # QUERY_LIMIT / TOP_K / GT_K / DURATION_S / CONCURRENCY | \n|  | # QDRANT_URL / MILVUS_URL / ELASTIC_URL point at your own endpoints | \n|  | # NO_DOCKER=1 use already-running services and skip compose entirely | \n|  | # FORCE=1 redo a step that would otherwise be skipped as cached | \n|  | # | \n|  | # Verified end to end on the Qdrant path. The Milvus and Elasticsearch sweep | \n|  | # schemas are less travelled — run `./bench.sh probe <engine>` first; it fails | \n|  | # fast and names the offending field. | \n|  | # ============================================================================= | \n|  | set -euo pipefail | \n|  | ROOT=\"${ROOT:-$HOME/qfw-bench}\" | \n|  | ENGINES=\"${ENGINES:-qdrant}\" | \n|  | SHARDS=\"${SHARDS:-1}\" | \n|  | QUERY_LIMIT=\"${QUERY_LIMIT:-1000}\" | \n|  | TOP_K=\"${TOP_K:-10}\" | \n|  | GT_K=\"${GT_K:-100}\" | \n|  | DURATION_S=\"${DURATION_S:-60}\" | \n|  | CONCURRENCY=\"${CONCURRENCY:-8}\" | \n|  | SLICE=\"$ROOT/slice\" | \n|  | QUERIES=\"$ROOT/qfw-queries\" | \n|  | REGEN=\"$ROOT/regenerated/dense_regenerated.parquet\" | \n|  | BFOUT=\"$ROOT/bf-out\" | \n|  | BFFILE=\"$BFOUT/bf_dense_regenerated_dense_k${GT_K}.parquet\" | \n|  | SWEEPOUT=\"$ROOT/sweep-out\" | \n|  | export QDRANT_URL=\"${QDRANT_URL:-http://localhost:6334}\" | \n|  | export MILVUS_URL=\"${MILVUS_URL:-http://localhost:19530}\" | \n|  | export ELASTIC_URL=\"${ELASTIC_URL:-http://localhost:9200}\" | \n|  | mkdir -p \"$ROOT\" \"$BFOUT\" \"$SWEEPOUT\" \"$(dirname \"$REGEN\")\" | \n|  | # cargo-installed binaries are not always on PATH in a non-login shell | \n|  | export PATH=\"${CARGO_HOME:-$HOME/.cargo}/bin:$PATH\" | \n|  | # ============================================================================= | \n|  | # PER-BACKEND SCHEMA | \n|  | # | \n|  | # Every engine spells its search-effort knob differently, and the sweep config | \n|  | # rejects unknown keys outright — so a wrong guess fails at config load with the | \n|  | # offending field named rather than silently producing a wrong number. If your | \n|  | # toolchain disagrees with the spellings below, this is the block to fix. | \n|  | # ============================================================================= | \n|  | # Search-effort knob per engine: Qdrant HNSW takes hnsw_ef, Milvus HNSW takes | \n|  | # ef (IVF takes nprobe instead), Elasticsearch kNN takes num_candidates. | \n|  | effort_key() { | \n|  | case \"$1\" in | \n|  | qdrant) echo \"hnsw_ef\" ;; | \n|  | milvus) echo \"ef\" ;; | \n|  | elastic) echo \"num_candidates\" ;; | \n|  | esac | \n|  | } | \n|  | effort_values() { | \n|  | case \"$1\" in | \n|  | elastic) echo \"[16, 32, 64, 128, 256]\" ;; # must be >= top_k | \n|  | *) echo \"[16, 32, 64, 128, 256]\" ;; | \n|  | esac | \n|  | } | \n|  | # Index-build knobs, from nova_sweep/backends/*.py. Qdrant nests them under a | \n|  | # `params:` sub-block, so it takes dotted paths. Milvus and Elastic vectorstore | \n|  | # blocks are FLAT — variant keys land directly on the block, no dots. Elastic | \n|  | # holds HNSW mapping params in `index_options`; Milvus in `index_type` + | \n|  | # `index_params`. | \n|  | index_block() { | \n|  | case \"$1\" in | \n|  | qdrant) printf ' hnsw.m: [16]\\n hnsw.ef_construct: [100]\\n' ;; | \n|  | elastic) printf ' index_options: [{type: hnsw, m: 16, ef_construction: 100}]\\n' ;; | \n|  | milvus) printf ' index_type: [HNSW]\\n index_params: [{M: 16, efConstruction: 100}]\\n' ;; | \n|  | esac | \n|  | } | \n|  | target_block() { | \n|  | case \"$1\" in | \n|  | qdrant) printf ' type: qdrant\\n url: ${QDRANT_URL}\\n recreate: always\\n' ;; | \n|  | milvus) printf ' type: milvus\\n url: ${MILVUS_URL}\\n recreate: always\\n' ;; | \n|  | elastic) printf ' type: elastic\\n url: ${ELASTIC_URL}\\n tls_insecure: true\\n recreate: always\\n' ;; | \n|  | esac | \n|  | } | \n|  | # ============================================================================= | \n|  | # ============================================================================= | \n|  | # Fail before spending five minutes getting to a failure. | \n|  | preflight() { | \n|  | local missing=0 | \n|  | for c in hf docker curl python nova; do | \n|  | command -v \"$c\" >/dev/null \\|\\| { echo \"MISSING: $c\"; missing=1; } | \n|  | done | \n|  | nova --help 2>/dev/null \\| grep -q '^ bf ' \\ | \n|  | \\|\\| { echo \"MISSING: the 'bf' subcommand of nova\"; missing=1; } | \n|  | for e in $ENGINES; do | \n|  | case \"$e\" in qdrant) continue ;; esac | \n|  | nova-load --help 2>&1 \\| grep -qi \"$e\" \\ | \n|  | \\|\\| echo \"WARNING: nova-load may lack $e support (rebuild with --features elastic,milvus)\" | \n|  | done | \n|  | python -c \"import pyarrow, pandas\" 2>/dev/null \\ | \n|  | \\|\\| { echo \"MISSING: pyarrow/pandas (pip install pyarrow pandas)\"; missing=1; } | \n|  | # SERVICES: compose must define qdrant / etcd minio milvus / elastic, matching | \n|  | # engine_services() below. NO_DOCKER=1 if you run them some other way. | \n|  | if [ -z \"${NO_DOCKER:-}\" ] && ! docker compose config --services >/dev/null 2>&1; then | \n|  | echo \"MISSING: a docker compose file here defining the engine services\" | \n|  | echo \" (or set NO_DOCKER=1 to use services you start yourself)\" | \n|  | missing=1 | \n|  | fi | \n|  | [ \"$missing\" -eq 0 ] \\|\\| { echo \"preflight failed\"; exit 1; } | \n|  | echo \"preflight OK — engines: $ENGINES\" | \n|  | } | \n|  | # Only the Python side. The nova toolchain is a prerequisite — install it | \n|  | # however you normally do and make sure it is on PATH before running this. | \n|  | setup() { | \n|  | pip install \"transformers<5\" sentence-transformers requests huggingface_hub \\ | \n|  | pyarrow pandas | \n|  | nova --help | \n|  | } | \n|  | fetch() { | \n|  | # FORCE=1 to re-download regardless. | \n|  | if [ -z \"${FORCE:-}\" ] && [ -d \"$SLICE/data\" ] && \\ | \n|  | [ \"$(find \"$SLICE/data\" -name '*.parquet' \\| wc -l)\" -ge \"$SHARDS\" ]; then | \n|  | echo \"skip fetch: $SHARDS shard(s) already present (FORCE=1 to redo)\" | \n|  | return 0 | \n|  | fi | \n|  | hf download Qdrant/FineWeb-10B --repo-type dataset \\ | \n|  | --include \"queries/gt_dense_k1000.parquet\" \"queries/scripts/*\" \"queries/_checksums.json\" \\ | \n|  | --local-dir \"$QUERIES\" | \n|  | python - \"$SHARDS\" \"$SLICE\" <<'PY' | \n|  | import subprocess, sys | \n|  | from huggingface_hub import HfApi | \n|  | n, dest = int(sys.argv[1]), sys.argv[2] | \n|  | files = [f for f in HfApi().list_repo_files('Qdrant/FineWeb-10B', repo_type='dataset') | \n|  | if f.startswith('data/')][:n] | \n|  | print(f\"fetching {len(files)} shard(s):\", *files, sep=\"\\n \") | \n|  | subprocess.run([\"hf\",\"download\",\"Qdrant/FineWeb-10B\",\"--repo-type\",\"dataset\", | \n|  | \"--local-dir\", dest, \"--include\", *files], check=True) | \n|  | PY | \n|  | } | \n|  | queries() { | \n|  | if [ -z \"${FORCE:-}\" ] && [ -f \"$REGEN\" ]; then | \n|  | echo \"skip queries: $REGEN exists (FORCE=1 to redo)\" | \n|  | return 0 | \n|  | fi | \n|  | cd \"$QUERIES/queries\" | \n|  | python scripts/regenerate_queries.py \\ | \n|  | --in gt_dense_k1000.parquet --out \"$REGEN\" \\ | \n|  | --vectors dense --limit \"$QUERY_LIMIT\" --device cpu | \n|  | # MS MARCO queries are licensed for non-commercial research use only. | \n|  | } | \n|  | groundtruth() { | \n|  | if [ -z \"${FORCE:-}\" ] && [ -f \"$BFFILE\" ]; then | \n|  | echo \"skip groundtruth: $BFFILE exists (FORCE=1 to redo)\" | \n|  | return 0 | \n|  | fi | \n|  | cat > \"$ROOT/bf.yaml\" <<YAML | \n|  | corpus: | \n|  | path: $SLICE/data | \n|  | dense_column: dense_embedding | \n|  | queries: | \n|  | path: $REGEN | \n|  | dense_column: dense_embedding | \n|  | id_column: msmarco_query_id | \n|  | payload_fields: | \n|  | - query | \n|  | - dense_embedding | \n|  | output: | \n|  | path: $BFOUT | \n|  | params: | \n|  | io_workers: 4 | \n|  | dense_batch_size: 4096 | \n|  | searches: | \n|  | - name: dense | \n|  | vector_type: dense | \n|  | metric: cosine | \n|  | k: $GT_K | \n|  | YAML | \n|  | nova bf compute \"$ROOT/bf.yaml\" | \n|  | python - \"$BFFILE\" <<'PY' | \n|  | import sys, pyarrow.parquet as pq | \n|  | t = pq.read_table(sys.argv[1]); s = t.column(\"hit_scores\")[0].as_py() | \n|  | assert \"dense_embedding\" in t.schema.names, \"query vectors missing from bf output\" | \n|  | print(f\"ground truth OK: {t.num_rows} queries, top {s[0]:.4f}, k-th {s[-1]:.4f}\") | \n|  | PY | \n|  | } | \n|  | write_config() { | \n|  | local e=\"$1\" | \n|  | cat > \"$ROOT/sweep-$e.yaml\" <<YAML | \n|  | collection_name: qfineweb_$e | \n|  | corpus: | \n|  | path: $SLICE/data | \n|  | dense_column: dense_embedding | \n|  | queries: | \n|  | uri: $BFFILE | \n|  | column: dense_embedding | \n|  | ground_truth_column: hit_ids | \n|  | limit: $QUERY_LIMIT | \n|  | target: | \n|  | $(target_block \"$e\") | \n|  | # The ground truth was computed with metric: cosine. If an engine's collection | \n|  | # is built with a different distance, recall drops for a reason that has | \n|  | # nothing to do with the engine — and it fails as a plausible number rather | \n|  | # than an error, so pin it explicitly rather than trusting per-backend defaults. | \n|  | data_layouts: | \n|  | vectors.dense.distance: [cosine] | \n|  | index_variants: | \n|  | $(index_block \"$e\") | \n|  | searches: | \n|  | top_k: [$TOP_K] | \n|  | $(effort_key \"$e\"): $(effort_values \"$e\") | \n|  | batch_size: [1] | \n|  | duration_s: [$DURATION_S] | \n|  | concurrency: [$CONCURRENCY] | \n|  | output: | \n|  | path: $SWEEPOUT/$e | \n|  | YAML | \n|  | echo \"wrote $ROOT/sweep-$e.yaml\" | \n|  | } | \n|  | # Validate a config in seconds instead of paying for a multi-minute ingest to | \n|  | # find out it was wrong. There is no --dry-run, so call the config loader | \n|  | # directly and let it name the offending field. | \n|  | # NOTE: this validates the SWEEP config only. The per-backend configs it | \n|  | # generates are checked further downstream, so a clean probe does not guarantee | \n|  | # a clean run. | \n|  | probe() { | \n|  | local e=\"${1:?usage: probe <engine>}\" | \n|  | write_config \"$e\" | \n|  | python - \"$ROOT/sweep-$e.yaml\" <<'PY' | \n|  | import sys | \n|  | from nova_sweep.config import load_config | \n|  | try: | \n|  | load_config(sys.argv[1]) | \n|  | except Exception as exc: | \n|  | print(f\"INVALID: {type(exc).__name__}\") | \n|  | print(exc) | \n|  | sys.exit(1) | \n|  | print(f\"config OK: {sys.argv[1]}\") | \n|  | PY | \n|  | } | \n|  | # Block until the engine answers, instead of a fixed sleep. Milvus in | \n|  | # particular takes well over a minute to become ready, and a broken pipe | \n|  | # mid-ingest is what an unready (or OOM-killed) server looks like. | \n|  | wait_ready() { | \n|  | local e=\"$1\" url deadline=$((SECONDS + 180)) | \n|  | case \"$e\" in | \n|  | qdrant) url=\"http://localhost:6333/readyz\" ;; | \n|  | milvus) url=\"http://localhost:9091/healthz\" ;; | \n|  | elastic) url=\"$ELASTIC_URL/_cluster/health\" ;; | \n|  | esac | \n|  | echo \"waiting for $e ...\" | \n|  | until curl -sf \"$url\" >/dev/null 2>&1; do | \n|  | [ $SECONDS -lt $deadline ] \\|\\| { echo \"ERROR: $e not ready after 180s\"; return 1; } | \n|  | sleep 3 | \n|  | done | \n|  | echo \"$e ready\" | \n|  | } | \n|  | run_engine() { | \n|  | local e=\"$1\" | \n|  | echo \"=== $e ===\" | \n|  | wait_ready \"$e\" \\|\\| { echo \"SKIPPING $e\"; return 0; } | \n|  | write_config \"$e\" | \n|  | # A sweep that fails still writes rows with ok=false; don't abort the others. | \n|  | nova sweep \"$ROOT/sweep-$e.yaml\" \\|\\| echo \"WARNING: $e sweep reported errors\" | \n|  | } | \n|  | compose_up() { docker compose up -d \"$@\" >/dev/null 2>&1 \\|\\| true; } | \n|  | compose_stop() { docker compose stop \"$@\" >/dev/null 2>&1 \\|\\| true; } | \n|  | engine_services() { | \n|  | case \"$1\" in | \n|  | qdrant) echo \"qdrant\" ;; | \n|  | milvus) echo \"etcd minio milvus\" ;; | \n|  | elastic) echo \"elastic\" ;; | \n|  | esac | \n|  | } | \n|  | engines() { | \n|  | for e in $ENGINES; do | \n|  | # One engine up at a time: concurrent engines contend for the same cores | \n|  | # and page cache, which makes every latency number incomparable. | \n|  | local svc; svc=$(engine_services \"$e\") | \n|  | [ -n \"${NO_DOCKER:-}\" ] \\|\\| compose_up $svc | \n|  | run_engine \"$e\" | \n|  | [ -n \"${NO_DOCKER:-}\" ] \\|\\| compose_stop $svc | \n|  | sleep 5 | \n|  | done | \n|  | } | \n|  | # Results without provenance can't be defended later. nova bf writes its own | \n|  | # manifest; the sweep output has none, so record the rest here. | \n|  | provenance() { | \n|  | { | \n|  | echo \"date: $(date -u +%FT%TZ)\" | \n|  | echo \"host: $(uname -mrs)\" | \n|  | echo \"engines: $ENGINES\" | \n|  | echo \"shards: $SHARDS query_limit: $QUERY_LIMIT top_k: $TOP_K gt_k: $GT_K\" | \n|  | echo \"duration_s: $DURATION_S concurrency: $CONCURRENCY\" | \n|  | echo \"nova: $(nova --version 2>/dev/null \\|\\| echo unknown)\" | \n|  | echo \"nova-load: $(nova-load --version 2>/dev/null \\|\\| echo unknown)\" | \n|  | for e in $ENGINES; do | \n|  | for s in $(engine_services \"$e\"); do | \n|  | echo \"image[$s]: $(docker compose images -q \"$s\" 2>/dev/null \\| head -1)\" | \n|  | done | \n|  | done | \n|  | } \\| tee \"$ROOT/provenance.txt\" | \n|  | } | \n|  | report() { | \n|  | python - \"$SWEEPOUT\" \"$ROOT/results.csv\" <<'PY' | \n|  | import glob, os, sys, pandas as pd, pyarrow.parquet as pq | \n|  | sweepout, outcsv = sys.argv[1], sys.argv[2] | \n|  | rows = [] | \n|  | for f in glob.glob(os.path.join(sweepout, \"*\", \"sweep_results.parquet\")): | \n|  | df = pq.read_table(f).to_pandas() | \n|  | df[\"engine\"] = os.path.basename(os.path.dirname(f)) | \n|  | rows.append(df) | \n|  | if not rows: | \n|  | raise SystemExit(\"no sweep results yet\") | \n|  | df = pd.concat(rows, ignore_index=True) | \n|  | # each backend names its effort knob differently; coalesce into one column | \n|  | keys = [c for c in (\"search.hnsw_ef\", \"search.ef\", \"search.num_candidates\") | \n|  | if c in df.columns] | \n|  | df[\"effort\"] = df[keys].bfill(axis=1).iloc[:, 0] if keys else None | \n|  | # A failed sweep still writes one row per search point with ok=false and every | \n|  | # metric null. Reporting those as NaN alongside real numbers invites reading a | \n|  | # blank as a result, so split them out and say what actually broke. | \n|  | if \"ok\" in df.columns: | \n|  | failed, df = df[~df[\"ok\"].fillna(False)], df[df[\"ok\"].fillna(False)] | \n|  | for eng, grp in failed.groupby(\"engine\"): | \n|  | msg = next((m for m in grp.get(\"error\", []) if isinstance(m, str)), \"unknown error\") | \n|  | print(f\"FAILED: {eng} — {len(grp)} point(s) did not run\") | \n|  | print(f\" {msg.strip().splitlines()[0][:200]}\\n\") | \n|  | if df.empty: | \n|  | raise SystemExit(\"every engine failed — nothing to report\") | \n|  | cols = [\"engine\", \"effort\", \"full_recall.mean\", \"p50_ms\", \"p95_ms\", \"p99_ms\", | \n|  | \"qps\", \"missing_from_gt\", \"reindex_seconds\"] | \n|  | out = df[[c for c in cols if c in df.columns]].sort_values([\"engine\", \"effort\"]) | \n|  | print(out.to_string(index=False)) | \n|  | out.to_csv(outcsv, index=False) | \n|  | print(f\"\\nwrote {outcsv}\") | \n|  | if \"missing_from_gt\" in df and (df[\"missing_from_gt\"] > 0).any(): | \n|  | print(\"\\nWARNING: missing_from_gt > 0 — ground truth and collection disagree.\") | \n|  | print(\"Recall numbers from those rows are not trustworthy.\") | \n|  | if df[\"engine\"].nunique() > 1: | \n|  | print(\"\\nCompare engines at EQUAL RECALL, not at equal effort — the knobs\") | \n|  | print(\"are not on the same scale. Milvus cosine goes over REST rather than\") | \n|  | print(\"its native SDK path, so its latency is not comparable to Qdrant's gRPC.\") | \n|  | PY | \n|  | } | \n|  | all() { preflight; fetch; queries; groundtruth; engines; provenance; report; } | \n|  | \"${@:-all}\" |", "url": "https://wpnews.pro/news/fineweb-10b-slice-benchmark-qdrant-vs-milvus-vs-elasticsearch-all-scored-against", "canonical_source": "https://gist.github.com/andrisgauracs/f44d5dd040844f4e83ccae60f0d8ef5e", "published_at": "2026-09-24 09:29:43+00:00", "updated_at": "2026-09-24 18:02:37.762404+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools"], "entities": ["Qdrant", "Milvus", "Elasticsearch", "FineWeb", "HuggingFace"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/fineweb-10b-slice-benchmark-qdrant-vs-milvus-vs-elasticsearch-all-scored-against", "markdown": "https://wpnews.pro/news/fineweb-10b-slice-benchmark-qdrant-vs-milvus-vs-elasticsearch-all-scored-against.md", "text": "https://wpnews.pro/news/fineweb-10b-slice-benchmark-qdrant-vs-milvus-vs-elasticsearch-all-scored-against.txt", "jsonld": "https://wpnews.pro/news/fineweb-10b-slice-benchmark-qdrant-vs-milvus-vs-elasticsearch-all-scored-against.jsonld"}}