RAG retrieval benchmarking and optimization

A retrieval miss doesn't look like an error. When the right passage isn't retrieved, the LLM still answers from the wrong data. Nothing in your logs points at retrieval.

In this article I will show you how to use Truvec to debug and optimize your retrieval.

Everything before the LLM changes which chunks come back:

Your contracts, support articles and code have their own vocabulary. You can not rely on other people setup to build your own retrieval system.

Questions about your docs, each tied to the chunk that answers it.

UI: Data Upload → Golden Set → Create Dataset → Generate Dataset. Review the questions, fix the bad ones, then Commit Dataset. Committed means locked: every run uses the same questions.

API:

API=https://app.truvec.dev
AUTH="Authorization: Bearer $TOKEN"   # from POST /api/auth/login

curl -s -X POST "$API/api/v1/documents/upload?project_id=$PROJECT" -H "$AUTH" \
  -F "file=@help-center.pdf" -F "chunking_strategy=recursive" \
  -F "chunk_size=1000" -F "chunk_overlap=200"

DATASET=$(curl -s -X POST $API/api/v1/golden-datasets -H "$AUTH" \
  -H "Content-Type: application/json" \
  -d "{\"project_id\": $PROJECT, \"name\": \"v1\", \"number_of_questions\": 20}" | jq .id)

curl -s -X POST $API/api/v1/golden-datasets/$DATASET/commit -H "$AUTH"

UI: Comparison → Compare models → pick the dataset, tick the models, Run Comparison.

Look at that table. Same hit rate: 90% for both. MRR splits them: 0.900 vs 0.850. And the winner is the one priced at $0.020 per 1M tokens, not $0.100.

Hit rate says the right chunk is in the top k. MRR says how high. If you pass 3 chunks to the LLM, that's the one to watch.

curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \
  -H "Content-Type: application/json" \
  -d "{\"project_id\": $PROJECT, \"golden_dataset_id\": $DATASET,
       \"name\": \"small vs large\", \"model_ids\": [1, 2], \"top_k\": 5}"

Keep one baseline. Add hybrid search in one configuration, a reranker in another. Same questions.

UI: Comparison → Compare strategies. The Hybrid search and Reranker buttons copy the baseline and change one thing.

API: the first variant is the baseline.

curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \
  -H "Content-Type: application/json" -d "{
  \"project_id\": $PROJECT, \"golden_dataset_id\": $DATASET,
  \"name\": \"Does a reranker help?\", \"top_k\": 5,
  \"variants\": [
    {\"model_id\": 1},
    {\"model_id\": 1, \"pipeline\": {\"retrieval\": \"hybrid\"}},
    {\"model_id\": 1, \"pipeline\": {\"reranker\": {\"provider\": \"cohere\", \"model\": \"rerank-v3.5\"}}}
  ]
}"

Then read each one against the baseline:

curl -s $API/api/v1/benchmarks/$BENCHMARK -H "$AUTH" \
  | jq '.results.comparison[] | {config: .label, mrr,
        mrr_delta: .vs_baseline.mrr_delta,
        fixed: .vs_baseline.fixed, broken: .vs_baseline.broken}'

fixed: questions the baseline missed and this setup found.

broken: the opposite.

Fixing 4 and breaking 3 is not the same as fixing 1 and breaking none, even if the averages match.

It's all API calls, so run it when your documents change or before you switch models. Retrieval first, generation after.

Free plan, no card, no provider keys to set up. About 15 minutes to a first leaderboard on your own docs.

👉 https://truvec.dev

API docs: https://docs.truvec.dev