RAG retrieval benchmarking and optimization

A developer demonstrated a retrieval-augmented generation (RAG) benchmarking workflow using Truvec, a tool for debugging and optimizing retrieval pipelines. The approach builds a locked golden question set from a user's own documents, then compares embedding models and retrieval strategies—hybrid search and a Cohere reranker—using MRR alongside hit rate, plus per-question fixed/broken counts against a baseline. In the example comparison, two models tied at 90% hit rate but split on MRR (0.900 vs 0.850), with the cheaper model winning.

A retrieval miss doesn't look like an error. When the right passage isn't retrieved, the LLM still answers from the wrong data. Nothing in your logs points at retrieval. In this article I will show you how to use Truvec to debug and optimize your retrieval. Everything before the LLM changes which chunks come back: Your contracts, support articles and code have their own vocabulary. You can not rely on other people setup to build your own retrieval system. Questions about your docs, each tied to the chunk that answers it. UI: Data Upload → Golden Set → Create Dataset → Generate Dataset . Review the questions, fix the bad ones, then Commit Dataset . Committed means locked: every run uses the same questions. API: API=https://app.truvec.dev AUTH="Authorization: Bearer $TOKEN" from POST /api/auth/login curl -s -X POST "$API/api/v1/documents/upload?project id=$PROJECT" -H "$AUTH" \ -F "file=@help-center.pdf" -F "chunking strategy=recursive" \ -F "chunk size=1000" -F "chunk overlap=200" DATASET=$ curl -s -X POST $API/api/v1/golden-datasets -H "$AUTH" \ -H "Content-Type: application/json" \ -d "{\"project id\": $PROJECT, \"name\": \"v1\", \"number of questions\": 20}" | jq .id curl -s -X POST $API/api/v1/golden-datasets/$DATASET/commit -H "$AUTH" UI: Comparison → Compare models → pick the dataset, tick the models, Run Comparison . Look at that table. Same hit rate: 90% for both. MRR splits them: 0.900 vs 0.850. And the winner is the one priced at $0.020 per 1M tokens, not $0.100. Hit rate says the right chunk is in the top k. MRR says how high. If you pass 3 chunks to the LLM, that's the one to watch. curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \ -H "Content-Type: application/json" \ -d "{\"project id\": $PROJECT, \"golden dataset id\": $DATASET, \"name\": \"small vs large\", \"model ids\": 1, 2 , \"top k\": 5}" Keep one baseline. Add hybrid search in one configuration, a reranker in another. Same questions. UI: Comparison → Compare strategies . The Hybrid search and Reranker buttons copy the baseline and change one thing. API: the first variant is the baseline. curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \ -H "Content-Type: application/json" -d "{ \"project id\": $PROJECT, \"golden dataset id\": $DATASET, \"name\": \"Does a reranker help?\", \"top k\": 5, \"variants\": {\"model id\": 1}, {\"model id\": 1, \"pipeline\": {\"retrieval\": \"hybrid\"}}, {\"model id\": 1, \"pipeline\": {\"reranker\": {\"provider\": \"cohere\", \"model\": \"rerank-v3.5\"}}} }" Then read each one against the baseline: curl -s $API/api/v1/benchmarks/$BENCHMARK -H "$AUTH" \ | jq '.results.comparison | {config: .label, mrr, mrr delta: .vs baseline.mrr delta, fixed: .vs baseline.fixed, broken: .vs baseline.broken}' fixed : questions the baseline missed and this setup found. broken : the opposite. Fixing 4 and breaking 3 is not the same as fixing 1 and breaking none, even if the averages match. It's all API calls, so run it when your documents change or before you switch models. Retrieval first, generation after. Free plan, no card, no provider keys to set up. About 15 minutes to a first leaderboard on your own docs. 👉 https://truvec.dev https://truvec.dev API docs: https://docs.truvec.dev https://docs.truvec.dev