Rebuilding the Cerebras Knowledge Base: Results Appendix (P1–P4)

Cerebras Systems' knowledge base rebuild, documented across four posts, shows that hybrid retrieval alone never outperformed pure vector search on the company's support corpus, but adding an LLM reranker delivered the first unqualified win, lifting recall@1 from 0.52 to 0.87 and MRR from 0.63 to 0.90. The project, led by developer Farid Gnank, evaluated 31 questions against a corpus of 3,000 issue threads and 687 code chunks, using BGE-M3 embeddings and a hybrid retrieval pipeline with reranking.

This is the data appendix for Posts 1–4 . The narrative and takeaways live in the main posts. This page is pure measurement. Eval set: 22 questions P1 → expanded to 31 questions P2 onward Corpus evolution: | Metric | Vector P2 | Hybrid P2 | Vector P3 | Hybrid P3 | Hybrid + Rerank P4 | |---|---|---|---|---|---| | recall@1 | 0.68 | 0.61 | 0.52 | 0.39 | 0.87 | | recall@3 | 0.84 | 0.65 | 0.71 | 0.65 | 0.94 | | recall@10 | 0.90 | 0.90 | 0.81 | 0.94 | 0.94 | | MRR | 0.77 | 0.67 | 0.63 | 0.57 | 0.90 | Takeaway: Hybrid alone never beat pure vector on this corpus. Hybrid + LLM rerank is the first clear win. Corpus: 3,000 raw issue threads + 687 code chunks Embeddings: BGE-M3 1024d , max seq length=1024, HNSW cosine | Metric | Score | |---|---| | recall@10 | 1.00 22/22 | | recall@3 | 0.95 | | recall@1 | 0.77 17/22 | TypeError: Object of type int64... , AttributeError: 'Depends'... — ranked 4–5 instead of 1 jsonable encoder code chunk outranked by issues Pattern: Dense search is strong on recall@10 but weak when the query has a sharp lexical signal. max seq length=1024 Corpus: Same size as P1, with better comment pagination and symbol-based code IDs Eval set: Expanded to 31 questions added exact error pastes + rare identifiers | Metric | Vector | FTS | Hybrid | |---|---|---|---| | recall@1 | 0.68 | 0.42 | 0.61 | | recall@3 | 0.84 | 0.48 | 0.65 | | recall@10 | 0.90 | 0.65 | 0.90 | | MRR | 0.77 | 0.47 | 0.67 | Headline: Hybrid is not a strict win over vector-only. TypeError: int64 is not JSON serializable → moved from rank 5 → 1 jsonable encoder , background tasks, APIKeyHeader — FTS noise pushed the correct code chunk out of the top ranks Pattern: Hybrid helps when the lexical signal is sharp. It hurts when the signal is diffuse or the answer is a code chunk. Corpus rebuilt: | Metric | Vector P2 | Vector P3 | Hybrid P2 | Hybrid P3 | |---|---|---|---|---| | recall@1 | 0.68 | 0.52 | 0.61 | 0.39 | | recall@3 | 0.84 | 0.71 | 0.65 | 0.65 | | recall@10 | 0.90 | 0.81 | 0.90 | 0.94 | | MRR | 0.77 | 0.63 | 0.67 | 0.57 | Headline: Distillation + bursting raised the recall ceiling but made top-of-ranking worse. The embedding is now computed on the clean LLM summary, not the raw thread. Exact error strings and identifiers that used to live in the vector are gone → error-paste queries drop out of the top-10. High-signal comments get their own embedding instead of being buried in long threads. This is what pushed hybrid recall@10 from 0.90 → 0.94. jsonable encoder Both are natural-language → code-location lookups. Retrieval tuning alone cannot solve them. Corpus: Same as P3 16,315 docs Mode: hybrid + LLM reranker over the fused top-20 | Metric | Vector | Hybrid | Hybrid + Rerank | |---|---|---|---| | recall@1 | 0.52 | 0.39 | 0.87 27/31 | | recall@3 | 0.71 | 0.65 | 0.94 29/31 | | recall@10 | 0.81 | 0.94 | 0.94 29/31 | | MRR | 0.63 | 0.57 | 0.90 | Headline: This is the first unqualified win in the series. The same two code-location questions. Reranking can only reorder what is already in the pool. These two never enter the top-20 → they require a different retriever symbol / grep style , which is the subject of later posts. One extra LLM call per query ≈20 candidates . Retrieval legs remain LLM-free. | Stage | What improved | What got worse | Net effect | |---|---|---|---| | P1 Naive vector | Strong baseline recall@10 | Weak precision on error pastes | Good starting point | | P2 Hybrid | Better on sharp lexical queries | Worse on code + paraphrase | Slight regression overall | | P3 Distill + Burst | Higher recall ceiling 0.94 | Current default: hybrid --rerank Code for the series: github.com/faridgnank02/cerebras knowledge base.