{"slug": "rebuilding-the-cerebras-knowledge-base-results-appendix-p1-p4", "title": "Rebuilding the Cerebras Knowledge Base: Results Appendix (P1–P4)", "summary": "Cerebras Systems' knowledge base rebuild, documented across four posts, shows that hybrid retrieval alone never outperformed pure vector search on the company's support corpus, but adding an LLM reranker delivered the first unqualified win, lifting recall@1 from 0.52 to 0.87 and MRR from 0.63 to 0.90. The project, led by developer Farid Gnank, evaluated 31 questions against a corpus of 3,000 issue threads and 687 code chunks, using BGE-M3 embeddings and a hybrid retrieval pipeline with reranking.", "body_md": "This is the data appendix for **Posts 1–4**.\n\nThe narrative and takeaways live in the main posts. This page is pure measurement.\n\n**Eval set:** 22 questions (P1) → expanded to 31 questions (P2 onward)\n\n**Corpus evolution:**\n\n| Metric | Vector P2 | Hybrid P2 | Vector P3 | Hybrid P3 | Hybrid + Rerank (P4) |\n|---|---|---|---|---|---|\n| recall@1 | 0.68 | 0.61 | 0.52 | 0.39 | 0.87 |\n| recall@3 | 0.84 | 0.65 | 0.71 | 0.65 | 0.94 |\n| recall@10 | 0.90 | 0.90 | 0.81 | 0.94 |\n0.94 |\n| MRR | 0.77 | 0.67 | 0.63 | 0.57 | 0.90 |\n\n**Takeaway:**\n\nHybrid alone never beat pure vector on this corpus.\n\nHybrid + LLM rerank is the first clear win.\n\n**Corpus:** 3,000 raw issue threads + 687 code chunks\n\n**Embeddings:** BGE-M3 (1024d), max_seq_length=1024, HNSW cosine\n\n| Metric | Score |\n|---|---|\n| recall@10 | 1.00 (22/22) |\n| recall@3 | 0.95 |\n| recall@1 | 0.77 (17/22) |\n\n`TypeError: Object of type int64...`\n\n, `AttributeError: 'Depends'...`\n\n) — ranked 4–5 instead of 1`jsonable_encoder`\n\ncode chunk outranked by issues **Pattern:** Dense search is strong on recall@10 but weak when the query has a sharp lexical signal.\n\n`max_seq_length=1024`\n\n**Corpus:** Same size as P1, with better comment pagination and symbol-based code IDs\n\n**Eval set:** Expanded to 31 questions (added exact error pastes + rare identifiers)\n\n| Metric | Vector | FTS | Hybrid |\n|---|---|---|---|\n| recall@1 | 0.68 | 0.42 | 0.61 |\n| recall@3 | 0.84 | 0.48 | 0.65 |\n| recall@10 | 0.90 | 0.65 | 0.90 |\n| MRR | 0.77 | 0.47 | 0.67 |\n\n**Headline:** Hybrid is **not** a strict win over vector-only.\n\n`TypeError: int64 is not JSON serializable`\n\n) → moved from rank 5 → 1`jsonable_encoder`\n\n, background tasks, APIKeyHeader) — FTS noise pushed the correct code chunk out of the top ranks**Pattern:** Hybrid helps when the lexical signal is sharp. It hurts when the signal is diffuse or the answer is a code chunk.\n\n**Corpus rebuilt:**\n\n| Metric | Vector P2 | Vector P3 | Hybrid P2 | Hybrid P3 |\n|---|---|---|---|---|\n| recall@1 | 0.68 | 0.52 |\n0.61 | 0.39 |\n| recall@3 | 0.84 | 0.71 |\n0.65 | 0.65 |\n| recall@10 | 0.90 | 0.81 | 0.90 | 0.94 |\n| MRR | 0.77 | 0.63 |\n0.67 | 0.57 |\n\n**Headline:** Distillation + bursting raised the recall ceiling but made top-of-ranking worse.\n\nThe embedding is now computed on the clean LLM summary, not the raw thread.\n\nExact error strings and identifiers that used to live in the vector are gone → error-paste queries drop out of the top-10.\n\nHigh-signal comments get their own embedding instead of being buried in long threads.\n\nThis is what pushed hybrid recall@10 from 0.90 → 0.94.\n\n`jsonable_encoder`\n\n)Both are natural-language → code-location lookups. Retrieval tuning alone cannot solve them.\n\n**Corpus:** Same as P3 (16,315 docs)\n\n**Mode:** hybrid + LLM reranker over the fused top-20\n\n| Metric | Vector | Hybrid | Hybrid + Rerank |\n|---|---|---|---|\n| recall@1 | 0.52 | 0.39 |\n0.87 (27/31) |\n| recall@3 | 0.71 | 0.65 |\n0.94 (29/31) |\n| recall@10 | 0.81 | 0.94 |\n0.94 (29/31) |\n| MRR | 0.63 | 0.57 | 0.90 |\n\n**Headline:** This is the first unqualified win in the series.\n\nThe same two code-location questions.\n\nReranking can only reorder what is already in the pool. These two never enter the top-20 → they require a different retriever (symbol / grep style), which is the subject of later posts.\n\nOne extra LLM call per query (≈20 candidates). Retrieval legs remain LLM-free.\n\n| Stage | What improved | What got worse | Net effect |\n|---|---|---|---|\n| P1 Naive vector | Strong baseline recall@10 | Weak precision on error pastes | Good starting point |\n| P2 Hybrid | Better on sharp lexical queries | Worse on code + paraphrase | Slight regression overall |\n| P3 Distill + Burst | Higher recall ceiling (0.94\n|\n\n**Current default:** `hybrid --rerank`\n\n*Code for the series:\ngithub.com/faridgnank02/cerebras_knowledge_base.*", "url": "https://wpnews.pro/news/rebuilding-the-cerebras-knowledge-base-results-appendix-p1-p4", "canonical_source": "https://dev.to/faridgnank02/rebuilding-the-cerebras-knowledge-base-results-appendix-p1-p4-4km8", "published_at": "2026-08-16 12:23:40+00:00", "updated_at": "2026-08-16 12:42:25.503348+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "ai-infrastructure", "developer-tools"], "entities": ["Cerebras", "Farid Gnank", "BGE-M3", "HNSW"], "alternates": {"html": "https://wpnews.pro/news/rebuilding-the-cerebras-knowledge-base-results-appendix-p1-p4", "markdown": "https://wpnews.pro/news/rebuilding-the-cerebras-knowledge-base-results-appendix-p1-p4.md", "text": "https://wpnews.pro/news/rebuilding-the-cerebras-knowledge-base-results-appendix-p1-p4.txt", "jsonld": "https://wpnews.pro/news/rebuilding-the-cerebras-knowledge-base-results-appendix-p1-p4.jsonld"}}