# Rebuilding the Cerebras Knowledge Base: Results Appendix (P1–P4)

> Source: <https://dev.to/faridgnank02/rebuilding-the-cerebras-knowledge-base-results-appendix-p1-p4-4km8>
> Published: 2026-08-16 12:23:40+00:00

This is the data appendix for **Posts 1–4**.

The narrative and takeaways live in the main posts. This page is pure measurement.

**Eval set:** 22 questions (P1) → expanded to 31 questions (P2 onward)

**Corpus evolution:**

| Metric | Vector P2 | Hybrid P2 | Vector P3 | Hybrid P3 | Hybrid + Rerank (P4) |
|---|---|---|---|---|---|
| recall@1 | 0.68 | 0.61 | 0.52 | 0.39 | 0.87 |
| recall@3 | 0.84 | 0.65 | 0.71 | 0.65 | 0.94 |
| recall@10 | 0.90 | 0.90 | 0.81 | 0.94 |
0.94 |
| MRR | 0.77 | 0.67 | 0.63 | 0.57 | 0.90 |

**Takeaway:**

Hybrid alone never beat pure vector on this corpus.

Hybrid + LLM rerank is the first clear win.

**Corpus:** 3,000 raw issue threads + 687 code chunks

**Embeddings:** BGE-M3 (1024d), max_seq_length=1024, HNSW cosine

| Metric | Score |
|---|---|
| recall@10 | 1.00 (22/22) |
| recall@3 | 0.95 |
| recall@1 | 0.77 (17/22) |

`TypeError: Object of type int64...`

, `AttributeError: 'Depends'...`

) — ranked 4–5 instead of 1`jsonable_encoder`

code chunk outranked by issues **Pattern:** Dense search is strong on recall@10 but weak when the query has a sharp lexical signal.

`max_seq_length=1024`

**Corpus:** Same size as P1, with better comment pagination and symbol-based code IDs

**Eval set:** Expanded to 31 questions (added exact error pastes + rare identifiers)

| Metric | Vector | FTS | Hybrid |
|---|---|---|---|
| recall@1 | 0.68 | 0.42 | 0.61 |
| recall@3 | 0.84 | 0.48 | 0.65 |
| recall@10 | 0.90 | 0.65 | 0.90 |
| MRR | 0.77 | 0.47 | 0.67 |

**Headline:** Hybrid is **not** a strict win over vector-only.

`TypeError: int64 is not JSON serializable`

) → moved from rank 5 → 1`jsonable_encoder`

, background tasks, APIKeyHeader) — FTS noise pushed the correct code chunk out of the top ranks**Pattern:** Hybrid helps when the lexical signal is sharp. It hurts when the signal is diffuse or the answer is a code chunk.

**Corpus rebuilt:**

| Metric | Vector P2 | Vector P3 | Hybrid P2 | Hybrid P3 |
|---|---|---|---|---|
| recall@1 | 0.68 | 0.52 |
0.61 | 0.39 |
| recall@3 | 0.84 | 0.71 |
0.65 | 0.65 |
| recall@10 | 0.90 | 0.81 | 0.90 | 0.94 |
| MRR | 0.77 | 0.63 |
0.67 | 0.57 |

**Headline:** Distillation + bursting raised the recall ceiling but made top-of-ranking worse.

The embedding is now computed on the clean LLM summary, not the raw thread.

Exact error strings and identifiers that used to live in the vector are gone → error-paste queries drop out of the top-10.

High-signal comments get their own embedding instead of being buried in long threads.

This is what pushed hybrid recall@10 from 0.90 → 0.94.

`jsonable_encoder`

)Both are natural-language → code-location lookups. Retrieval tuning alone cannot solve them.

**Corpus:** Same as P3 (16,315 docs)

**Mode:** hybrid + LLM reranker over the fused top-20

| Metric | Vector | Hybrid | Hybrid + Rerank |
|---|---|---|---|
| recall@1 | 0.52 | 0.39 |
0.87 (27/31) |
| recall@3 | 0.71 | 0.65 |
0.94 (29/31) |
| recall@10 | 0.81 | 0.94 |
0.94 (29/31) |
| MRR | 0.63 | 0.57 | 0.90 |

**Headline:** This is the first unqualified win in the series.

The same two code-location questions.

Reranking can only reorder what is already in the pool. These two never enter the top-20 → they require a different retriever (symbol / grep style), which is the subject of later posts.

One extra LLM call per query (≈20 candidates). Retrieval legs remain LLM-free.

| Stage | What improved | What got worse | Net effect |
|---|---|---|---|
| P1 Naive vector | Strong baseline recall@10 | Weak precision on error pastes | Good starting point |
| P2 Hybrid | Better on sharp lexical queries | Worse on code + paraphrase | Slight regression overall |
| P3 Distill + Burst | Higher recall ceiling (0.94
|

**Current default:** `hybrid --rerank`

*Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.*