This is the data appendix for Posts 1–4.
The narrative and takeaways live in the main posts. This page is pure measurement.
Eval set: 22 questions (P1) → expanded to 31 questions (P2 onward) Corpus evolution:
| Metric | Vector P2 | Hybrid P2 | Vector P3 | Hybrid P3 | Hybrid + Rerank (P4) |
|---|---|---|---|---|---|
| recall@1 | 0.68 | 0.61 | 0.52 | 0.39 | 0.87 |
| recall@3 | 0.84 | 0.65 | 0.71 | 0.65 | 0.94 |
| recall@10 | 0.90 | 0.90 | 0.81 | 0.94 | |
| 0.94 | |||||
| MRR | 0.77 | 0.67 | 0.63 | 0.57 | 0.90 |
Takeaway:
Hybrid alone never beat pure vector on this corpus.
Hybrid + LLM rerank is the first clear win.
Corpus: 3,000 raw issue threads + 687 code chunks
Embeddings: BGE-M3 (1024d), max_seq_length=1024, HNSW cosine | Metric | Score |
|---|---|
| recall@10 | 1.00 (22/22) |
| recall@3 | 0.95 |
| recall@1 | 0.77 (17/22) |
TypeError: Object of type int64...
, AttributeError: 'Depends'...
) — ranked 4–5 instead of 1jsonable_encoder
code chunk outranked by issues Pattern: Dense search is strong on recall@10 but weak when the query has a sharp lexical signal.
max_seq_length=1024
Corpus: Same size as P1, with better comment pagination and symbol-based code IDs
Eval set: Expanded to 31 questions (added exact error pastes + rare identifiers)
| Metric | Vector | FTS | Hybrid |
|---|---|---|---|
| recall@1 | 0.68 | 0.42 | 0.61 |
| recall@3 | 0.84 | 0.48 | 0.65 |
| recall@10 | 0.90 | 0.65 | 0.90 |
| MRR | 0.77 | 0.47 | 0.67 |
Headline: Hybrid is not a strict win over vector-only.
TypeError: int64 is not JSON serializable
) → moved from rank 5 → 1jsonable_encoder
, background tasks, APIKeyHeader) — FTS noise pushed the correct code chunk out of the top ranksPattern: Hybrid helps when the lexical signal is sharp. It hurts when the signal is diffuse or the answer is a code chunk.
Corpus rebuilt:
| Metric | Vector P2 | Vector P3 | Hybrid P2 | Hybrid P3 |
|---|---|---|---|---|
| recall@1 | 0.68 | 0.52 | ||
| 0.61 | 0.39 | |||
| recall@3 | 0.84 | 0.71 | ||
| 0.65 | 0.65 | |||
| recall@10 | 0.90 | 0.81 | 0.90 | 0.94 |
| MRR | 0.77 | 0.63 | ||
| 0.67 | 0.57 |
Headline: Distillation + bursting raised the recall ceiling but made top-of-ranking worse.
The embedding is now computed on the clean LLM summary, not the raw thread.
Exact error strings and identifiers that used to live in the vector are gone → error-paste queries drop out of the top-10.
High-signal comments get their own embedding instead of being buried in long threads.
This is what pushed hybrid recall@10 from 0.90 → 0.94.
jsonable_encoder
)Both are natural-language → code-location lookups. Retrieval tuning alone cannot solve them.
Corpus: Same as P3 (16,315 docs) Mode: hybrid + LLM reranker over the fused top-20
| Metric | Vector | Hybrid | Hybrid + Rerank |
|---|---|---|---|
| recall@1 | 0.52 | 0.39 | |
| 0.87 (27/31) | |||
| recall@3 | 0.71 | 0.65 | |
| 0.94 (29/31) | |||
| recall@10 | 0.81 | 0.94 | |
| 0.94 (29/31) | |||
| MRR | 0.63 | 0.57 | 0.90 |
Headline: This is the first unqualified win in the series.
The same two code-location questions.
Reranking can only reorder what is already in the pool. These two never enter the top-20 → they require a different retriever (symbol / grep style), which is the subject of later posts.
One extra LLM call per query (≈20 candidates). Retrieval legs remain LLM-free.
| Stage | What improved | What got worse | Net effect |
|---|---|---|---|
| P1 Naive vector | Strong baseline recall@10 | Weak precision on error pastes | Good starting point |
| P2 Hybrid | Better on sharp lexical queries | Worse on code + paraphrase | Slight regression overall |
| P3 Distill + Burst | Higher recall ceiling (0.94 | ||
Current default: hybrid --rerank
Code for the series:
github.com/faridgnank02/cerebras_knowledge_base.