# Python Retrieval Architecture for Private Knowledge Managers: Relevance Tuning in 4 Stages

> Source: <https://dev.to/ignazcole6453/python-retrieval-architecture-for-private-knowledge-managers-relevance-tuning-in-4-stages-13ep>
> Published: 2026-11-11 23:11:47+00:00

For a privacy-focused personal knowledge manager, index cost at scale changes the retrieval decision: tune a small, inspectable pipeline before increasing embedding dimensions or reranking every chunk.

Short answer: keep sparse and dense retrieval separate, fuse their ranked results, then tune chunking and reranking against a private evaluation set; this preserves relevance while keeping the number of indexed vectors under control.

I first tried one embedding per paragraph and a top-10 vector search. It looked clean in a notebook. In a personal archive, it missed exact names, file paths, and error codes because semantic similarity is not a synonym for lexical match. A long meeting note also occupied several neighboring slots, hiding a short note with the answer.

The useful experiment has a fixed corpus, a fixed question set, and a cost ledger. For each question, record the expected source, whether the answer appears in the first 5 and 20 results, token count sent to the generator, and index bytes. Do not copy a setting because its demo feels good. Measure recall, precision, latency, and storage together. I have found that this ledger also catches quiet regressions: a chunker change can leave recall flat while doubling vectors, and a prompt tweak can preserve answer quality while pushing context beyond the model's useful window. Those are different failures with different owners, so the record needs both retrieval and generation fields.

Measure first.

Here is a deliberately plain data shape for an evaluation row:

``` python
from dataclasses import dataclass

@dataclass
class EvalCase:
    query: str
    relevant_ids: set[str]
    sensitive: bool = True

cases = [
    EvalCase("Which note contains the 429 retry policy?", {"ops-2024-07"}),
    EvalCase("How did I choose the backup retention period?", {"decision-backup"}),
]
```

The numbers are intentionally small. A dozen hand-labeled cases expose ranking mistakes earlier than a dashboard full of aggregate averages.

Treat retrieval as staged evidence gathering. Stage one applies access rules and metadata filters before any similarity calculation. Stage two runs a sparse method such as BM25 for exact terms and a dense method for paraphrases. Stage three fuses the two lists, for example with reciprocal rank fusion. Stage four reranks only a narrow candidate set and passes compact excerpts to generation.

This ordering matters for private data. A deleted document must disappear from both the lexical and vector paths; filtering after fusion can leak its title or score into logs. Keep source identifiers, collection labels, and timestamps alongside each chunk, and make the deletion operation idempotent.

Chunk boundaries deserve their own test. Split on headings and paragraphs first, then enforce a token ceiling with a small overlap. If a definition routinely crosses the boundary, store the parent section and retrieve neighboring chunks as a controlled expansion. Blindly increasing overlap multiplies index size without guaranteeing better answers.

The following interface is intentionally generic. It makes ranking decisions visible and keeps provider-specific clients outside the evaluation code.

``` python
from collections.abc import Iterable

def rrf(*ranked_lists: Iterable[str], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for result_list in ranked_lists:
        for rank, item_id in enumerate(result_list, start=1):
            scores[item_id] = scores.get(item_id, 0.0) + 1.0 / (k + rank)
    return [item_id for item_id, _ in sorted(scores.items(), key=lambda pair: pair[1], reverse=True)]

def retrieve(query: str, limit: int = 20) -> list[str]:
    sparse_ids = sparse_search(query, limit=limit)
    dense_ids = dense_search(query, limit=limit)
    fused_ids = rrf(sparse_ids, dense_ids)
    return rerank(query, fused_ids[:limit])
```

The code leaves three operations explicit so an eval harness can replace each one with a deterministic fake. In production, log the query hash, filter set, candidate counts, and final identifiers, but never the private text by default. For prompt-cost awareness, cap the final context by tokens rather than by chunk count; one huge chunk can otherwise crowd out five useful snippets. I've kept the interfaces boring on purpose: a notebook prototype can call local functions, while a service can put the same contract behind a queue or HTTP boundary without changing the scoring test.

I use a paired comparison: run the old pipeline and the candidate pipeline on the same cases, then inspect every changed answer. A higher recall@20 with a lower precision@5 may be a good trade for exploratory search, but it is usually bad for an assistant that cites one source. Your mileage may vary when queries are mostly names instead of concepts; that uncertainty is exactly why the labeled set matters.

| Choice | Helps | Costs or limits | 
|---|---|---|
| Larger chunks | Preserves context across sentences | More tokens per result and fewer independent matches | 
| Higher overlap | Recovers boundary-spanning facts | More vectors and duplicate evidence | 
| Dense-only search | Handles paraphrases | Misses exact identifiers and rare terms | 
| Rerank every candidate | Can improve ordering | Adds latency and compute; unsuitable for interactive bulk import | 
| Strict metadata filters | Protects private collections | Can remove useful context when labels are incomplete | 

The catch is that this architecture is not suitable when the corpus is tiny, static, and made of short, exact records; a local inverted index can be simpler and faster. Stick with a single lexical index when semantic paraphrase is not part of the query workload. Conversely, if legal or medical review requires a reproducible explanation for every ranking, add stored feature scores and a human approval step instead of relying on an opaque reranker.

Index maintenance is part of relevance. Version the chunker and embedding model, write the version into each record, and re-embed in batches while serving the previous version. Track tombstone lag, duplicate chunks, empty extraction results, and p95 retrieval latency. An eval run that cannot tell which version produced a result is not a trustworthy regression test.

Ship the smallest pipeline that meets three thresholds on held-out private cases: required recall at the chosen cutoff, an acceptable p95 latency, and a storage budget per document. Review misses by category: lexical miss, boundary miss, filter miss, or reranker mistake. Each category points to a different fix, so changing the model for every miss wastes both time and index capacity.

The RAG paper describes retrieval-augmented generation as combining a parametric generator with non-parametric memory. That division is useful here: keep personal knowledge in the retriever you can delete, inspect, and re-index, while keeping prompts short enough to evaluate. I am not sure one universal cutoff exists; the right value follows from the privacy policy and the user's tolerance for a missed note.