RAG Without the Hype: Make Retrieval Observable, Testable, and Replaceable

A developer detailed the design of a retrieval-augmented generation (RAG) pipeline for an LLM-powered support agent, emphasizing a keyword-overlap scorer that is fully assertable and replaceable. The system exposes retrieval as a tool with visible queries and scores, enabling debugging and quality tracking. The developer argues that ranked candidates combined with human or rule-based judgment outperform pure search or generation.

How my agent actually finds answers — and what happens when it doesn't Part 5 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code. "What's your refund policy?" Something has to know the answer. The model doesn't. Not reliably. The answer lives in documents the company wrote. Getting the right one in front of the model at the right moment has an intimidating name: retrieval-augmented generation RAG . And most explanations make it sound like magic. It's a pipeline. Score the documents, rank them, hand back the best few. That's all. The interesting part is what you do with the score. Fuzzy results behind a hard contract — that's the split this system is built on, and here it is made real. The agent doesn't get knowledge silently injected into its prompt. It gets a tool , the same way it gets customer lookup: // dev/tonal/support/knowledge/KnowledgeBase.java public interface KnowledgeBase { / Returns up to query.topK articles, best match first. / List<ScoredArticle search Query query ; } The agent decides when to search and what to ask. It never redefines what searching means, and every call is visible: query in, ranked articles with scores out. php flowchart LR A "Agent needs an answer" -- B "Query: text + topK" B -- C{"Scorer"} C -- D "Ranked articles + scores" D -- E "Top-k back to the agent<br/ as tool result" C -.- F "keyword overlap shipped " C -.- G "embeddings same port " classDef step fill: eef2f6,stroke: 8fa3b8,color: 24313f classDef decision fill: f7f4ec,stroke: b3a988,color: 24313f classDef alt fill: f7f9fb,stroke: c5d1dc,color: 24313f class A,B,D,E step class C decision class F,G alt Here's the part that breaks with convention: the shipped implementation scores articles by keyword overlap — plain code, no embeddings, no API key. // dev/tonal/support/knowledge/KeywordScoringKnowledgeBase.java public List<ScoredArticle search Query query { Set<String queryTokens = tokens query.text ; return articles.stream .map article - new ScoredArticle article, score article, queryTokens .filter scored - scored.score 0 .sorted Comparator.comparingDouble ScoredArticle::score .reversed .limit query.topK .toList ; } Why ship the dumb version? Because it's fully assertable. Four tests pin the whole behaviour: When an embedding-backed scorer replaces this class — same port, better matching on paraphrases — those tests define what honouring the contract means. Swap the implementation, keep the guarantees. Scores are also why retrieval is debuggable. Every match carries its number: bash $ java ... dev.tonal.support.knowledge.KnowledgeMain GET http://localhost:8080/rag/search?q=refund&k=3 1.00 Refund Policy billing GET http://localhost:8080/rag/search?q=xylophone No articles matched. When the agent later cites a policy, you can replay the exact query and see exactly what it was shown. No black box between the corpus and the answer. Retrieval being probabilistic means sometimes the ranker surfaces the wrong document — a rate-limit page for an SLA question. That's a failure mode like any other in this system: enumerated, mitigated, measured. The mitigation starts with honesty about scores a 0.2 match should be treated differently from a 1.0 , continues through grounding answers in what was actually retrieved rather than what the model remembers, and ends with the eval suite scoring whether answers follow from sources. A wrong document isn't a bug you fix once. It's a quality property you track. The pattern generalizes past support bots: Ranked candidates plus human-or-rule judgment beats either pure search or pure generation everywhere it matters.