{"slug": "rag-without-the-hype-make-retrieval-observable-testable-and-replaceable", "title": "RAG Without the Hype: Make Retrieval Observable, Testable, and Replaceable", "summary": "A developer detailed the design of a retrieval-augmented generation (RAG) pipeline for an LLM-powered support agent, emphasizing a keyword-overlap scorer that is fully assertable and replaceable. The system exposes retrieval as a tool with visible queries and scores, enabling debugging and quality tracking. The developer argues that ranked candidates combined with human or rule-based judgment outperform pure search or generation.", "body_md": "*How my agent actually finds answers — and what happens when it doesn't*\n\nPart 5 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The\n\n[companion repo]contains the full code.\n\n\"What's your refund policy?\"\n\nSomething has to know the answer. The model doesn't. Not reliably.\n\nThe answer lives in documents the company wrote. Getting the right one in front of the model at the right moment has an intimidating name: **retrieval-augmented generation (RAG)**. And most explanations make it sound like magic.\n\nIt's a pipeline. Score the documents, rank them, hand back the best few. That's all. **The interesting part is what you do with the score.**\n\nFuzzy results behind a hard contract — that's the split this system is built on, and here it is made real.\n\nThe agent doesn't get knowledge silently injected into its prompt. It gets a *tool*, the same way it gets customer lookup:\n\n```\n// dev/tonal/support/knowledge/KnowledgeBase.java\npublic interface KnowledgeBase {\n\n    /** Returns up to query.topK() articles, best match first. */\n    List<ScoredArticle> search(Query query);\n}\n```\n\nThe agent decides *when* to search and *what* to ask. It never redefines what searching means, and every call is visible: query in, ranked articles with scores out.\n\n``` php\nflowchart LR\n    A[\"Agent needs an answer\"] --> B[\"Query: text + topK\"]\n    B --> C{\"Scorer\"}\n    C --> D[\"Ranked articles + scores\"]\n    D --> E[\"Top-k back to the agent<br/>as tool result\"]\n    C -.-> F[\"keyword overlap (shipped)\"]\n    C -.-> G[\"embeddings (same port)\"]\n    classDef step fill:#eef2f6,stroke:#8fa3b8,color:#24313f\n    classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f\n    classDef alt fill:#f7f9fb,stroke:#c5d1dc,color:#24313f\n    class A,B,D,E step\n    class C decision\n    class F,G alt\n```\n\nHere's the part that breaks with convention: the shipped implementation scores articles by keyword overlap — plain code, no embeddings, no API key.\n\n```\n// dev/tonal/support/knowledge/KeywordScoringKnowledgeBase.java\npublic List<ScoredArticle> search(Query query) {\n    Set<String> queryTokens = tokens(query.text());\n    return articles.stream()\n            .map(article -> new ScoredArticle(article, score(article, queryTokens)))\n            .filter(scored -> scored.score() > 0)\n            .sorted(Comparator.comparingDouble(ScoredArticle::score).reversed())\n            .limit(query.topK())\n            .toList();\n}\n```\n\nWhy ship the dumb version? **Because it's fully assertable.**\n\nFour tests pin the whole behaviour:\n\nWhen an embedding-backed scorer replaces this class — same port, better matching on paraphrases — those tests define what honouring the contract means. Swap the implementation, keep the guarantees.\n\nScores are also why retrieval is debuggable. Every match carries its number:\n\n``` bash\n$ java ... dev.tonal.support.knowledge.KnowledgeMain\n# GET http://localhost:8080/rag/search?q=refund&k=3\n[1.00] Refund Policy (billing)\n\n# GET http://localhost:8080/rag/search?q=xylophone\nNo articles matched.\n```\n\nWhen the agent later cites a policy, you can replay the exact query and see exactly what it was shown. No black box between the corpus and the answer.\n\nRetrieval being probabilistic means sometimes the ranker surfaces the wrong document — a rate-limit page for an SLA question. That's a failure mode like any other in this system: enumerated, mitigated, measured.\n\nThe mitigation starts with honesty about scores (a 0.2 match should be treated differently from a 1.0), continues through grounding answers in what was actually retrieved rather than what the model remembers, and ends with the eval suite scoring whether answers follow from sources. **A wrong document isn't a bug you fix once. It's a quality property you track.**\n\nThe pattern generalizes past support bots:\n\n**Ranked candidates plus human-or-rule judgment beats either pure search or pure generation everywhere it matters.**", "url": "https://wpnews.pro/news/rag-without-the-hype-make-retrieval-observable-testable-and-replaceable", "canonical_source": "https://dev.to/tonal/rag-without-the-hype-make-retrieval-observable-testable-and-replaceable-gl0", "published_at": "2026-08-31 10:13:10+00:00", "updated_at": "2026-08-31 10:21:57.661130+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/rag-without-the-hype-make-retrieval-observable-testable-and-replaceable", "markdown": "https://wpnews.pro/news/rag-without-the-hype-make-retrieval-observable-testable-and-replaceable.md", "text": "https://wpnews.pro/news/rag-without-the-hype-make-retrieval-observable-testable-and-replaceable.txt", "jsonld": "https://wpnews.pro/news/rag-without-the-hype-make-retrieval-observable-testable-and-replaceable.jsonld"}}