A RAG demo that answers a handful of questions takes an afternoon to build. Getting it ready for real users is a different job. Data quality, security, monitoring, and cost all need attention, along with problems the demo never had to face.
To design a production-grade RAG system, start with an evaluation set, then improve retrieval using query routing with metadata filters, query decomposition, reranking, and hybrid search. Admittedly, that covers only retrieval quality, one part of a longer production checklist, but retrieval is where a RAG answer is won or lost. In n8n, each technique becomes its own node in a visual workflow, so you can test, debug, and monitor every step.
This guide walks through each technique using an AI assistant I built for questions about Claude Code, Cursor, Copilot, and Codex. Easy questions worked, but harder ones broke it: “Cursor pricing plan” cited a third-party blog instead of Cursor’s own page, and “Which tool has the most generous free tier?” got an answer that named no tool. I learned the first four techniques by fixing failures like these
Here is the trap: you add a technique, try one question, and the answer looks better. But that is one question out of hundreds, and you judged it hoping it worked.
So before changing anything, I built an evaluator: 17 questions in five groups (pricing, capability, comparison, rare exact terms, and out of scope). It scores every answer with fixed rules in code, like “did it cite the official pricing page?” and “does it quote the right price?” Deterministic rules are cheap to run and return the same verdict for the same input.
Remember this: “Cursor pricing plan” cited a third-party blog instead of Cursor’s own page. The system searched every source for every question, even when it knew the tool. Two techniques fixed that.
Three rules keep routing safe:
Routing fixed the Cursor question, but the free tier question named no tool, so no filter applied. One search returns the 10 nearest chunks, and nothing stops one tool from taking eight of them. Worse, no single chunk holds the answer; it must be assembled from four facts.
Query decomposition splits the question into parts and searches for each. A model can write sub-questions. When the names are known, you can instead run one filtered search per name. My parts were four fixed tools, so I chose the cheaper, more predictable second option. Each tool now gets its own reserved slots.
The model still skipped tools because Cursor’s free tier chunk never says “Cursor.” So I labelled every chunk in the prompt with its tool, and gave comparisons their own prompt: one paragraph per tool, then a conclusion. Decomposition multiplies searches, so run it only for comparisons.
Once everything passed, I added a harder question: “What does each tool cost per month for a single developer?” With only 2 chunks allowed per tool, it quoted $40 for Cursor and $39 for Copilot instead of $20 and $10. Here is what the vector search returned for Cursor:
Rank Chunk Cosine similarity 1 Teams, $40/user/mo 0.574 2 Enterprise, custom 0.535 3 Individual, $20/mo 0.524
At three chunks per tool, the correct result was only one position away from being dropped.
Normal search turns the question and each chunk into lists of numbers called embeddings (or vectors), then compares the lists. Nothing ever reads “single developer” next to “per user, team billing.” A reranker does: it reads the question and one chunk together and scores the pair. Scoring every pair is too slow for a whole index, so you search wide first, then rerank the shortlist.
You can run a reranker locally with the CrossEncoder class from sentence-transformers or call a hosted one. I chose Pinecone's hosted bge-reranker-v2-m3 because my vectors already lived there, and it returned the correct prices. I fetch 6 chunks per tool, keep 2, and rerank inside each tool’s group, because a global rerank can wipe out a tool.
Note: a reranker can only reorder what search hands it. Even though I fetched 6 chunks per tool, yours might vary: let your evaluation set decide. If the right chunk misses the shortlist, fetch more. If it makes the shortlist but still gets cut, check the reranker.
My exact-term questions passed with embedding search alone, so I never added hybrid search. Systems full of codes and product names often do.
Imagine an API support assistant asked, “What does ERR_SYNC_409 mean?” An embedding captures what a passage is about, so that code becomes one faint signal among many. Embedding search can easily return a general sync troubleshooting chunk that never mentions it. Keyword search ignores meaning but matches the exact term. Hybrid search runs both and merges the results.
Each side uses its own kind of vector:
Three decisions follow:
My advice: add hybrid search only when exact-term questions fail, and start with RRF, which avoids score calibration and is a good default to evaluate first. Apply the same metadata filters to both searches, and rerank after the merge.
In n8n, each technique becomes a node, and every execution shows what each step returned:
For hybrid search, the [**Weaviate Vector Store**](https://docs.n8n.io/integrations/builtin/cluster-nodes/root-nodes/n8n-nodes-langchain.vectorstoreweaviate/) node works inside *Get Many*: fill *Hybrid: Query Text* and set *Hybrid: Fusion Type* to *Ranked*. Filters and reranking still apply. The Pinecone node has no hybrid option, so add a **Postgres** keyword search, join both lists with a **Merge** node, compute RRF in a **Code** node by shared chunk ID, and rerank through an **HTTP Request** node.
For evaluation, store test questions in a data table, add an [Evaluation Trigger](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.evaluationtrigger/), score answers in a Code node, and record them with the [Evaluation](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.evaluation/) node’s *Set Metrics* operation. [Metric-based evaluations](https://docs.n8n.io/build/integrate-ai/test-and-improve-ai-workflows/use-metrics-to-measure-quality) need Cloud Pro or Enterprise, or self-hosted Enterprise. Registered Community and Starter allow one workflow.
How to Design a Production-Grade RAG System was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.