RAG for Beginners: 5 Levels of Building an AI That Actually Knows Your Stuff

A developer published a beginner's guide explaining retrieval-augmented generation (RAG) through five progressive levels of building an AI assistant that answers questions from a user's own documents rather than only its training data. The walkthrough covers document loading, chunking, embedding, vector storage, and cosine-similarity retrieval feeding an LLM such as GPT-4o, then addresses the limitations of pure semantic search for exact matches like invoice numbers.

Have you ever wished ChatGPT could answer questions about your documents — your company's HR policy, your class notes, your product manual — instead of just whatever it happened to learn from the internet? That's exactly the problem RAG solves. If you've seen the term "RAG" floating around and felt a little lost, don't worry — by the end of this post you'll understand it from the ground up, and see how a simple RAG app can grow, step by step, into the kind of AI assistant real companies run internally. We'll climb through 5 levels, each one a small, understandable upgrade on the last. No prior AI experience needed. Let's go 🚀 RAG stands for R etrieval- A ugmented G eneration. That's a mouthful, so let's translate it: 💡 Analogy : Picture two students taking a test. One has to answer purely from memory — that's a regular chatbot. It only knows what it was trained on, it has a knowledge cutoff, and it sometimes "hallucinates" confidently makes stuff up . The other student gets an open-book exam and can flip to the right page before answering — that's RAG. Same brain, very different accuracy, because now it can look things up before speaking. RAG lets an AI look things up in your documents before answering, instead of relying only on what it memorized during training. Let's build one, level by level. A basic RAG system has exactly two jobs: organize your documents so they're searchable, and search them when someone asks a question. Before anyone can ask anything, the documents need to be prepped: 📄 Load Files → ✂️ Chunk → 🔢 Embed → 🗄️ Store in a Vector Database 💡 Analogy : You wouldn't hand someone an entire 300-page manual just to answer "what's the return policy?" You'd flip to the one paragraph that matters. Chunking pre-cuts the book into paragraph-sized pieces so the AI can grab just the relevant bit later, instead of drowning in the whole document. 💡 Analogy : Think of an embedding like GPS coordinates, except instead of location, it represents meaning . Two chunks that mean similar things land near each other on this "meaning map," even with completely different wording — "I love my dog" and "my puppy is the best" would sit close together, while "stock market crash" would land far away. Now someone asks, "What's our vacation policy?" Here's the flow: ❓ Question → 🔢 Embed the Question → 🔍 Find Top 4 Closest Chunks → 🤖 LLM Writes an Answer The question gets embedded the same way the documents were, so it lands somewhere on that same meaning-map. The system then finds the 4 chunks sitting closest to it — measured with cosine similarity , which is really just a mathy way of asking "how similarly do these two arrows point?" Those 4 chunks, plus the original question, go to an LLM the AI model that actually writes the answer — here, GPT-4o , which drafts a response grounded in what was actually retrieved. 💡 Analogy : It's like asking a librarian a question. Instead of answering from foggy memory, they run to the shelf, grab the 4 most relevant books, skim them, and answer based on what's actually written down — not a guess. The catch: this works great as a first version, but it has a real blind spot — it's bad at exact matches. Search for "Invoice 4471" and pure meaning-based search might miss it entirely, because "meaning-close" isn't the same as "text-identical." That's exactly what Level 2 fixes. Let's meet the two search styles: 💡 Analogy : Imagine two friends helping you pick a restaurant. One is great at reading the vibe of what you want "cozy and quiet" even if you don't use the exact right words. The other is extremely literal — say "sushi" and they only think sushi, word for word. Each one misses things alone. Ask both and combine their answers, though, and you get a much better recommendation. Say someone asks "What is the leave policy?" Semantic search ranks Doc A highest, then Doc B, then Doc D. Keyword search ranks Doc C highest, then Doc A, then Doc B. The two lists disagree, and their scores live on totally different scales — so we can't just compare the raw numbers. Scary name, simple idea: instead of comparing raw scores, just look at where each document placed 1st, 2nd, 3rd... on each list, and combine the ranks: score = 1 / 60 + rank — added up across both lists a document appears in 💡 Analogy : It's like merging two friends' "Top 3 restaurants" lists. A place that shows up at 1 on both lists should win overall — even if it wasn't the single highest score on either one. That's a stronger signal of being genuinely good than acing one list and being absent from the other. In our example, Doc A wins the fusion — not because it topped either list alone, but because it did well on both . Now the search understands what you mean and what you typed. We still have a problem, though: this system can only look things up. It can't do math, take multi-step actions, or handle a request with two parts. On to Level 3. Suppose someone asks: "What's our travel policy, and how much is the per diem for a 5-day trip?" That needs two different skills — looking something up the per diem rate and doing math multiplying by 5 . A basic RAG pipeline can't do both in one shot. An agent is an AI that doesn't just blurt out an answer — it can pause, decide it needs a tool, use it, look at the result, and decide what to do next. This loop has a name: ReAct Reason + Act . 🤔 Think → 🛠️ Act use a tool → 👀 Observe the result → 🤔 Think again → ... → ✅ Final Answer 💡 Analogy : Think of a sharp personal assistant instead of a search engine. Ask them something tricky and they don't guess — they say "let me check," pick up the right tool a phone, a calculator, a filing cabinet , get the info, and then answer. They repeat this loop as many times as it takes. Our agent has 3 tools available: For the travel policy question, the agent's thought process looks like: search knowledge base → calculator → This runs on a framework called LangGraph , which tracks what's already happened and decides which path to take next — like a flowchart the AI follows live. This is the real turning point in the whole roadmap: the system stopped just looking things up and started completing tasks . 🎉 One agent juggling lots of tools works, but it strains as things get more complex — like one person trying to be a doctor, a lawyer, and an accountant all at once. Decent at all three, great at none. 💡 Analogy : Walk into a hospital and you don't head straight for a brain surgeon because you have a cold. There's a receptionist at the front who listens and sends you to the right specialist. That's exactly what an orchestrator agent does — except with questions instead of patients. ❓ Question → 🧭 Orchestrator classifies it → routes to → the right specialist agent Three specialists, each tuned for a different job: | Specialist | Best at | How it works | |---|---|---| | 📖 RAG Agent | Simple factual questions | Searches docs top 4 chunks , answers only from what it finds | | 🔎 Search Agent | Exact terms, codes, IDs | Uses keyword search BM25 to nail exact matches | | 🧩 Reasoning Agent | Comparisons & judgment calls | Pulls a wider set of chunks top 6 and reasons step by step | Ask "Compare our leave policy with market standards" and the orchestrator recognizes this isn't a simple lookup — it needs judgment — so it routes it to the Reasoning Agent , not the basic one. All three specialists share a common notebook shared state tracking the conversation, the question type, which sources were used, and a trace of what happened — so nothing gets lost when a question is handed off. The final answer comes back with receipts: sources, an agent trace, and a clean structured response, not just a paragraph. We've built something smart, but it still only works through one specific chat app. What if a Slack bot, an internal dashboard, and a mobile app should all share the same brain, without rebuilding it three times? 💡 Analogy : Before USB-C, every device had its own charging cable — one for your phone, one for your camera, one for your laptop. Chaos. USB-C fixed that by becoming one standard plug anything can use. MCP Model Context Protocol is basically USB-C for AI systems — a standard way for any app to plug into the same knowledge base and tools, without custom wiring every time. 🧑‍💼 Employee's App ⇄ MCP Server FastMCP ⇄ 🧠 RAG System The MCP server exposes two kinds of things: documents://policies , documents://faqs , like labeled folders anyone can open search documents , compare documents , search raw chunks , like buttons anyone can press Internally, the RAG pipeline gets one more upgrade too: Embed → Retrieve → Rerank → LLM . That new Rerank step is a second, more careful pass that re-checks the top results and puts the truly best ones first — a second opinion after the initial search. And instead of a loose paragraph, the reply now comes back as structured, predictable data built with a tool called Pydantic : { "answer": "Employees get 20 days of annual leave.", "confidence": "high", "sources": "hr policy.pdf" , "follow up questions": "How does unused leave carry over?" } 💡 Analogy : A random paragraph back is like asking a friend for directions and getting a rambling story. Structured JSON back is like turn-by-turn directions from Google Maps — predictable, and any app can use it without guessing what it means. This is what makes a system genuinely "enterprise-ready": any other piece of software can plug in and trust the shape of the response, every single time. | Level | What's new | In one sentence | |---|---|---| | 1️⃣ Basic RAG | Vector search | The AI can finally read your documents | | 2️⃣ Hybrid Search | Keyword + semantic fusion | It stops missing exact matches | | 3️⃣ Single Agent | Tools + the ReAct loop | It can act, not just answer | | 4️⃣ Multi-Agent | Orchestrator + specialists | It picks the right expert for the job | | 5️⃣ Knowledge Assistant | MCP + structured output | It becomes a service anything can plug into | If you're just starting out, don't try to build Level 5 on day one. Start with Level 1 — it's genuinely useful on its own — and treat every level after it as a targeted fix for one specific weakness you'll actually run into. Build, notice what breaks, climb a level, repeat. That's honestly how most real-world AI systems get built anyway. If this helped, I'd love to know which level you're building toward — drop a comment 👇