# RAG for Beginners: 5 Levels of Building an AI That Actually Knows Your Stuff

> Source: <https://dev.to/ajmal_hasan/rag-for-beginners-5-levels-of-building-an-ai-that-actually-knows-your-stuff-4mmg>
> Published: 2026-09-13 10:26:44+00:00

Have you ever wished ChatGPT could answer questions about *your* documents — your company's HR policy, your class notes, your product manual — instead of just whatever it happened to learn from the internet? That's exactly the problem **RAG** solves.

If you've seen the term "RAG" floating around and felt a little lost, don't worry — by the end of this post you'll understand it from the ground up, and see how a simple RAG app can grow, step by step, into the kind of AI assistant real companies run internally.

We'll climb through 5 levels, each one a small, understandable upgrade on the last. No prior AI experience needed. Let's go 🚀

RAG stands for **R** etrieval-**A** ugmented **G** eneration. That's a mouthful, so let's translate it:

💡 **Analogy**: Picture two students taking a test. One has to answer purely from memory — that's a regular chatbot. It only knows what it was trained on, it has a knowledge cutoff, and it sometimes "hallucinates" (confidently makes stuff up). The other student gets an open-book exam and can flip to the right page before answering — that's RAG. Same brain, very different accuracy, because now it can *look things up* before speaking.

RAG lets an AI look things up in your documents before answering, instead of relying only on what it memorized during training. Let's build one, level by level.

A basic RAG system has exactly two jobs: organize your documents so they're searchable, and search them when someone asks a question.

Before anyone can ask anything, the documents need to be prepped:

```
📄 Load Files  →  ✂️ Chunk  →  🔢 Embed  →  🗄️ Store in a Vector Database
```

💡 **Analogy**: You wouldn't hand someone an entire 300-page manual just to answer "what's the return policy?" You'd flip to the one paragraph that matters. Chunking pre-cuts the book into paragraph-sized pieces so the AI can grab just the relevant bit later, instead of drowning in the whole document.

💡 **Analogy**: Think of an embedding like GPS coordinates, except instead of location, it represents *meaning*. Two chunks that mean similar things land near each other on this "meaning map," even with completely different wording — "I love my dog" and "my puppy is the best" would sit close together, while "stock market crash" would land far away.

Now someone asks, *"What's our vacation policy?"* Here's the flow:

```
❓ Question  →  🔢 Embed the Question  →  🔍 Find Top 4 Closest Chunks  →  🤖 LLM Writes an Answer
```

The question gets embedded the same way the documents were, so it lands somewhere on that same meaning-map. The system then finds the 4 chunks sitting closest to it — measured with **cosine similarity**, which is really just a mathy way of asking "how similarly do these two arrows point?" Those 4 chunks, plus the original question, go to an LLM (the AI model that actually writes the answer — here, GPT-4o), which drafts a response grounded in what was actually retrieved.

💡 **Analogy**: It's like asking a librarian a question. Instead of answering from foggy memory, they run to the shelf, grab the 4 most relevant books, skim them, and answer based on what's actually written down — not a guess.

**The catch:** this works great as a first version, but it has a real blind spot — it's bad at *exact* matches. Search for "Invoice #4471" and pure meaning-based search might miss it entirely, because "meaning-close" isn't the same as "text-identical." That's exactly what Level 2 fixes.

Let's meet the two search styles:

💡 **Analogy**: Imagine two friends helping you pick a restaurant. One is great at reading the *vibe* of what you want ("cozy and quiet") even if you don't use the exact right words. The other is extremely literal — say "sushi" and they only think sushi, word for word. Each one misses things alone. Ask both and combine their answers, though, and you get a much better recommendation.

Say someone asks *"What is the leave policy?"* Semantic search ranks Doc A highest, then Doc B, then Doc D. Keyword search ranks Doc C highest, then Doc A, then Doc B. The two lists disagree, and their scores live on totally different scales — so we can't just compare the raw numbers.

Scary name, simple idea: instead of comparing raw scores, just look at *where* each document placed (1st, 2nd, 3rd...) on each list, and combine the ranks:

```
score = 1 / (60 + rank)   — added up across both lists a document appears in
```

💡 **Analogy**: It's like merging two friends' "Top 3 restaurants" lists. A place that shows up at #1 on *both* lists should win overall — even if it wasn't the single highest score on either one. That's a stronger signal of being genuinely good than acing one list and being absent from the other.

In our example, Doc A wins the fusion — not because it topped either list alone, but because it did well on *both*. Now the search understands what you mean **and** what you typed. We still have a problem, though: this system can only look things up. It can't do math, take multi-step actions, or handle a request with two parts. On to Level 3.

Suppose someone asks: *"What's our travel policy, and how much is the per diem for a 5-day trip?"* That needs **two** different skills — looking something up (the per diem rate) **and** doing math (multiplying by 5). A basic RAG pipeline can't do both in one shot.

An **agent** is an AI that doesn't just blurt out an answer — it can pause, decide it needs a tool, use it, look at the result, and decide what to do next. This loop has a name: **ReAct** (Reason + Act).

```
🤔 Think  →  🛠️ Act (use a tool)  →  👀 Observe the result  →  🤔 Think again  → ...  →  ✅ Final Answer
```

💡 **Analogy**: Think of a sharp personal assistant instead of a search engine. Ask them something tricky and they don't guess — they say "let me check," pick up the right tool (a phone, a calculator, a filing cabinet), get the info, and *then* answer. They repeat this loop as many times as it takes.

Our agent has 3 tools available:

For the travel policy question, the agent's thought process looks like:

`search_knowledge_base` → `calculator` → This runs on a framework called **LangGraph**, which tracks what's already happened and decides which path to take next — like a flowchart the AI follows live. This is the real turning point in the whole roadmap: the system stopped just *looking things up* and started *completing tasks*. 🎉

One agent juggling lots of tools works, but it strains as things get more complex — like one person trying to be a doctor, a lawyer, and an accountant all at once. Decent at all three, great at none.

💡 **Analogy**: Walk into a hospital and you don't head straight for a brain surgeon because you have a cold. There's a receptionist at the front who listens and sends you to the right specialist. That's exactly what an **orchestrator agent** does — except with questions instead of patients.

```
❓ Question  →  🧭 Orchestrator (classifies it)  →  routes to  →  the right specialist agent
```

Three specialists, each tuned for a different job:

| Specialist | Best at | How it works | 
|---|---|---|
| 📖 RAG Agent | Simple factual questions | Searches docs (top 4 chunks), answers *only* from what it finds | 
| 🔎 Search Agent | Exact terms, codes, IDs | Uses keyword search (BM25) to nail exact matches | 
| 🧩 Reasoning Agent | Comparisons & judgment calls | Pulls a wider set of chunks (top 6) and reasons step by step | 

Ask *"Compare our leave policy with market standards"* and the orchestrator recognizes this isn't a simple lookup — it needs judgment — so it routes it to the **Reasoning Agent**, not the basic one.

All three specialists share a common notebook (**shared state**) tracking the conversation, the question type, which sources were used, and a trace of what happened — so nothing gets lost when a question is handed off. The final answer comes back with receipts: sources, an agent trace, and a clean structured response, not just a paragraph.

We've built something smart, but it still only works through one specific chat app. What if a Slack bot, an internal dashboard, and a mobile app should all share the *same* brain, without rebuilding it three times?

💡 **Analogy**: Before USB-C, every device had its own charging cable — one for your phone, one for your camera, one for your laptop. Chaos. USB-C fixed that by becoming one standard plug anything can use. **MCP (Model Context Protocol)** is basically USB-C for AI systems — a standard way for *any* app to plug into the same knowledge base and tools, without custom wiring every time.

```
🧑‍💼 Employee's App  ⇄  MCP Server (FastMCP)  ⇄  🧠 RAG System
```

The MCP server exposes two kinds of things:

`documents://policies`, `documents://faqs`, like labeled folders anyone can open` search_documents()`, `compare_documents()`, `search_raw_chunks()`, like buttons anyone can press
Internally, the RAG pipeline gets one more upgrade too: **Embed → Retrieve → Rerank → LLM**. That new **Rerank** step is a second, more careful pass that re-checks the top results and puts the truly best ones first — a second opinion after the initial search.

And instead of a loose paragraph, the reply now comes back as **structured, predictable data** (built with a tool called Pydantic):

```
{
  "answer": "Employees get 20 days of annual leave.",
  "confidence": "high",
  "sources": ["hr_policy.pdf"],
  "follow_up_questions": ["How does unused leave carry over?"]
}
```

💡 **Analogy**: A random paragraph back is like asking a friend for directions and getting a rambling story. Structured JSON back is like turn-by-turn directions from Google Maps — predictable, and any app can use it without guessing what it means.

This is what makes a system genuinely "enterprise-ready": any other piece of software can plug in and trust the shape of the response, every single time.

| Level | What's new | In one sentence | 
|---|---|---|
| 1️⃣ Basic RAG | Vector search | The AI can finally read your documents | 
| 2️⃣ Hybrid Search | Keyword + semantic fusion | It stops missing exact matches | 
| 3️⃣ Single Agent | Tools + the ReAct loop | It can act, not just answer | 
| 4️⃣ Multi-Agent | Orchestrator + specialists | It picks the right expert for the job | 
| 5️⃣ Knowledge Assistant | MCP + structured output | It becomes a service anything can plug into | 

If you're just starting out, don't try to build Level 5 on day one. Start with Level 1 — it's genuinely useful on its own — and treat every level after it as a targeted fix for one specific weakness you'll actually run into. Build, notice what breaks, climb a level, repeat. That's honestly how most real-world AI systems get built anyway.

If this helped, I'd love to know which level you're building toward — drop a comment! 👇