You’ve heard the terms AI agents, RAG, evals, multi-agents. Maybe you’ve used ChatGPT or Claude and wondered how you’d build something like that yourself. Or maybe you’re already building and the pieces aren’t clicking together yet.
This post is for both of those people.
Most guides throw you into frameworks, code, and jargon before you understand why any of it exists. This one starts with the map five nodes that show you exactly how AI products are structured, from the base model to production. Once you see how the pieces connect, everything else makes sense.
Roadmaps go stale. Mind maps don’t. Here’s the one you actually need.
Everything starts here. An LLM is a model trained on a massive slice of text the internet, books, code to predict what comes next. That training process is how it learns to reason, write, summarize, and converse.
But that same training process is also its fundamental constraint.
The LLM only knows what it was trained on. It has a cutoff date. It has no idea what happened last week, what's in your internal docs, what your customers said in yesterday's meeting, or what your product actually does. That knowledge is invisible to it.
On top of that, LLMs are unpredictable by nature. The same input or even a slightly different phrasing of the same input can produce widely different outputs. This is not a bug you can patch. It's the character of the thing.
Those two properties limited knowledge and probabilistic outputs are the source of almost every problem you'll encounter building AI products. The rest of this map is about dealing with them.
Running example: Imagine you’re building a customer support bot for an e-commerce store. You point it at GPT-4. Out of the box it knows nothing about your products, return policy, or order statuses. And every time a customer asks the same question, it might answer slightly differently. That’s Node 1 in practice you’re starting with a powerful but context-blind, non-deterministic engine.
A raw LLM is a powerful engine with no wheels, no steering wheel, and no fuel gauge. The wrapping layer is everything you build around it to make it actually useful.
There are three things the wrapping layer gives the LLM:
┌──────────────────────┐ │ │ Tools ───────►│ LLM │◄─────── Memory │ │ └──────────┬───────────┘ │ ▼ RAG / Knowledge
When you wrap an LLM with all three, you have what people call an AI agent.
An LLM can only answer based on knowledge it was trained on. Anything new, anything specific to your work, is invisible to it. RAG fixes that.
The idea is simple: open-book question answering. You build a knowledge base your documents, internal data, product info, policies, whatever the LLM needs to know that it couldn't have learned during training. When a question comes in, the most relevant information gets pulled from that knowledge base and handed to the LLM alongside the question. The LLM uses both to answer.
Question ──► [ Retrieve relevant docs ] ──► [ LLM + context ] ──► Answer
RAG has grown into its own field, but the easiest way to navigate it is to hold the name in your head: Retrieval + Augmented Generation.
Almost every variant of RAG is an improvement to one of those two halves. "Hybrid search" retrieval. "Reranking before prompting" retrieval. "Better prompt structure with citations" augmented generation. Once you see it, you can't unsee it.
An LLM by default has no memory across conversations. Each session starts fresh.
There are two layers:
** Short-term memory** : lives within a single session basically the conversation history. The LLM can refer back to what was said earlier in the chat. Most frameworks handle this for you automatically.
** Long-term memory** : is what persists across different sessions. Preferences, facts about the user, ongoing projects, past decisions. This is what makes an AI assistant feel like it actually knows you over time.
Some systems break long-term memory down further:
Underneath the taxonomy, the mechanics are always the same: preferences get stored in some persistent storage usually a database and get pulled back on demand. The sophistication is in what you store, when you store it, and how you decide to surface it.
A raw LLM can only produce text. It can’t search the web, check a database, send an email, or book a flight. It just generates the next word.
** Tools** are what change that. A tool is an external action the agent can trigger and this is what separates a chatbot from an agent.
When an agent has tools, the loop looks like this:
User: "What's the weather in Mumbai right now?"↓Agent thinks: "I need to call the weather API"↓Tool call → weather_api("Mumbai")↓Tool returns: "32°C, partly cloudy"↓Agent answers: "It's 32°C and partly cloudy in Mumbai right now."
The LLM doesn’t run the code it just decides which tool to call and with what input. Your harness actually executes it and feeds the result back.
Common tools in real AI products:
- Search (web or internal knowledge base)- Database queries- Sending emails or Slack messages- Calling external APIs (payment, calendar, CRM)- Running code
Without tools, an agent can only talk. With tools, an agent can act.
Whenever you sit down to build something, ask these three questions before writing a single line of code:
Answer these three and your architecture will mostly write itself.
Real example: For the customer support bot:
Tools: look up order status by order ID, initiate a refund, escalate to a human agent
Knowledge (RAG): your return policy docs, product FAQs, shipping information
Memory: remember the customer’s name and their last order within a session; remember their preferred language across sessions
Three questions answered. Architecture defined.
Most builders skip this. It's also the part that costs them the most later.
Consider what happened with Air Canada in 2024. Their chatbot told a customer they could book a bereavement fare and claim the discount retroactively a policy that didn't exist. The company was held legally liable for what their chatbot said. In hindsight, a simple check does the chatbot's response match the actual policy on the site could have caught this before it ever reached a user.
Those kinds of checks in AI products are called evals.
Normal software testing is simple: give it an input, you know exactly what the output should be, it either passes or it doesn't every time, without fail.
None of that works for LLMs.
Remember: LLMs are unpredictable by nature. The same input, or a slightly different phrasing, can produce widely different outputs. And there's often no single correct answer if you ask an LLM to summarize a document or draft an email, there are multiple valid ways to do it right and multiple ways to go wrong. The old testing playbook doesn't apply here.
Evals is the discipline of defining what "good" looks like for your system and systematically measuring whether the system delivers it.
One of the most popular ways to implement evals is using an LLM as a judge.
You take an AI model and use it to grade the output of your AI system. You give the judge:
The judge scores how well the response meets the rubric. You often run this against a small curated dataset a sample of example inputs and outputs you already know are good.
AI judges are faster than human review and more nuanced than simple unit tests.
The hard part is figuring out what belongs in the rubric for your specific use case:
Those three questions are your rubric. Get them written down before you go live.
At some point, whatever you're building will hit a wall. The task is too big, too complex, or needs too many different kinds of thinking for one agent to handle well. That's where more than one agent comes in.
But here's the thing most people get wrong: the moment something gets complicated, they immediately add a subagent. More agents always means more complexity, more failure points, and more things to debug.
There are two legitimate reasons to go multi-agent:
One agent can only juggle so many different things before it gets sloppy. When the work needs distinct skills, breaking it across agents lets each one tune for its part one handles research, one handles drafting, one handles checking. Each does its job better than one agent trying to master all three.
- Parallelization
Some work doesn't have to happen one step at a time. If done right, multi-agent systems can take on problems where multiple things happen in parallel faster and at a scale no single agent working in series could match.
Think about how Claude Code or Codex work under the hood: they spawn subagents to handle different parts of the work as needed. Not because one agent couldn't eventually do it but because specialized parallel work is faster and more reliable.
Before going multi-agent, ask the honest question:
Is a single agent genuinely hitting its limit, or is the real fix a better prompt, better retrieval, better memory, or better evals?
Most of the time it's the latter. Multi-agent is the move when you've genuinely exhausted single-agent headroom not when things feel complicated.
You built it. You evaluated it. You shipped it. Now what?
Production is a completely different beast from the lab. Real users surface edge cases your evals never imagined. The McDonald's AI drive-through is a good example it worked in testing, failed with real customers, and got pulled.
The difference between an AI product that improves and one that quietly degrades is whether you've built the AI Ops cycle:
Design │ Build │ Evaluate ◄──────────────────────────┐ │ │ Ship to Production │ │ │ Monitor │ │ │ Feedback ───────────────────────────┘
An AI product gets built once but improved almost every day in production. That's where the real work lives.
This loop never ends. The teams that treat it as the main job — not a post-launch afterthought are the ones whose AI products get better over time instead of quietly embarrassing them.
Node 1: LLM ─── The engine and its limitsNode 2: Wrapping Layer ─── Tools + Knowledge + Memory = AgentNode 3: Evals ─── Proving any of it actually worksNode 4: Multi-Agents ─── Scaling when one agent isn't enoughNode 5: Production Ops ─── Keeping it good after launch
Stop memorizing roadmaps that go out of date. Build the right fundamental framework and you'll always know where to look when something breaks, when to add complexity, and when to keep it simple.
The map doesn't change. The tools do.
Once you understand the 5 nodes, you’ll inevitably run into jargon. Here are the terms that come up constantly when people talk about AI agents explained plainly, mapped back to where they live in the framework.
Think of this as your translation layer: next time someone says “the harness is breaking the loop” or “we need semantic routing before the orchestrator,” you’ll know exactly what they mean.
An agent doesn’t just call an LLM once and stop. It cycles think, act, observe, think again. That cycle is called the agent loop.
Perceive (read the input) ↓Reason (what should I do?) ↓Act (call a tool, retrieve, write) ↓Observe (what came back?) ↓Repeat — until done or stuck
Loop engineering is designing that cycle deliberately — deciding how many times it can run, what counts as “done,” and what happens if it gets stuck. Most production failures in agents are loop failures: the agent spins, costs spiral, and nothing useful comes out.
Lives in: Node 2 (Wrapping Layer) and Node 5 (Production Ops)
The harness is everything that runs around the LLM the scaffolding that holds the agent together.
It’s what assembles the prompt before each call, routes tools, injects memory, catches errors, retries failures, and parses the output into something usable.
A weak harness means the LLM gets inconsistent inputs, has no safety net when something breaks, and returns raw text that nothing downstream can parse. Most production agent failures trace back to the harness, not the model.
Lives in: Node 2 (Wrapping Layer)
When a user sends a message, your system needs to decide what to do with it. Semantic routing is using the meaning of the message not just keywords to direct it to the right path.
For example:
Usually a small, cheap LLM does the routing before the main agent even sees the question. This keeps the system fast and the main agent focused.
Lives in: Node 2 (Wrapping Layer) and Node 4 (Multi-Agents)
When an LLM states something confidently that is simply not true. Not a lie the model has no intent but a plausible-sounding fabrication.
This is a direct consequence of Node 1: LLMs predict the next likely token, not ground truth. Hallucinations happen most when the model doesn’t have the right context and fills the gap with pattern-matching.
Grounding is the fix giving the LLM retrieved facts (RAG) so it answers from evidence rather than pattern memory.
Lives in: Node 1 (LLM) and Node 2 (Wrapping Layer RAG)
Instead of asking the LLM to jump straight to an answer, you ask it to reason step by step first.
“Think through this carefully before answering…”
This dramatically reduces hallucinations and errors on complex questions because the model builds its reasoning out into the open rather than compressing it into a single response.
Lives in: Node 1 (LLM) it's a way of getting better out of the base model
In a multi-agent system, the orchestrator is the agent that holds the big picture it plans, decides what needs to happen, and delegates.
Subagents are the workers each one has a focused job, its own tools, and its own context. They finish their piece and report back to the orchestrator.
Think of it like a project manager and a team. The PM doesn’t write every line of code; they decide who does what.
Lives in: Node 4 (Multi-Agents)
Rules and filters that constrain what your agent can say or do. They run around the LLM, not inside it.
Input guardrails check what comes in (block harmful prompts, strip PII). Output guardrails check what comes out (block policy violations, verify facts, flag confidence). Without guardrails, you’re relying entirely on the LLM’s internal alignment which is not enough for production.
The Air Canada chatbot had no output guardrail checking whether its responses matched actual company policy. That’s what a single guardrail could have caught.
Lives in: Node 2 (Wrapping Layer) and Node 3 (Evals)
A dial that controls how random the LLM’s outputs are.
For most production AI products you want low temperature. Unpredictability is already a property of the model you don’t need to turn it up.
Lives in: Node 1 (LLM)
The total amount of text an LLM can “see” at once your prompt, the conversation history, the retrieved documents, the tool outputs. Everything that goes into one call has to fit inside it.
Context windows have grown enormously (some models now handle millions of tokens), but cost, latency, and focus still push production systems to be selective. That’s why chunking in RAG, context compression, and semantic routing all exist to make sure the most relevant content fills the window, not the most recent.
Lives in: Node 1 (LLM) and Node 2 (Wrapping Layer)
A way of turning text into numbers specifically, a list of numbers (a vector) that represents the meaning of that text. Similar meanings produce similar vectors.
When you search a knowledge base in RAG, you’re not matching exact words. You’re finding vectors that are close in meaning to your question’s vector. That’s why semantic search finds the right passage even when the user phrases things differently than the document.
Lives in: Node 2 (Wrapping Layer — RAG)
The prompt is the full input you send to the LLM. The system prompt is the part that comes before the conversation it sets the LLM’s role, rules, and behaviour for the entire session.
“You are a helpful customer support agent for Acme Inc. You only answer questions about our products. You never discuss competitors.”
The system prompt is the first and most powerful guardrail you have. Most of the personality, consistency, and safety of an AI product lives here.
Lives in: Node 1 (LLM) and Node 2 (Wrapping Layer)
More about what do you in need production ready AI systems: https://www.gocodeo.com/post/what-makes-an-ai-agent-framework-production-ready
Understanding AI Agents: What Actually Works When Building AI Products was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.