AI Agents: Architecture

Neciu Dan, technical co-founder of an ed-tech startup and Staff Software Engineer, published a guide on AI agent architecture, explaining that an AI agent is a model choosing steps inside a loop your code runs, enabled by tool calling added to APIs after ChatGPT's launch in late 2022. The article details the four key differences of AI APIs—non-deterministic, hallucination-prone, per-token pricing, and inability to do math—and distinguishes between a single call, a workflow, and an agent.

· ai /category/ai · 21 min read AI Agents: Architecture This series focuses on building AI agents in production. In the first part we're discussing different types of architecture and how to connect multiple agents working together with all the pros and cons. Neciu Dan Hi there, it's Dan, a technical co-founder of an ed-tech startup, host of Señors at Scale - a podcast for Senior Engineers, Organizer of ReactJS Barcelona meetup, international speaker and Staff Software Engineer, I'm here to share insights on combining technology and education to solve real problems. I write about startup challenges, tech innovations, and the Frontend Development. Subscribe to join me on this journey of transforming education through technology. Want to discuss Tech, Frontend or Startup life? Let's connect. https://www.linkedin.com/in/neciudan/ In late 2022, ChatGPT launched. For most of us that was our first contact with AI, and back then we had only two ways to use it: a chat box or a simple API call. That changed when the providers added tool calling to their APIs. A model can now ask your code to run a function, read the result, and decide what to do next. That combination, a model choosing steps inside a loop your code runs, is what people call an AI agent. Let’s imagine we have a travel product. One of our users types: “find us a week in Portugal in September, near a beach, under 1,500 euros, and we’re bringing a toddler.” Our code sends that request to the model along with four tools: search flights, search hotels, check airport transfers, and draft an itinerary. The model asks to search flights first. Our code runs the search and returns the prices. Mid-September comes back at half the price of early September, so the model shifts the dates and asks for hotels near the coast with cribs. It finds three, notices the cheapest one sits an hour from the airport while the flight lands at 11 pm, then checks the transfer options. Out of that come two complete itineraries, with a note asking whether we’d accept a red-eye to save 200 euros. The user picks one. Everybody’s happy, including the agent. This guide covers what agents are and how to architect this agentic flow, as part of a multi-article series where will discuss how to setup a harness for your agent, how to run evals and how to build a self improvement loop. Let’s go What an AI agent is Underneath every SDK, an AI call is an HTTP request: you POST text to an endpoint, and you get text back. js const res = await fetch 'https://api.anthropic.com/v1/messages', { method: 'POST', headers: { 'x-api-key': KEY, 'content-type': 'application/json' }, body: JSON.stringify { model: 'claude-opus-5', max tokens: 1000, messages: { role: 'user', content: 'Write a haiku about databases.' } , } , } ; It differs from every other API you’ve integrated in four ways: Non-deterministic. Call it twice with identical input and you get two different answers. It doesn’t know when it’s wrong. A false answer arrives with the same confidence as a true one. There’s no error code for “I made this up.” This is called hallucination. You pay per token, both directions. Your text gets chopped into tokens, chunks of roughly three quarters of a word, before the model reads any of it. A long document in your prompt costs money on every send. In a multi-step system that’s every step. It can’t do math. Counting the characters it wrote is a computation it doesn’t perform. Precise arithmetic is almost always wrong unless you hand it a tool. Now, the word “agent” itself can mean three different things: A single call. You send one request and you get one response back. Our Portugal message goes in, and one word comes out saying whether the traveler wants a new trip or a change to a booking she already has. A workflow. Several calls wired together by your code, in an order fixed in advance. You draft it, you review it, then you fix whatever the review found. An independent agent. The model decides what happens next and when it’s finished. You don’t know the sequence before it runs. Picking a model At the top sits a frontier model, their smartest and slowest one, which costs the most per call. Small, fast models fill the bottom of that range, answering in under a second for a fraction of a cent. So which one does our vacation agent use? The honest answer is several, because an agent is composed of jobs that each need a different level of intelligence. For example the agent that is driving the loop. This is the model that reads “find us a week in Portugal with a toddler,” decides that searching for flights comes first, reads the prices that come back, realizes the dates are wrong, and needs better information. Driving takes judgment. Paying the frontier rate per call gives us a model that picks the right tool the first time and recognizes a bad result when it reads one. If we put a small model in the driver’s seat, it picks the wrong tool, misreads results, and loops. Sometimes forever. The second kind of job is the simple, repetitive one. Somewhere in the pipeline, something reads each incoming message and answers one small question about it. Is this email a booking request or a complaint? That question, picking one label from a fixed list, is called classification . What’s the traveler’s budget in this message? Pulling a specific value out of free text like that is extraction . And once the message has a label, sending it to the right handler, complaints to one prompt, bookings to another, is called routing . All three go to the cheapest model in the range, since every answer is short and easy to check against the message it came from. Another job a model can do is reviewing. Does this itinerary make sense for a family with a toddler? Would I send this reply to an angry customer? Nothing below our strongest model answers those. A judge who’s weaker than the writer approves whatever the writer produces, turning the review step into a rubber stamp. There’s one more thing that determines which models we pick: how long the user is willing to wait. When we build something new, we start every job with the strongest model, because our first question is whether the feature works at all. A weak model makes that question unanswerable: when the output is bad, we can’t tell if the idea failed or the model did. Then we can downgrade later, one job at a time, after we have evals and we can measure the result of the cheaper models vs the frontier models. Architecture types So far, we’ve decided that our feature needs a model and have picked which models to use. The next decision is how the calls fit together: does one call do everything, do several calls run in a fixed order, or does one model check another’s work? Those arrangements repeat across products. After you build a few of these, the same ten patterns keep showing up. The code examples use a small ask helper that wraps the SDK call from the first section: you give it a system prompt, the user content, and optionally tools. 1. Single call input ──▶ model ──▶ output This diagram describes any feature that transforms one input into one output. Before any agent runs, something has to read each incoming message and decide what the user wants: async function classifyRequest message { return ask { system: 'Classify this travel message. Reply with JSON: ' + '{ "intent": "new trip" | "change booking" | "question" }', user: message, } ; } This is how we were building agents 3 years ago. And it works, but you will hit the limits pretty fast. As people add more requests to a single message, you end up adding more instructions to your prompt. You will start adding: “and if the message mentions a booked trip, also…”, then another clause, then another. The accuracy drops with each one, because the model juggles too many rules at once. 2. Chain input ──▶ call 1 ──▶ call 2 ──▶ call 3 ──▶ output A chain runs a fixed sequence where each output feeds the next call. For example, every itinerary the agent produces gets turned into a friendly confirmation email, in the user’s language. js async function confirmationEmail itinerary, userLang { const draft = await ask { system: WRITE CONFIRMATION PROMPT, user: itinerary } ; const local = await ask { system: Translate to ${userLang}. , user: draft } ; const report = await ask { system: CHECK NUMBERS PROMPT, user: JSON.stringify { itinerary, email: local } } ; if report.mismatches.length === 0 return local; return ask { system: FIX NUMBERS PROMPT, user: JSON.stringify { local, report } } ; } The steps never change order; no matter what the input says, that means our code owns the sequence. But be aware that errors compound down a chain, because every step inherits the previous step’s mistakes and adds its own. If we chain five steps that each succeed 95% of the time, the arithmetic leaves us a run that succeeds about 77% of the time. 60% of the time, it works everytime 3. Router ┌──▶ refund handler input ──▶ classify ──▶ bug handler └──▶ sales handler A router classifies first, then dispatches to a specialist prompt. Our new-trip handler knows how to gather requirements for the agent, our change-booking handler knows the rebooking rules, and our question handler knows the product FAQ, without any of them carrying the others’ rules. js const HANDLERS = { new trip: { system: NEW TRIP PROMPT }, change booking: { system: CHANGE BOOKING PROMPT }, question: { system: QUESTION PROMPT }, }; async function respond message { const { intent } = await classifyRequest message ; // shape 1, reuse return ask { ...HANDLERS intent , user: message.body } ; } But it can easily go wrong, first you need all the options beforehand and then when the classifier sends a change-booking request to the question handler, the question handler doesn’t complain; it produces a confident, well-written FAQ answer to someone whose flight leaves tomorrow. 4. Fan-out and voting input ──┬──▶ chapter 1 ──┐ ├──▶ chapter 2 ──┼──▶ combine ──▶ output └──▶ chapter 3 ──┘ Fan-out splits independent work and runs it in parallel. async function compareDestinations requirements, cities { // 'Lisbon', 'Faro', 'Madeira' const briefs = await Promise.all cities.map c = ask { system: DESTINATION BRIEF PROMPT, user: JSON.stringify { city: c, requirements } } ; return ask { system: COMPARE PROMPT, user: briefs.join '\n---\n' } ; } Each brief was written blind to the others, so if Faro’s brief doesn’t know Lisbon’s flights cost half as much, the final comparison rests on three documents that dont know anything about each other. Voting runs the same task several times and aggregates the answers. js async function readFareTotal farePage { const reads = await Promise.all 1, 2, 3 .map = ask { system: EXTRACT FARE PROMPT, user: farePage } ; const counts = tally reads.map r = r.total ; const winner, votes = counts 0 ; return votes = 2 ? { total: winner } : { needsHuman: true, reads }; } 5. Generator plus reviewer input ──▶ writer ──▶ draft ──▶ reviewer ──┬─▶ good ──▶ output ▲ │ └────── "fix these" ───────────┘ A second model critiques whatever the first one writes, so the writer revises against that critique until it passes or until we run out of rounds. js async function draftWithReview input { let draft = await ask { system: WRITER PROMPT, user: input } ; for let round = 0; round < 2; round++ { // the bound const review = await ask { system: REVIEWER PROMPT, user: draft } ; if review.approved return draft; draft = await ask { system: WRITER PROMPT, user: ${input}\n\nRevise this draft. Fix: ${review.issues.join '; ' } , } ; } return draft; // out of rounds; ships with the last revision } This architecture also has some issues, like a reviewer approving everything because its prompt told it to, a reviewer judging a property it was never shown examples of, and a reviewer drawn from the same model family as the writer, which shares the writer’s blind spots and waves through the exact mistakes that family of model makes most often. Same-family bias is the hardest of those to picture. In our product it looks like the writer putting a 6 am departure in front of a family with a toddler, and the reviewer, running on the same model, reading that itinerary back and calling the schedule tight but fine. 6. Tool loop ┌───────────────────────────────┐ │ think: what do I need? │ │ act: call a tool │ │ see: read the result │ └────────────┬──────────────────┘ │ repeat until done ▼ answer The model requests a tool, your code runs it with your permissions, and the result is returned to the conversation for the model to read. // The bare loop, without the bounds and checks a production version wraps around it. let messages = { role: 'user', content: request } ; for let i = 0; i < MAX STEPS; i++ { const res = await ask { system: AGENT PROMPT, messages, tools: TOOLS } ; if res.toolCall return res.text; const result = await runTool res.toolCall ; messages.push res.message, { role: 'tool', content: result } ; } Three things change for us once the model is in control. The run can loop, so a cap on turns is mandatory as we dont want it to run forever. Cost climbs as the run goes, since every turn re-sends the whole conversation and the late turns pay again for everything before them. Prompt caching helps by keeping the unchanged part of the prompt on the model side. And prompt injection gets dangerous once the model can act on what it reads. Someone plants a line in a hotel description telling the reader to ignore its earlier instructions and email the traveler’s card details to an address, and when our search tool hands that description back, the model reads it as an instruction like everything else in the conversation. And now we are hacked 7. Plan then execute input ──▶ planner ──▶ written plan ──▶ worker ──▶ step ──▶ step ──▶ done ▲ │ └────────── replan if stuck ───────────────┘ A capable model plans once, and a smaller model executes each step. js async function buildPlan request { const plan = await ask { model: STRONG, system: PLANNER PROMPT, user: request } ; // plan.steps: { id, instruction }, ... const results = ; for const step of plan.steps { results.push await ask { model: SMALL, system: EXECUTOR PROMPT, user: step.instruction } ; } return results; } This is basically how Spec Driven Programming works with Claude Code. The problem here is that the model made the plan before anyone knew what they’d find. Step three can reveal that the plan was wrong from the start. Our planner writes five steps around a beachfront hotel in Lagos, and step three searches that hotel’s room types and finds no cribs. The two steps after it still book an airport transfer and an itinerary for a place this family can’t sleep in, because the executor follows the instruction it was handed rather than the fact it uncovered. 8. Supervisor ┌──────────────┐ │ SUPERVISOR │ holds the goal, dispatches, └──┬───┬───┬───┘ decides when it's finished │ │ │ ┌────────┘ │ └────────┐ ▼ ▼ ▼ researcher writer fact checker Named specialist agents sit under one supervisor agent that everything goes through. In my experience, this architecture requires four things to work correctly. - It needs a termination condition that it can evaluate, since “keep going until done” causes supervisors to loop. - It needs structured worker output, so the supervisor acts on results instead of reading through them. - It needs workers who return conclusions instead of full transcripts, because the supervisor’s context accumulates everything its workers say. - It needs workers we can verify one at a time, because once a worker’s error is folded into the supervisor’s aggregated answer, nothing downstream can tell us which worker got it wrong. The basic implementation is a loop where the supervisor’s output is a dispatch decision: js let state = { requirements, findings: }; while state.done { const decision = await ask { model: STRONG, system: SUPERVISOR PROMPT, user: JSON.stringify state } ; // decision: { worker: 'flights' | 'hotels' | 'activities' | 'finish', task } if decision.worker === 'finish' break; const result = await WORKERS decision.worker decision.task ; state.findings.push { worker: decision.worker, result } ; // conclusions, never transcripts } 9. Handoff triage ──▶ billing ──▶ refunds ▲ │ └────────────┘ Peers pass control directly, with no coordinator above them. We see this on our travel product’s support side: a triage agent hands the conversation to the booking-changes agent. Halfway through, the user mentions the airline already canceled the flight, which makes it a refund case for a different specialist. js let current = 'triage'; for let turn = 0; turn < GLOBAL BUDGET; turn++ { // nobody owns "done", so the budget does const res = await AGENTS current .respond conversation ; conversation.push res.message ; if res.handoffTo current = res.handoffTo; // changes passes to refunds mid-thread else if res.finished return conversation; } Handoffs fail in a specific, almost comic way: no single agent owns “we’re done,” so control bounces between changes and refunds indefinitely, with each one politely handing the customer back to the other. 10. Human in the loop agent ──▶ proposal ──▶ you ──┬─▶ approve ──▶ do it ├─▶ edit ──▶ do the edited version └─▶ reject In this shape, a person checks the agent’s work before anything irreversible happens. The agent drafts the email and you press send, or it stops at a proposed migration until you approve. The implementation is a pause: the agent writes a proposal and stops, and a separate handler handles the response when the human responds. async function proposeBooking runId, itinerary { await db.saveProposal runId, { itinerary, status: 'awaiting approval' } ; await notifyUser runId ; // the run now sits idle, costing nothing } async function onUserDecision runId, decision { // approve | edit | reject await db.recordDecision runId, decision ; // the training data, one row if decision.action == 'reject' { await bookTrip decision.itinerary ; // the edited version, if edited } } This implementation produces the best training data you will ever collect, because approve, edit, and reject are graded labels generated by users in the course of their work. Picking one architecture | Your situation | Architecture | Number | |---|---|---| | One well-defined transformation | Single call | 1 | | Stages you can name in advance | Chain | 2 | | Your prompt fills up with “if the user asks about X…” | Router | 3 | | Independent pieces of one big job, or a wrong answer that costs more than three model calls | Fan-out and voting | 4 | | Quality improves when told what’s wrong | Generator + reviewer | 5 | | The next step depends on what it finds | Tool loop | 6 | | Many simple steps, cost matters | Plan then execute | 7 | | Named specialists, one accountable place | Supervisor | 8 | | Specialist unknowable until it unfolds | Handoff | 9 | | Anything irreversible | Human in the loop | 10 | As you can imagine, its not as simple as it looks. There are all sort of issues and problems when building multi-agent systems. Here are some that I encountered: - Multiple agents multiply what we spend. - Every extra worker multiplies the chance of being wrong, without us knowing - Passing information between agents takes longer than it looks. - When agents are expecting something concrete and they receive something else, they dont complain and do what they are told, which usually means bad results What you should do before committing to multiple agents: describe each agent’s job in one sentence, then ask whether another engineer would independently agree which agent handles a given input. Our architecture for the vacation agent Everything so far was theory, so let’s run it against our original example. A user lands on our website and types this into our one chat interface: “My husband and I want to go on vacation, find us a week in Portugal in September, near a beach, under 1,500 euros, and we’re bringing a toddler.” How would we architect this to optimize results and costs? We can trace the request through the architectures above. At every stage we ask the same two questions: which architecture fits this piece of work, and which model tier does the job deserve. The first thing that touches the message is a classifier, our first architecture, using a small, dumb, cheaper model. It answers one question with one label, picked from the three intents classifyRequest already knows about, and this message comes back with new trip . We learned earlier that classification has a checkable answer, so the cheap model handles it for a fraction of a cent. That label feeds our router form architecture 3. A new trip goes to the trip planner, a change booking goes to the rebooking handler, and a question gets the FAQ prompt. Our user’s message routes to the trip planner. The trip planner is the tool loop, architecture 6, and here is where we need the frontier model because dates can change, we need to consider transfer to the hotel, different types of rooms. We need judgement. When the loop produces an itinerary, we do a plain code check against the given budget if there is any . We extract the budget with an extraction agent and call this function: function checkBudget itinerary, budgetEur { if budgetEur == null { return { skipped: 'no budget stated' }; // recorded, so a silent skip } // never masquerades as a pass const total = itinerary.items.reduce sum, i = sum + i.priceEur, 0 ; return total budgetEur ? { violations: Total ${total} EUR exceeds ${budgetEur} EUR. Cut ${total - budgetEur} EUR. } : { violations: }; } Then an itinerary that passes the code check meets the reviewer, architecture 5, which also runs on the frontier model. It judges what code can’t: does this trip make sense with a toddler, and is an 11 pm landing followed by an hour of transfer something we’d propose to this family? We run the same flow two times second time it has the first itenerary in context . Then we hand the user the two options and let her pick one. This is the human-in-the-loop approach. Her approval, edit, or rejection lands in recordDecision as a labeled example of what she wanted. And can be used to fine-tune the model in the future After she approves and selects one itenerary, we trigger one more run just to confirm that all prices have not changed just to be extra safe that we dont charge the user more than she agreed to. And after the booking is confirmed, the chain from architecture 2 takes over on the small model: it drafts her personalised confirmation email. We can put the whole flow on one diagram: message │ ▼ classify architecture 1, small model │ new trip ▼ router architecture 3 │ ▼ extract architecture 1 again, small model │ { budgetEur, month, nights, toddler, near } ▼ tool loop architecture 6, frontier model │ itinerary search flights / search hotels / check transfers ▼ budget check plain code, zero tokens │ ▼ reviewer architecture 5, frontier model, two rounds max │ proposal ▼ the user architecture 10: approve / edit / reject │ approve ▼ fare voting architecture 4, three reads, small model │ ▼ book ──▶ confirmation chain architecture 2, small model An example of how it looks put together: async function handleMessage runId, message { const { intent } = await classifyRequest message ; // architecture 1, small model if intent == 'new trip' { return HANDLERS intent message ; // architecture 3 } const req = await extractRequirements message ; // small model, nulls for gaps if req.budgetEur == null { return askUser runId, 'What budget should I plan against?' ; } const itinerary = await runTripLoop message, req ; // the architecture 6 loop from above, // re-running until checkBudget // on its exit passes const proposal = await draftWithReview itinerary ; // architecture 5 await proposeBooking runId, proposal ; // architecture 10; we stop and wait } // onUserDecision from architecture 10 calls this when the decision is an approval async function bookApprovedTrip runId, decision { const fare = await readFareTotal decision.itinerary ; // architecture 4, three reads if fare.needsHuman return escalate runId, fare ; // a human queue, not a retry const booking = await bookTrip decision.itinerary ; return confirmationEmail booking, decision.userLang ; // architecture 2, small model } Why not one call to one powerful model? Everything above looks complicated, so why not push the whole job into one frontier model and have it handle all of this? The frontier model could handle the whole job on its own. We hand it the 4 tools and one big prompt. It classifies the intent itself, identifies the 1,500 budget in the text, plans, evaluates its own plan, and writes the confirmation email at the end. In a demo, it produces the same two itineraries as our pipeline, with a tenth of the code. But at scale, our pipeline has at least four benefits: - Money. Every token the solo version touches, input and output, is billed at frontier rates, where the frontier model costs 5 to 10 times more per token than the small one. Especially because the growing transcript adds an extra charge, because the loop re-sends the conversation on every turn. Then the classification, extraction, and confirmation email still run as frontier tokens. - Seams. Meaning places where our code can step in. Our checkBudget runs between the loop and the reviewer, reads a plain JSON itinerary, and feeds exact numbers back. If we had only the frontier model do the whole thing then the budget arithmetic happens inside the model, which is the one place we’ve established it sometimes fails. - Visibility. In our pipeline when a run goes wrong, we can see which part failed. When the frontier model proposes a hotel with no crib, we’re left staring at one long transcript, asking whether the model missed the toddler while reading, ignored it while planning, or failed to catch it while reviewing itself. Our pipeline leaves an artifact at every stage: a label, a requirements object, an itinerary, a review verdict. - Optimisations. With separate jobs, we can move one job at a time to a cheaper model and measure whether anything got worse and fine tune each model as we see fit. Also keep in mind that the architecture is still missing everything that keeps an agent alive in production: records, keeping the model alive, shared memory, etc. Everything that wraps the model calls is called the harness. We’re gonna discuss it in the next part. References Effective context engineering for AI agents https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents - Anthropic Building effective agents https://www.anthropic.com/engineering/building-effective-agents - Anthropic Prompt caching https://platform.claude.com/docs/en/build-with-claude/prompt-caching - Claude Platform docs Don’t Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks https://arxiv.org/abs/2601.06007 - the 41 to 80% measurement across three providers Discover more from The Neciu Dan Newsletter A weekly column on Tech & Education, startup building and occasional hot takes. Over 1,000 subscribers