Designing Smart AI Agents: Architecture Patterns That Survive Production

A practical field guide to agent topologies, state design, and the failure modes that separate a demo from a system that survives 90 days in production.

Six months ago, a logistics company in Dubai flew me in to look at their "autonomous customer operations" pilot. The vendor demo was stunning. An agent quoted delivery timelines, resolved address discrepancies, and flagged high-risk shipments for human review. In the controlled demo, it resolved 94% of test cases without a human in the loop.

I asked for the production numbers. A . Then the CTO pulled up a dashboard. On real traffic — about 4,000 tickets a day, messy addresses, late tracking feeds, angry customers — the same agent resolved 11% of cases, and it hallucinated a delivery promise onto at least a dozen of the rest. Some of those promises cost the company real money in refunds and re-shipping.

The demo was not fake. The model was fine. The problem was that nobody had designed the system's architecture. They had pointed a capable model at a prompt, wrapped it in a loop, and called it an agent.

This article is the pattern language I wish that vendor had used: the topologies, the state design, the tool contracts, and the failure modes that decide whether an agent survives production. By the end you will be able to look at any agent project, name the pattern it is using, and — more importantly — say whether it is the right one.

"Agent" is a marketing word. "Topology" is an engineering word. A topology is the shape of your system: how many reasoning loops exist, how they talk to each other, who owns the state, and who decides what happens next. When a production agent collapses, it is almost never the model's fault. It is a topology that did not match the task.

Every agent architecture, no matter how clever the slideware, is one of six topologies. Learn to spot them, because each has a cost profile, a failure mode, and a narrow range of tasks it is genuinely good at.

One model, one context window, a set of tools, a while loop. This is the default and the workhorse. The system prompt holds the goal, the context window holds working state, tools give it hands, and budget counters stop it from running forever.

Cost: lowest. Latency: lowest. Good for: narrow, well-scoped tasks — balance lookups, form extraction, single-domain Q&A with tools.

A small, fast model classifies the request and dispatches it to one of several specialized handlers. A ticket-triaging router sends payment disputes to a refund workflow, delivery questions to a tracking tool, and everything else to a general agent.

Cost: low. Latency: adds one cheap call. Good for: high-volume traffic where most requests are one of a few known shapes. This is the most underrated pattern in production, and the one I reach for first.

One orchestrator decomposes a task into subtasks and hands each to a worker agent (or a plain function, or a search job). Workers return results; the orchestrator synthesizes. This is what people actually mean when they say "multi-agent," and most of the time it is one orchestrator with several specialized workers.

Cost: medium. Latency: medium. Good for: report generation, research, code review — tasks with a natural breakdown.

Agents manage agents. A lead orchestrator spawns sub-orchestrators, each managing its own workers. This is how you scale orchestrator–worker to genuinely huge tasks, and it is also where complexity and cost start to compound.

Cost: high. Latency: high. Good for: enterprise research pipelines with thousands of documents. Usually a mistake for anything pattern 3 handles.

Multiple agents with equal standing converse or work in parallel toward a shared goal — the classic CrewAI and AutoGen picture: a researcher, a writer, and a critic arguing over a document until they agree.

Cost: high — every peer turn is a full model call and coordination overhead is real. Latency: high. Good for: creative drafting and debate-style tasks. Bad for: anything with a deadline and a strict budget.

No loop at all. A directed graph of steps — query, validate, charge, confirm — where each step is deterministic or model-assisted. LangGraph's graph model and n8n's node model are this pattern wearing graph paper.

Cost: lowest per step. Latency: predictable. Good for: anything that is 80% a known process with a few fuzzy decision points. Most "agent" use cases are secretly this, and it is the most honest pattern in the list.

Here is the quick reference I put in front of clients:

The most useful question I ask before writing any code: is this task a process with a few judgment calls, or an open-ended goal with unknown steps? Process → state machine. Open-ended → single loop or orchestrator–worker. Almost never a peer team on day one.

Now the part that kills more production agents than any topology choice: state.

An agent's state is everything it carries between steps — the task definition, what it has already tried, what it has ruled out, the results of tool calls, and the budget it has left. If state lives only in the model's context window, you have a memory problem: context windows are bounded, noisy, and easy to poison. If state lives in your database, you have an engineering problem: every step needs a save, a load, and a version.

Here is the rule I now enforce with clients. Working state (what is on the model's desk right now) goes in the context window, trimmed ruthlessly. Durable state (what this task has accomplished, across retries and restarts) goes in a store — Postgres for structured task state, a vector store for retrieved knowledge, Redis for ephemeral job state. Every step is a pure function of durable state plus the model's decision. That one discipline — "state in the store, not in the prompt" — fixed more agent projects than any model upgrade I have ever shipped.

Concretely, a task row in Postgres looks like this:

CREATE TABLE agent_tasks (
  id            uuid PRIMARY KEY,
  pattern       text NOT NULL,             -- which topology
  goal          text NOT NULL,
  status        text NOT NULL DEFAULT 'queued',
  step_count    int  NOT NULL DEFAULT 0,
  tool_calls    jsonb NOT NULL DEFAULT '[]',
  result        jsonb,
  created_at    timestamptz NOT NULL DEFAULT now()
);

Every tool call is appended to tool_calls

. If the process crashes, a worker picks up the row and replays from step_count

. That is the entire secret of "reliable" agents: they are just jobs that can resume.

I keep saying tools are the agent's hands, but the part people get wrong is the description. The model reads your tool description and decides whether to use the tool. Write a lazy description and the model will misuse it in production, every single time.

Treat the description as a contract with three clauses:

get_transactions

."{balance: number}

. Returns an error object if the account is not verified."One more rule: validate inputs server-side before execution. The model's arguments are model output — they can be wrong, and in adversarial inputs they can be malicious. A SQL injection string smuggled through a tool argument is not a joke; it is a Tuesday.

Here is the smallest orchestrator–worker I would ship, with the state discipline above. No framework — just Postgres, a queue, and two model calls per task.

import json
from typing import Any

def orchestrator(task: dict) -> str:
    plan = llm_call(
        "You are a research lead. Split this task into 3-5 subtasks "
        "that can be executed independently. Return JSON.",
        task["goal"],
    )
    subtasks = json.loads(plan)["subtasks"]

    results = []
    for sub in subtasks:
        results.append(worker(sub))           # worker may call tools
        save_task_state(task["id"], results)  # durable state every step

    return llm_call(
        "You are a synthesis editor. Combine these subtask results "
        "into one coherent answer for the original task.",
        json.dumps({"goal": task["goal"], "results": results}),
    )

def worker(subtask: dict) -> Any:
    return run_tool(subtask["tool"], subtask["args"])

def save_task_state(task_id: str, results: list) -> None:
    pass

Run this against real traffic and you will find the handoffs — the exact spots where context gets lost and tasks stall. That is the point: you want your failures in the handoff layer, because handoffs are cheap to instrument and cheap to fix. A hallucinated subtask decomposition, by contrast, is expensive to catch and expensive to repair.

After a year of shipping these systems across fintech, logistics, and support clients, here is my honest list of what breaks, ranked by how much it hurts:

And the biggest one, which is not technical at all: the demo/test gap. Your evaluation set was curated by the same person who built the system. Measure success on held-out production traffic from week one, or you will discover your own 11% version in a client call, like the logistics company did.

Back to the logistics company in Dubai. We rebuilt the pilot as a router plus a state machine: a cheap classifier sent tickets to one of five deterministic workflows, and only the genuinely fuzzy cases reached a single-loop agent with strict budgets and a permission layer. Resolution climbed from 11% to 78% in the first month — not because the model got better, but because the shape of the system finally matched the shape of the work.

The model was never the problem. The topology was.

Start by naming the pattern you are actually building. If you cannot name it, you do not have an architecture — you have a prompt with a cost. Draw the graph, put the state in the store, write the tool contracts, and treat the checklist above as your last step before production.

*Gulshan Yad