A customer writes to support: "My parcel never arrived, give me my money back." You can write an agent that answers this in one evening. The loop from part 1, three tools, and in the demo it answers correctly. Then the process restarts in the middle of a refund, the model provider goes down for half an hour, the customer hits "send" twice. And it turns out the agent is a small part of the system.
This is the third and last part of a series. In the first we looked at the agent loop, tokens and caching. In the second we collected the practices: retries, idempotency, sagas, Temporal, human approval, evals. Here we put them together into one system. We start with an agent in one file, and at every step we run into a problem. Each problem adds a component, and the system diagram grows along with the text. The code, as before, is pseudocode: English names, as in real Python.
I draw the diagrams with C4. It's a way to draw architecture at different zoom levels, like a map. Context shows the system as one box among people and external systems. Container breaks it into applications and data stores. Component breaks one application into parts. The C4 authors recommend the first two levels to every team and the third only when it adds something. A container in C4 isn't Docker: it's anything that has to be running for the system to work, such as a service, a database or a queue.
TL;DR
Let's start with what we can do after part 1. The script keeps the dialog history in memory, calls the model and runs the tools the model asked for. There are three tools: track_order_status checks the order status with the delivery service, get_refund_policy fetches the refund rules, refund_payment returns the money. A similar track_order_status appears in OpenAI's guide to agents, where the example is also about support.
history = []
def handle(message):
history.append(message)
while True:
response = llm(history, tools=[track_order_status,
get_refund_policy, refund_payment])
history.append(response)
if response.stop_reason != "tool_use":
return response.text
for call in response.tool_calls:
history.append(run_tool(call))
At the Context level the system looks like this. A customer writes; an operator joins now and then. Outside are the model provider, the shop with its orders, the delivery service, the payment system and a knowledge base with articles.
C4, level 1. External systems aren't ours; we only call them.
In the demo this works. The problems start when there is more than one customer. The history lives in the process memory, so any restart wipes every dialog. The script serves one customer at a time. And if the process crashed between refund_payment and the reply to the customer, nobody knows whether the money went back or not.
The first thing we add is an entry point and memory. The script from step 0 did everything at once: it took the message, called the model and waited for its answer. We split it into two programs.
The API is a web service the chat sends the customer's message to. It validates the request, saves the message to the database, queues a "process the dialog" task there and immediately tells the chat the message was accepted. The API doesn't call the model, so it answers in milliseconds.
A worker is a separate background process where the agent itself lives, that is, the loop from step 0. The worker takes a task from the database, reads the dialog history, calls the model and the tools and writes the reply to the database, where the chat picks it up. You run several workers, and each takes its own task. The worker never talks to the customer directly.
The dialog history now lives in the database and survives any restart. SDKs have ready-made history stores. In the OpenAI Agents SDK these are sessions: SQLite, Redis, SQLAlchemy, MongoDB and your own implementations.
C4, level 2. The orange frame marks what was added in this step.
Why split them? The agent spends anywhere from seconds to a minute on one case. If the API did that work, every case would hold a connection open the whole time. So the slow work goes to workers: the API stays fast, and you add workers as load grows.
But the main problem is still there. Say a worker crashes in the middle of a case. It has checked the order status and called refund_payment, but the response never came back. A new worker picks the case up from the start. Call the refund again? If the first one went through, the customer gets paid twice. Don't call it? If it didn't go through, the customer gets nothing. The database knows what the customer said, but not at which step processing stopped.
This is where we need durable execution, which we covered in part 2. Temporal records every step in a history. After a crash another worker replays the history and continues from the same place. Finished steps don't run a second time: their results come from the history.
It's important to understand what Temporal is physically made of, because this is where people get confused most often. The Temporal Service is a server plus its own database. Inside, the server is split into four services. Frontend accepts calls, rate-limits them and checks permissions. History holds workflow state, Matching keeps the task queues, and Worker runs internal background jobs. Temporal has its own database: PostgreSQL, MySQL or Cassandra. That database is the only thing it needs to run.
And Temporal doesn't run your code. The documentation says it plainly: workers live outside the Temporal Service, and the application team writes and runs them. A worker polls the task queue itself and takes a new task only when it's free. That spreads the load across workers without a load balancer. If a worker crashes, the task stays in the queue and waits for the next one.
Temporal keeps the processing progress; the app database keeps the data.
Here's how the agent loop fits in. The loop lives in the workflow, and each model call and each tool call become activities. That's how Temporal's official integration with the OpenAI Agents SDK works: the loop, tool selection and handoffs between agents run in the workflow, model calls run as activities and aren't repeated on recovery. According to Temporal itself, OpenAI's Codex web agent runs on it. That's the vendor's claim; I couldn't find a write-up from OpenAI.
How does a case get into a workflow? You choose the workflow ID, and it should carry business meaning, such as an order number. We'll use the dialog id. Temporal guarantees that only one open execution can exist with a given ID. If the customer hits "send" twice, the second start won't create a duplicate. Update-with-Start is even more convenient. It's one call from the API: if a workflow for this dialog is already running, the new message goes into it; if not, the workflow starts. The call must specify a workflow ID conflict policy, here Use Existing. The API can also wait for the reply. The API no longer queues a task in the database: Temporal now holds the task queue, and the app database stores only the messages themselves.
@post("/chat/{dialog_id}")
def new_message(dialog_id, text):
db.save_message(dialog_id, text)
return temporal.update_with_start(
workflow=support_case, workflow_id=f"dialog-{dialog_id}",
id_conflict_policy=USE_EXISTING, # workflow already running: the message goes into it
update="new_message", arg=text)
There are three ways to write into a workflow. A Signal is an asynchronous message with no reply. An Update is a synchronous write: the sender waits for a result or an error, and the workflow can validate the request before accepting it. A Query reads the current state and writes nothing to the history. A new customer message fits an Update, an operator's decision fits a Signal, and the UI reads the status with a Query.
So why not keep everything in Temporal, since it stores the history anyway? The history of one execution is limited: 51,200 events or 50 MB. After that the execution is terminated. For a long dialog there's Continue-As-New: the workflow hands its state to a new execution with the same ID and a clean history. But keep customer data, orders and the conversation history the operator reads in the app database. One more reason: the Workflow ID is stored in plain text and visible in the UI and logs, so it must not contain personal data.
Say the model provider goes down for half an hour. Temporal does what it should: it retries the activity with growing backoff, by default with no limit on attempts. A hundred workers with a thousand open cases all hammer the dead API. Every attempt waits for a timeout and holds a connection. When the provider comes back, the whole queue hits it at once.
A retry is built for a failure that clears in seconds. For a long outage there's a different pattern, the circuit breaker. Martin Fowler describes it like this. A wrapper around the call counts failures. After a threshold it trips, and further calls get an error right away, without going to the service. After a while the breaker lets a trial call through. If it succeeds, the breaker closes again.
Microsoft's description of the pattern stresses the difference. A retry repeats an operation expecting it to succeed. A circuit breaker prevents an operation that is likely to fail. You can combine them: the retry goes through the breaker. But the retry must understand the breaker's error and stop trying if the failure isn't transient.
Where do you put the breaker? There are many workers, and each has its own memory. If the breaker lives in worker memory, each worker learns separately that the provider is down. So the breaker state has to be shared. A convenient place is an LLM gateway, a separate service between the workers and the providers. LiteLLM, for example, does what we need. It takes a model out of rotation if it failed more than a set number of times in a short window (in the docs example, more than three failures, and the model "cools down" for 30 seconds), switches to a fallback model and keeps this state in Redis, shared by all instances.
The breaker in the gateway is shared by all workers. Its state is in Redis.
What follows is my advice, not a rule from the docs. Let the model-call activity tell two kinds of errors apart. A transient error from the provider is retried by the Temporal policy. A "breaker open" error means the provider is down, and the workflow decides for itself: switch to a fallback model, wait longer or hand the case to an operator.
@activity
def call_llm(history):
response = gateway.chat(history, model="main", fallback="backup")
if response.error == "all_models_cooling_down":
raise NonRetryableError("providers unavailable") # don't burn Temporal retries
return response
@workflow
def support_case(...):
try:
response = activity(call_llm, history, retry=retry_policy)
except NonRetryableError:
return hand_off_to_operator("model unavailable")
The same breakers go in front of any external API that can be down for a long time: the delivery service, the payment system. The rule is the same: the breaker state is shared by all workers.
Now the refund itself. It touches three systems. First, create a return label with the delivery service. Then pay the money back through the payment system. Then notify the customer and update the order in the shop. There's no shared transaction across these systems. How a saga solves this was covered in detail in part 2: every step has a compensation prepared in advance, and on failure the completed steps are undone in reverse order.
Here I'll add a concept that Microsoft's description of the pattern calls a pivot transaction, the point of no return. Steps before it can be undone. After it there's nothing to undo, and the remaining steps have to be driven to completion, retried as many times as it takes. In our case the pivot is paying the money back. The label can be cancelled. Once the money is gone, the email to the customer and the order update must happen, even if that takes a day of retries.
def refund(decision):
label = activity(create_return_label, decision.order, key=decision.id)
try:
activity(refund_payment, decision.amount, key=decision.id) # point of no return
except PermanentError:
activity(cancel_return_label, label) # compensation
raise
activity(notify_customer, decision, key=decision.id, retry=FOREVER)
activity(update_order, decision, key=decision.id, retry=FOREVER)
Every action gets an idempotency key: the decision id. If Temporal repeats a step after a failure, the payment system returns the old response for the same key instead of charging again. A saga in Temporal is written as ordinary code in the workflow. Microsoft calls this variant orchestration: one coordinator knows all the steps and runs the compensations itself.
The model shouldn't carry out a large refund on its own. OpenAI's guide explicitly lists cancelling an order, a large refund and a payment among the reasons to bring in a human. An operator console appears in the system. The workflow shows it the decision and waits for a signal. The wait is recorded in the history, so the operator can answer an hour later, and the workers can restart in the meantime. How this works and what rules apply to approval is in part 2.
A waiting workflow doesn't occupy a worker: its state lives in the history.
The system is now reliable, but expensive and slow. Every case goes through the full agent loop, even the question "how do I return an item?", which has an article in the knowledge base. Yet cases come in three kinds. A question from the knowledge base needs one model call with search. "Where is my order" is solved by a chain: extract the number, call track_order_status, write a reply. Only a disputed refund needs judgment: the order may have reached the pickup point, the customer may not have collected it in time, the rules differ between products.
A chain (workflow, agents chain) is a system where the steps are written in code in advance. An agent is a system where the model itself decides which step to take next. That's how Anthropic, OpenAI and LangGraph separate them. Microsoft adds a ladder of complexity: a direct model call, a single agent with tools, multiple agents. Their advice is simple: take the lowest level that reliably solves the task. If a good prompt is enough, you don't need an agent.
Anthropic describes splitting by case type as the "routing" pattern, and its example is exactly support: general questions, refunds and technical support go to different processes with their own prompts and tools. A router on its own isn't an agent yet. The model makes one decision there and doesn't spin in any loop.
Inside the worker we can now see several components. This is the Component level in C4.
C4, level 3. A dashed orange frame marks where the model decides; plain code and the saga move the money.
kind = activity(classify, message, model="small") # router: one call
if kind == "faq":
answer = activity(answer_from_kb, message) # single call
elif kind == "where_is_order":
status = activity(track_order_status, message.order)
answer = activity(write_reply, status) # chain
else:
decision = agent_loop(message, tools=READ_ONLY_TOOLS) # agent
check_rules(decision) # code: limits, threshold
if decision.amount > THRESHOLD:
approved = wait_for_signal("operator_decision") # Signal from the console
if not approved:
return hand_off_to_operator(decision)
refund(decision) # saga
Two details are worth noting. The agent gets only read tools: order status, rules, customer history. It proposes a decision but doesn't move money. The money is moved by the saga, plain code, after the rule check and, for a large amount, operator approval.
The second detail is the right model for the job. OpenAI advises putting a small model on classification and trusting the refund decision to a strong one. The difference is noticeable: a thousand classifications at 2,000 input tokens each cost $0.23 on Claude Haiku 5.5 and $4.50 on Sonnet 5.5, at list prices as of the end of October 2026.
An operator opens a complaint: "The agent refused a refund even though the parcel never arrived." Why did it decide that? Which tools did it call, what did they return, how much did the case cost? Without a trace there are no answers.
We add trace collection. Every case gets a trace that shows each model and tool call, tokens, time and cost. The format is standard, OpenTelemetry with the GenAI conventions, which we covered in part 2. I would link the trace to the Workflow ID: then from a complaint the operator can get to both the Temporal history and the call trace.
One last small thing: events. After a refund the CRM has to learn that the order is closed, and analytics that a refund happened. If the worker updates the database first and then crashes before sending the event to the broker, the CRM never learns about the refund. That's what the transactional outbox pattern is for. The event is written to an outbox table in the same transaction as the order change. A separate process reads the table and sends the events to the broker. The record and the event either both appear or neither does.
C4, level 2, final version. The numbers on the containers show where each pattern lives.
The one-file agent turned into a system of about ten components. The easiest way to sum it up is which pattern sits where and what it protects against:
What matters in this diagram is less the list of components than the order in which they appeared. Each one appeared when something broke without it. In a small project half of them may not be needed for a long time.
The agent that answers "my parcel never arrived, give me my money back" takes up one box out of ten in this system. The other nine are there for the customer's money and the operator's peace of mind, not for the model. Almost all of them are familiar from ordinary backend work. Only one thing is new: inside there's now a component that decides differently every time. All the architecture around it is there so that this doesn't become a problem.
Which box in this diagram would you start with in your own system, and which would you throw out?
This is a series of three articles.