Written by Farhan Hasin Chowdhury.
AI agents are getting a lot of attention these days, and it’s easy to see why. They can handle the repetitive parts of multi-step tasks while leaving the important decisions to a human.
Think about an SRE agent responsible for investigating production alerts. It queries metrics and logs and checks recent deployments for evidence. It produces a diagnosis and recommends an action. An on-call engineer reviews and approves the action. After carrying out the approved action, the agent checks whether the service has recovered and records the result in an incident summary.
But what is an agent?
In the context of this article, an agent is a software system that can call a model or tool, write state, start a subagent, or retry a failed step. It may complete a task on its own or handle one stage of a larger workflow.
In the SRE agent example, each production alert starts a separate run of the agent. A run is one active execution of that agent. It can complete the investigation on its own or start subagents to handle parts of it. The agent works well in a prototype with a single active run, so the company decides to run multiple instances simultaniously.
Multiple agents make sense when different parts of a task can run at the same time or require different information. During an SRE agent run, one agent might inspect logs while another checks metrics or recent deployments. A linear task may still work better with one agent.
The move from a single run to many sounds simple. Once multiple SRE agent runs start sharing the same infrastructure, however, problems not present in the single-run prototype may start to show up. The runs compete for limited resources, write state at the same time, and share access to external systems. Most of this new pressure falls on the infrastructure around the agents.
In this article, we’ll work through six infrastructure problems involving capacity, state isolation, failure recovery, identity, tracing, and framework boundaries.
Resources are finite. With one active SRE agent run, each step gets uncontested access to the resources it needs. However, with more runs in parallel, demand for shared resources increases. So the platform now has to decide which runs can start and which ones have to wait.
Each SRE agent run adds several kinds of demand. It sends requests to the model provider, consumes tokens, calls tools such as the telemetry backend, and writes checkpoints as the investigation progresses. The total load depends on how many runs are active and how often each one performs those operations.
A little math can make the jump in demand easier to visualize:
model_requests_per_minute = active_runs * model_calls_per_run_per_minutemodel_tokens_per_minute = model_requests_per_minute * average_tokens_per_calltool_requests_per_minute = active_runs * tool_calls_per_run_per_minutestate_writes_per_minute = active_runs * checkpointed_steps_per_run_per_minute
For a simple example, assume there are n active SRE agent runs. Each model call run takes 1 minute, and each call contains 4,000 input tokens plus 1,000 output tokens. Leave retries, subagents, retrieval, embeddings, reranking, and evaluation traffic out for now. Even then, the workload reaches n requests and 5,000 * n tokens per minute.
If each run calls the model every 15 seconds, the same system reaches 4 * n requests and 20,000 * n tokens per minute. The higher call frequency alone produces this increase without changing the SRE agent's behavior.
Now consider two SRE agent runs arriving while the model provider and telemetry backend are close to their limits. One is investigating an outage that is affecting users right now. The other is preparing a report for an incident that has already been resolved. Starting both immediately would cause them to compete for the same model and telemetry capacity, even though the first one is more urgent.
Admission control makes that choice before either run consumes those resources. It can start the outage investigation and leave the report in a queue until capacity becomes available. The control point can reserve room for high-priority incidents, cap active investigations per service or team, and limit how many retries a single run can consume. Lower-priority work can wait or run with a smaller token budget, leaving more capacity for more important scenarios like an active incident.
Even with admission control, providers will reject some requests due to rate limits. In such cases, the agent may retry after a delay. One common approach is exponential backoff: the agent waits longer after each failed attempt and adds a small randomized delay within configured limits. This spreads retries over time. OpenAI recommends this approach, and its documentation notes that unsuccessful requests still count toward per-minute limits.
But if several delayed jobs retry together, they consume the limited quota again and push fresh work further back. Pressure from one service can then delay other services sharing that quota. Exponential backoff with jitter reduces the chance of those retries arriving together, but it does not limit the amount of work competing for capacity. Admission control limits that work, while per-service quotas prevent one service from consuming capacity reserved for others.
Model quotas are rarely the first bottleneck. Retrieval systems, operational databases, telemetry backends, and checkpoint stores often saturate earlier once many runs compete for them simultaneously, and admission control needs to govern all of that load, model traffic included.
The second big problem is about the state. In this context, state simply means the information an SRE agent run carries between steps, retries, and workers. In a prototype, all of that state may live in a single conversation object inside the agent process. But with multiple runs active, the platform has to keep its state separate.
For the SRE example, it helps to group the information associated with each run into four categories:
This four-way split is one useful way to think about it, not an industry-standard taxonomy. Other treatments use as few as two categories, splitting only thread-scoped state from longer-lived cross-thread facts, or as many as five, breaking tool state and file state out on their own.
Keeping all four categories in one conversation object makes it difficult to control which agent can read or update each piece of information. The system also needs to know which version of the incident report is current and which checkpoint a resumed worker should load. Each run and branch, therefore, needs its own state boundary, along with clear rules for anything the branches share.
You can find a similar approach to conversation and workflow state in LangGraph. It can save an agent’s progress through a checkpoint. The application gives each thread a unique identifier, which the checkpointer uses to store and retrieve the correct progress. In the SRE example, each agent run can have its own thread and resume without the state of another investigation. The checkpoints and data shared across runs can both live in the same durable store, while the application still decides which agents can read or update each record.
LangGraph’s checkpointer and store specifically cover conversation and workflow state. Neither has a built-in artifact or audit primitive, so those two categories still need to be modeled separately, wherever the state lives.
Crashes in software are inevitable, and an agent is no different. Suppose the SRE agent identifies a suspicious deployment, receives approval to roll it back, and calls the release system. The worker then crashes before recording whether the call succeeded or not.
When the worker restarts, the saved state does not show whether the release system received or completed the rollback request. Calling the release system again could submit the same rollback twice. Restarting the entire investigation does not remove that uncertainty. It also repeats model and tool calls that have already been completed.
A durable execution records enough progress to continue from the last safe point. For the SRE agent, the system can save checkpoints after gathering evidence, receiving approval, and getting a response from the release system. Stable identifiers such as run_id, agent_id, and step_id connect each checkpoint to the correct execution.
The approval record should live outside the worker process so that it survives a crash. Record the approval separately from the release-system call and use an idempotency key when the release system supports one. Store the call result and retry history too. A restarted worker can then check what happened before deciding whether to try the change again.
The system on the other end of the call has to enforce idempotency; a retry by itself doesn’t guarantee it. Durable execution engines commonly key each side-effecting call to something stable, such as the run ID combined with the step ID, so a duplicate call carrying the same key is recognized and skipped rather than repeated.
An agent operating in production needs its own identity. That identity lets the platform control which systems the agent can access and what it’s allowed to do. The agent might have read-only access to one system but permission to change another. These systems authenticate callers using API keys, access tokens, service account credentials, or a similar mechanism. A prototype might store a single shared credential in an environment variable and reuse it across every agent run.
But sharing a single credential across agent runs blurs several identities together. For a rollback request, the system needs to connect the engineer who approved the change, the agent who submitted it, and the service it targeted. When all agent use the same credential, the release system sees only one caller. Its records identify the shared credential rather than the agent and engineer behind the request.
One way to clean this up is to place a tool gateway between the agents and the production systems they use. Instead of giving each agent direct access to downstream credentials, the platform sends agent requests through the gateway. For a rollback, the request includes the agent’s identity and a reference to the engineer’s approval. The gateway checks whether that agent can modify the affected service, calls the release system with the required credentials, and records the decision and result.
The access rules now live at the gateway. The team can remove one agent’s access without affecting other agents. The implementation can vary, but it still needs to keep the agent’s identity, the engineer’s authority, the target service, and the requested action separate.
This pattern is now backed by a published standard. The Model Context Protocol’s authorization specification treats each tool server as its own OAuth 2.1 resource server and requires a token to carry a resource indicator naming which server it’s meant for. It also prohibits passing that token through to a different upstream system than the one it was issued for, the same confused-deputy problem a single shared credential creates.
An SRE agent produces events throughout an investigation. For example, the system may record a model call, a tool response, or an approval decision. A trace connects events from one run through shared identifiers so the team can reconstruct how the agent moved from the initial alert to the final outcome.
An audit record is one event within that trace. For a protected action, it can record who requested it, what they requested, which system they targeted, whether the request was allowed, and what happened as a result. A tool gateway can create this record whenever an agent requests access to a protected production system.
Consider a rollback request. Its audit record may show that the release system accepted the request. It does not explain what triggered the rollback, which evidence shaped the diagnosis, how many retries occurred, or why the run made so many model and tool calls. Those details come from the rest of the trace.
For the SRE example, the following context could connect trace events to one investigation and explain its resource use:
ContextWhat it helps explainEngineerWhich engineer reviewed or approved an actionAffected serviceWhich production service the agent was investigatingIncidentWhich incident the event belongs toAgent runWhich execution of the SRE agent produced the eventAgentWhich agent performed the work within the runStepWhere the event occurred within the runApprovalWhich approval record belongs to a production actionModelWhich model handled a callToken countsHow many input and output tokens the call usedEstimated costThe approximate cost assigned to the call or stepRetriesHow many times the operation was retriedCache useWhether the operation reused a cached resultRun outcomeWhether the investigation was completed, timed out, or required manual takeover
Several of these fields already have a standard home. OpenTelemetry’s GenAI conventions define spans and attributes for agent and conversation identifiers, the model, and token counts. Estimated cost, retries, cache use, and the approver’s identity aren’t part of that standard yet, so most teams add them as custom attributes alongside it.
In the case of an expensive run, the incident and run identifiers link model usage and cost to one investigation. The agent and step identifiers locate where the resources were consumed. Model, token, cost, retry, and cache information explain the increase. The engineer and approval identifiers connect a production action to its human approval, while the run outcome shows how the investigation ended.
Recording cost and outcome together lets the team compare different designs across a set of investigations. Cost per correctly investigated incident is more useful than token usage alone because it accounts for whether the run produced the right result. One design might use a single agent. Another might divide independent work among several. Useful parallel progress can shorten an investigation, while repeated tool calls and extra handoffs can increase costs.
Coordination becomes harder once an agent runs branches. The system needs clear rules for who retries a failed branch, who owns partial results, who cancels downstream work, and accounts for a sudden increase in cost. A trace can show those decisions only when the system records them.
Parallel branches only pay off when the coordination they require costs less than the time they save. Anthropic’s own multi-agent research system runs on roughly four times the tokens of a single chat interaction, and a full multi-agent run on roughly fifteen times more, with token usage alone accounting for eighty percent of the difference in outcome quality between runs, ahead of tool-call count and model choice. Each additional branch adds checkpoints, partial results, and cancellation paths to manage. Anthropic reports that architecture pays off for broad, independent research tasks and struggles once subagents need to share context or depend heavily on each other’s results.
An agent framework is the software that connects an agent to models and tools. It can also coordinate work between agents. Running many agents at the same time also requires infrastructure for capacity, state, access control, and tracing.
These infrastructure responsibilities can be grouped into four planes:
PlaneResponsibilitySRE exampleExecution and capacityDecides which work starts, waits, or receives priorityLet an active incident proceed while a lower-priority analysis waitsState and durabilitySeparates workflow state and preserves progress across failuresRecover from a worker crash after an approved rollbackTool access and identityControls which agent can use each system and under whose authorityKeep telemetry access read-only and require approval for a production changeObservability and costConnects actions, failures, usage, and outcomes to a runIdentify which branch caused a cost increase or contributed to a wrong diagnosis
Think of these as groups of responsibilities, not four products you need to buy. One platform can cover several groups, or different services can divide the work.
In practice, removing unnecessary seams between systems often produces bigger gains than adding new infrastructure. Every extra system stitched together for retrieval, checkpointing, or governance is another place where identity, auditing, and failure recovery can drift out of sync.
One more distinction matters here. Active agent runs and workers are not the same thing. In the SRE example, many investigations while waiting for a tool or human approval. A runtime pool can use a smaller set of workers to execute whichever steps are ready. That is why capacity planning tracks agent runs separately from workers and database connections.
How do you know whether those four planes are ready? Start with a few concrete questions. Does excess work wait when the model or telemetry capacity is exhausted? Can an investigation recover safely when a worker crashes after an approved rollback? Can one noisy service be throttled while other incidents proceed? Can the team identify which agent called the release system and which engineer approved it? Can it calculate the cost of a correctly investigated incident?
If you cannot find clear answers to these questions, add the missing capacity, state, access, and tracing controls to the production platform before increasing concurrency.
Building one agent surfaces one set of problems. Running many at once surfaces a second set, underneath it, that a single-run prototype never has to answer. Before increasing concurrency, run the checklist above against the platform you actually have.
Multi-Agent Systems at Enterprise Scale was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.