A user clicks “Run agent”, your backend receives a normal HTTP request, and the agent starts doing what agents do: calling tools, reading documents, querying APIs, waiting for a human approval, retrying a flaky search, and generating a long report.
Two minutes later, your load balancer returns 504 Gateway Timeout.
The user sees an error.
The agent, depending on where it is running, may still be alive. It may still be spending tokens. It may have already sent an email, created a ticket, or updated a record. Then the user clicks retry. Now you may have two agents doing overlapping work, with no clean way to know which one is authoritative.
That is the core problem:
An HTTP request is a short-lived delivery mechanism.
An AI agent run is often a long-lived state machine.
When the agent outlives the request, the request/response model stops being the right abstraction. You need run IDs, durable state, queues, idempotency, cancellation, progress events, and a way to resume after failure.
This article is about what breaks when an AI agent runs longer than the HTTP request that started it — and how to design systems that survive that reality.
202 Accepted, and execute the work asynchronously.
A typical HTTP request has a simple lifecycle:
client connects
client sends request
server processes
server responds
connection closes
An AI agent run has a much messier lifecycle:
queued
started
planning
waiting for model
waiting for tool
waiting for approval
retrying
streaming progress
completed / failed / cancelled
Those two lifecycles do not line up.
HTTP assumes the server can finish quickly enough for the client to stay interested. Agent work often cannot make that promise. The agent may be waiting on:
The first mistake is pretending the HTTP request is the agent. It is not. The request is only the trigger.
Once you accept that, the architecture changes.
Scenario:
Your frontend calls POST /agent-runs and waits for the final answer. The agent takes longer than your gateway timeout. The client receives a 504, but the backend worker continues executing.
Why it matters:
Now you have split-brain behavior. The user thinks the operation failed. The system may still be doing work, consuming tokens, calling APIs, and mutating state.
This is especially dangerous when the agent has side effects. A timed-out read-only query is annoying. A timed-out agent that can send messages, create orders, or delete records is an incident waiting to happen.
Solution:
Do not run long-lived agents synchronously inside the HTTP request. Create a run record, return 202 Accepted, and execute the agent asynchronously.
import express from "express";
import crypto from "node:crypto";
const app = express();
app.use(express.json());
app.post("/agent-runs", async (req, res) => {
const idempotencyKey = req.get("Idempotency-Key") ?? crypto.randomUUID();
const run = await startAgentRunOnce({
idempotencyKey,
input: req.body,
});
res
.status(202)
.setHeader("Location", `/agent-runs/${run.id}`)
.json({
runId: run.id,
status: run.status,
statusUrl: `/agent-runs/${run.id}`,
eventsUrl: `/agent-runs/${run.id}/events`,
});
});
The client receives a run ID immediately. The actual agent work happens in a worker, queue, or workflow engine.
Why this works:
The HTTP request is no longer responsible for completing the agent run. It is only responsible for starting it safely.
The run becomes a first-class resource:
POST /agent-runs → start
GET /agent-runs/:id → status
GET /agent-runs/:id/events → progress stream
POST /agent-runs/:id/cancel → cancel
💡 Practical note:
202 Accepted is the honest response for work that has been received but not completed. It is not a hack. It is the correct HTTP semantics.
Scenario:
The user does not see a result, so they click “Run” again. Or the mobile app automatically retries after a network drop. Now two agent runs are executing for the same intent.
Why it matters:
Agents are not pure functions. If the agent can call tools, retrying from scratch can produce different decisions and duplicate side effects.
A normal API retry problem becomes much worse with agents because the retry may:
Solution:
Require an idempotency key for agent creation.
A simple Postgres pattern:
CREATE TABLE agent_runs (
id uuid PRIMARY KEY,
idempotency_key text NOT NULL UNIQUE,
input jsonb NOT NULL,
status text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
Then insert safely:
INSERT INTO agent_runs (id, idempotency_key, input, status)
VALUES ($1, $2, $3, 'queued')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id;
If no row is returned, fetch the existing run:
SELECT id, status
FROM agent_runs
WHERE idempotency_key = $1;
In application code:
async function startAgentRunOnce(params: {
idempotencyKey: string;
input: unknown;
}) {
const runId = crypto.randomUUID();
const inserted = await db.query(
`
INSERT INTO agent_runs (id, idempotency_key, input, status)
VALUES ($1, $2, $3, 'queued')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id, status
`,
[runId, params.idempotencyKey, JSON.stringify(params.input)]
);
if (inserted.rows.length > 0) {
return inserted.rows[0];
}
const existing = await db.query(
`
SELECT id, status
FROM agent_runs
WHERE idempotency_key = $1
`,
[params.idempotencyKey]
);
return existing.rows[0];
}
Why this works:
The idempotency key becomes the client’s declaration of intent. If the same intent is submitted again, the system returns the existing run instead of creating a new one.
⚠️ Gotcha:
Idempotency keys need a meaningful scope. A key like run-agent is useless. Use something tied to the user, tenant, action, and input hash, or let the client generate a UUID per logical attempt.
Scenario:
Your agent keeps its conversation history, current plan, and tool-call progress in process memory. Then you deploy a new version, a container scales in, or the machine restarts.
The run disappears.
Why it matters:
Long-running agents need durable state. If the process can die before the run finishes, the state must live somewhere else.
This is not just about crashes. In production, processes restart all the time:
If your agent state is only in memory, you do not have a long-running agent. You have a fragile process with amnesia.
Solution:
Persist the agent run state in a durable store. The exact store can be Postgres, Redis, DynamoDB, or a workflow engine, but the state model should be explicit.
A useful run state shape:
type AgentRunStatus =
| "queued"
| "running"
| "waiting_for_approval"
| "completed"
| "failed"
| "cancelled";
interface AgentRunState {
runId: string;
status: AgentRunStatus;
input: unknown;
cursor?: string;
messages: AgentMessage[];
pendingToolCall?: ToolCall;
artifacts: Artifact[];
error?: string;
updatedAt: string;
}
More important than the exact fields is the discipline:
Why this works:
The worker becomes replaceable. The run state survives independently of any one process.
This also makes debugging much easier. When someone asks, “What is the agent doing right now?” you can answer from stored state instead of guessing from logs.
🚨 Production warning:
If you cannot answer “What step is this agent on?” without attaching a debugger, your agent is not production-ready.
Scenario:
You use Server-Sent Events or WebSockets to stream agent progress. The connection drops. The user reloads the page. Now the frontend has missed events and has no reliable way to recover.
Why it matters:
Streaming is excellent for user experience. It is terrible as the only record of what happened.
A stream is ephemeral. It tells you what is happening now, or what happened recently, but it does not by itself answer:
Solution:
Keep durable run state as the source of truth. Use streaming as a notification layer on top of that state.
A resilient SSE endpoint should support reconnection:
app.get("/agent-runs/:id/events", async (req, res) => {
const runId = req.params.id;
res.setHeader("Content-Type", "text/event-stream");
res.setHeader("Cache-Control", "no-cache");
res.setHeader("Connection", "keep-alive");
res.flushHeaders();
const lastEventId = Number(req.headers["last-event-id"] ?? "0");
const missedEvents = await eventStore.eventsSince(runId, lastEventId);
for (const event of missedEvents) {
sendEvent(res, event);
}
const unsubscribe = eventStore.subscribe(runId, (event) => {
sendEvent(res, event);
});
const heartbeat = setInterval(() => {
res.write(": ping\n\n");
}, 15000);
req.on("close", () => {
clearInterval(heartbeat);
unsubscribe();
});
});
function sendEvent(
res: express.Response,
event: { id: number; type: string; payload: unknown }
) {
res.write(`id: ${event.id}\n`);
res.write(`event: ${event.type}\n`);
res.write(`data: ${JSON.stringify(event.payload)}\n\n`);
}
The important part is Last-Event-ID. When the client reconnects, the server can replay missed events instead of pretending the connection never dropped.
Why this works:
The client can lose the stream without losing the run. The UI becomes a projection of durable state, not the only place where progress exists.
💡 Practical note:
Heartbeats are not optional. Many proxies and load balancers close idle connections quietly. A periodic comment line like : ping helps keep the stream alive.
Scenario:
Your agent calls three tools: search, create ticket, send email. The email tool succeeds, then the run crashes before the state is saved. The worker restarts and retries. Now the email may be sent again.
Why it matters:
Agents with tools are not just inference loops. They are distributed systems that take actions.
Once an agent can affect external systems, you have the usual distributed-workflow problems:
But agents make these problems harder because the sequence of actions may not be deterministic.
Solution:
Make tool execution idempotent wherever possible. Give each meaningful tool call a stable idempotency key derived from the run and the logical operation.
async function sendInvoiceEmail(run: AgentRunState, invoiceId: string) {
const toolCallKey = `${run.runId}:send_invoice_email:${invoiceId}`;
const existing = await externalCallStore.find(toolCallKey);
if (existing) {
return existing.result;
}
const result = await emailClient.send({
idempotencyKey: toolCallKey,
to: run.input.customerEmail,
subject: `Invoice ${invoiceId}`,
template: "invoice",
});
await externalCallStore.record(toolCallKey, result);
return result;
}
If the downstream API does not support idempotency keys, you still need local deduplication:
const alreadyPerformed = await sideEffectLog.exists(toolCallKey);
if (alreadyPerformed) {
return sideEffectLog.resultFor(toolCallKey);
}
For destructive or irreversible operations, consider requiring explicit approval or using a two-phase pattern:
propose action
store proposal
wait for approval
execute once
record result
Why this works:
You are separating “the agent decided to do something” from “the system actually did it.” That separation gives you a place to enforce safety, retries, and auditing.
🧠 The important part:
If a tool call can happen twice, the agent run is not safe to retry unless that tool call is idempotent or guarded.
Scenario:
The user clicks “Cancel” while the agent is running. Your API updates a database row. The agent, currently waiting on a model call or external tool, has no idea.
Why it matters:
Cancellation is easy as a UI concept and hard as an execution concept. If the agent does not check for cancellation, it keeps doing work. If it checks too late, it may perform side effects after the user asked it to stop.
Solution:
Treat cancellation as a cooperative protocol.
First, expose a cancellation endpoint:
app.post("/agent-runs/:id/cancel", async (req, res) => {
const runId = req.params.id;
await agentRunStore.requestCancellation(runId);
res.status(202).json({
runId,
status: "cancel_requested",
});
});
Then make the worker check cancellation at safe boundaries:
async function executeAgentRun(runId: string, signal: AbortSignal) {
while (!signal.aborted) {
const run = await agentRunStore.get(runId);
if (!run) return;
if (run.status === "cancel_requested") {
await agentRunStore.markCancelled(runId, {
reason: "user_requested",
});
return;
}
const nextStep = await planNextStep(run);
if (!nextStep) {
await agentRunStore.markCompleted(runId);
return;
}
await executeStep(runId, nextStep, signal);
}
}
The critical detail is where you check.
Good cancellation points:
Bad cancellation strategy:
Why this works:
Cancellation becomes part of the run lifecycle instead of an afterthought.
⚠️ Gotcha:
Cancellation does not automatically undo side effects. If the agent already sent the email, cancellation may only mean “stop doing more work.” Your system needs to know the difference.
Scenario:
Your agent needs approval before sending a high-risk email. The approval may come in ten seconds, ten hours, or three days.
Now your agent is not merely long-running. It is suspended.
Why it matters:
Many agent systems are designed for “slow API calls,” not for “ until a human responds.” These are different problems.
A slow API call can be handled with timeouts and retries. A human requires:
If you keep the agent process alive while waiting for a human, you are wasting resources. If you do not persist the , you lose the run.
Solution:
Model waiting states explicitly.
await agentRunStore.update(runId, {
status: "waiting_for_approval",
pendingAction: {
type: "send_customer_email",
payload: emailDraft,
requestedAt: new Date().toISOString(),
expiresAt: new Date(Date.now() + 48 * 60 * 60 * 1000).toISOString(),
},
});
Then resume when the approval arrives:
app.post("/agent-runs/:id/approvals", async (req, res) => {
const runId = req.params.id;
const { approved, approver } = req.body;
await agentRunStore.resolveApproval(runId, {
approved,
approver,
resolvedAt: new Date().toISOString(),
});
await queue.enqueue("agent.resume", { runId });
res.status(202).json({ runId, status: "resuming" });
});
This is where durable execution engines start to make sense. They are designed for workflows that can wait for long periods without keeping a process alive.
Why this works:
The agent run becomes a workflow with explicit and resume points. That is far more honest than pretending a human approval is just another fast function call.
💡 Practical note:
If your agent can wait for humans, design expiration early. A pending action that lives forever becomes a security and compliance problem.
Scenario:
The agent starts with the user’s access token. Twenty minutes later, it needs to call another API. The token has expired.
Why it matters:
User sessions and agent lifetimes do not naturally align.
A user may:
Meanwhile, the agent may still be running.
If you casually pass the original request token into a long-lived background worker, you create both security and reliability problems.
Solution:
Decide explicitly what identity the agent runs under.
Common patterns:
Useful when the run is brief and the token lifetime is sufficient.
Risk: the token expires mid-run.
Useful when the agent acts on behalf of the user for longer periods.
Requirements:
Useful when the agent performs system-level work.
Useful when the agent needs approval for high-risk actions.
Example:
agent requests permission to refund payment
system s run
user re-authenticates
system resumes run with fresh approval
A practical authorization check before a sensitive tool call:
async function authorizeToolCall(run: AgentRunState, tool: ToolDefinition) {
const policy = await policyStore.forRun(run.runId);
if (!policy.allows(tool.name)) {
throw new Error(`Tool ${tool.name} is not allowed for this run`);
}
if (tool.requiresRecentUserApproval) {
const approval = await approvalStore.latest(run.runId, tool.name);
if (!approval || approval.approvedAt < minutesAgo(5)) {
await agentRunStore.update(run.runId, {
status: "waiting_for_approval",
pendingAction: {
type: tool.name,
payload: tool.input,
},
});
throw new ToolCalld("Recent approval required");
}
}
}
Why this works:
You stop assuming that the original HTTP request’s auth context is valid forever.
🚨 Production warning:
Do not solve token expiration by giving agents broad admin tokens “just to keep things moving.” That turns every long-running agent into a privilege-escalation risk.
Not every agent needs the same architecture. The right choice depends on how long the agent can run, whether it has side effects, and whether humans are involved.
| Approach | Best for | Complexity | Weakness |
|---|---|---|---|
| Synchronous HTTP | Very fast, read-only agent calls | Low | Breaks as soon as work is slow or unreliable |
| Async run + polling | Simple background agent jobs | Medium | Polling can be inefficient; needs run store |
| SSE/WebSocket progress | Interactive UX with live updates | Medium | Connection loss handling is required |
| Queue workers | Scalable background execution | Medium | Retry and state discipline needed |
| Durable workflow engine | Multi-step, long-lived, human-in-loop agents | High | More operational complexity |
Use synchronous request/response when:
POST /summarize-text
Use async runs when:
POST /agent-runs
GET /agent-runs/:id
Use SSE or WebSockets when:
But keep durable state underneath.
Use a durable workflow engine when:
This is the category where “agent” starts looking less like a chat endpoint and more like business process automation.
If I were shipping an AI agent feature that could run longer than a normal HTTP request, I would use this shape:
Client
↓
API layer
↓
Run store
↓
Queue / workflow engine
↓
Agent worker
↓
Tool execution layer
↓
Event store / notifications
Responsible for:
Stores:
The most important design decision is this:
The HTTP request starts and observes the run.
It does not own the run.
That one mental shift prevents a large class of production pain.
Before letting an agent run longer than the HTTP request that started it, I’d want these boxes checked.
The deeper truth is that long-running AI agents are not just “slower APIs.” They are workflows with non-deterministic planning, external side effects, and user expectations.
HTTP can trigger them. HTTP can report on them. HTTP can stream progress from them.
But once the agent can outlive the request, HTTP should not be the container for the entire execution.
The request is the doorway.
The agent run is the process.
Design them separately, and the system becomes far easier to operate. Design them as the same thing, and every timeout, retry, deploy, and disconnected browser becomes a potential corruption of the run.