AI Agent Run Leases: Stop Stale Workers From Repeating Side Effects

An agent run can look dead, get claimed by a replacement worker, and then wake up just long enough to send the email you thought the replacement was handling. A queue lease helps, but it is not the whole safety boundary.

The reliable pattern is a run lease plus a fencing token. The lease decides who may work right now. The fencing token lets the system that performs a side effect reject an older worker even if that worker still has credentials and a network connection. This guide shows how to use both for long-running AI agent jobs.

Imagine an agent that prepares and sends a contract summary:

r_42 for 60 seconds. Nothing has to be malicious or badly coded for this to happen. A slow garbage-collection , a network partition, a container reschedule, or a long tool call is enough. Message systems deliberately make work visible again after a visibility timeout so it can recover. AWS explicitly notes that a consumer that runs past that timeout can cause another consumer to process the same message; at-least-once delivery is a normal constraint, not an edge case (AWS guidance).

For a read-only task, duplicate work wastes tokens. For a write, it can duplicate a ticket, overwrite a CRM field, or send a second customer message. Treat a run lease as an ownership protocol, not as a timer around a model call.

A lease is a time-bounded claim on a specific run. Store the minimum state in a durable database or queue, never only in worker memory.

The core rule is simple: a worker may execute only while its lease is valid and its fence is the current fence for that run. Do not let a worker infer continued ownership because it started the task.

Short leases recover quickly but need more renewals. Long leases reduce renewal traffic but leave work unavailable longer after a crash. Start with a duration slightly above the p99 time between durable checkpoints—not the whole run time.

For example, a 45-second lease with a 15-second heartbeat works when the agent saves after each tool call or model stage. An eight-minute uncheckpointed tool call does not become safe because the lease is eight minutes; it merely delays recovery.

The same trade-off appears in managed queues. Azure Service Bus returns a message for redelivery when its lock expires, and supports renewal for active processing (Microsoft documentation). Use that behavior as a delivery primitive, then implement application-level ownership around agent state.

A lease alone protects the scheduler. It does not protect a downstream API from a stale worker.

Each successful claim increments a fence number:

Worker A claims r_42 -> fence 17
Lease expires
Worker B claims r_42 -> fence 18
Worker A wakes up -> must be rejected

Pass run_id and fence to every internal side-effect boundary. That boundary stores the highest fence it has accepted for the run and rejects lower values. This is called fencing because each replacement owner raises a barrier behind it.

The critical comparison belongs where the effect happens, not only in the orchestrator. A stale worker can miss a failed heartbeat, keep an old HTTP client alive, or retry after a process restart. The email service, write proxy, or domain transaction must be able to say: “fence 17 is older than the accepted fence 18; no action.”

Here is a compact PostgreSQL-style claim. The transaction makes availability, ownership, and the fence update one decision.

UPDATE agent_runs
SET state       = 'running',
    lease_owner = :worker_id,
    lease_until = now() + interval '45 seconds',
    fence       = fence + 1,
    updated_at  = now()
WHERE run_id = :run_id
  AND state IN ('queued', 'waiting', 'running')
  AND (lease_until IS NULL OR lease_until < now())
RETURNING run_id, fence, lease_until, checkpoint;

If no row returns, do not work. Another worker owns the run, the run is terminal, or it is not ready. Avoid a read-then-write claim; two workers can both observe an expired timestamp before either writes.

When a worker resumes from a checkpoint, record the checkpoint version and only move forward. Replaying a model call is usually acceptable. Repeating an external write is not. Split planning from committing so the side effect has its own durable operation record.

Renewal must be conditional on the owner and fence you originally acquired. A timer that extends any lease on the row can revive an owner that has already been replaced.

type Lease = { runId: string; workerId: string; fence: number };

async function renewLease(db: Db, lease: Lease): Promise<boolean> {
  const result = await db.query(
    `UPDATE agent_runs
       SET lease_until = now() + interval '45 seconds', updated_at = now()
     WHERE run_id = $1
       AND lease_owner = $2
       AND fence = $3
       AND state = 'running'
       AND lease_until > now()
     RETURNING lease_until`,
    [lease.runId, lease.workerId, lease.fence]
  );
  return result.rowCount === 1;
}

If renewal returns false, immediately enter lost ownership mode:

Do not “try one last time.” Ownership has changed or can no longer be proven. A worker should be pessimistic here. This is also why an agent needs cancellation semantics separate from a UI button: stopping the visible stream does not automatically stop a background action.

Suppose agents send customer email through an internal delivery service. Make the service accept an idempotency key and a fence, then enforce both in one transaction.

INSERT INTO agent_effects (run_id, effect_key, fence, status, payload)
VALUES (:run_id, :effect_key, :fence, 'accepted', :payload)
ON CONFLICT (run_id, effect_key)
DO UPDATE SET
  fence = EXCLUDED.fence,
  payload = EXCLUDED.payload,
  status = 'accepted'
WHERE agent_effects.fence < EXCLUDED.fence
RETURNING fence, status;

This sketch needs a policy choice. For an irreversible send, it is often safer to accept only the first fence that reaches a sent state and reject all later attempts. For a replaceable draft, a newer fence may supersede an earlier accepted draft. Make that choice explicit per effect type.

An external vendor may not understand fences. Put a small internal outbox or action proxy in front of it. The proxy can store (run_id, effect_key, fence), enforce the rule, and use its own idempotency key with the vendor. Never assume the external call failed just because your request timed out; reconcile its result before retrying.

The safest agent loop has short, recoverable segments:

This gives a replacement worker something concrete to resume. It also prevents a long context transcript from becoming the only record of what happened. Keep the checkpoint purposeful: selected inputs, validated intermediate result, operation IDs, and the next permitted step. Do not persist hidden reasoning as a substitute for observable evidence.

Leases are not a replacement for the rest of your reliability design.

Use them together. A dead-letter queue holds a task that needs a decision; a lease and fence make sure that decision is not undermined by an old worker continuing in the background.

Happy-path tests prove very little. Add deterministic failure tests to your agent runtime:

Record the run ID, fence, worker ID, lease expiry, checkpoint version, external operation ID, and final disposition in each trace. Those fields make a production incident answerable without guessing from logs.

You do not need to rebuild every agent at once. Start with the highest-risk write action: email, record update, deployment, payment, or a browser action that changes customer data. Add an action proxy with idempotency and fence checks. Then add run leases to the workers that invoke it. Finally, split long runs into checkpointed stages.

Measure lost-lease events, rejected stale-fence attempts, renewal failures, and time spent in recovery. A nonzero stale-fence rejection count is not automatically a bug; it can be proof that the boundary stopped a duplicate. Investigate trends, especially when a single model or tool stage routinely outlives its lease.

The useful mental model is blunt: leases are permissions with an expiry date. A worker that cannot prove it still holds permission must lose the ability to act. That one rule turns a queue timeout from a duplicate-side-effect incident into a recoverable handoff.

An AI agent run lease is a time-limited ownership claim for one background run. Only the worker holding the unexpired lease may advance the run, renew it, or request a side effect.

A queue lock protects delivery, but a worker can continue after its lock expires or after a network partition. A fencing token lets the downstream action boundary reject that stale worker’s older ownership generation.

Set it slightly above the p99 interval between durable checkpoints, then renew well before expiry. Avoid using one long lease for a whole multi-minute run; split the work into checkpointed stages instead.

Usually the cost of duplicate read-only work is lower, so an idempotency key and lease may be enough. Add fencing when stale output could overwrite newer state, drive a costly downstream job, or create an observable user-facing result.

It should stop issuing new work, cancel interruptible calls, avoid committing local results, and allow the current owner to resume from durable state. Do not retry a side effect from the old worker.

No. If an external provider does not enforce your fence and a request times out after it may have succeeded, you still need an idempotency key and reconciliation. The fence protects your own boundary; reconciliation resolves uncertain outcomes.