AI Agent Run Leases: Stop Stale Workers From Repeating Side Effects

A developer outlined a pattern for preventing stale AI agent workers from repeating side effects by combining a durable run lease with a monotonically increasing fencing token. The lease, stored in a database or queue with fields like lease_owner, lease_until and fence, decides which worker may act at a given moment, while the fence is checked at every side-effect boundary so an expired worker's writes are rejected even if it still holds credentials. The writeup recommends sizing leases just above the p99 time between durable checkpoints and passing run_id and fence to downstream services such as email or write proxies.

An agent run can look dead, get claimed by a replacement worker, and then wake up just long enough to send the email you thought the replacement was handling. A queue lease helps, but it is not the whole safety boundary. The reliable pattern is a run lease plus a fencing token . The lease decides who may work right now . The fencing token lets the system that performs a side effect reject an older worker even if that worker still has credentials and a network connection. This guide shows how to use both for long-running AI agent jobs. Imagine an agent that prepares and sends a contract summary: r 42 for 60 seconds. Nothing has to be malicious or badly coded for this to happen. A slow garbage-collection pause, a network partition, a container reschedule, or a long tool call is enough. Message systems deliberately make work visible again after a visibility timeout so it can recover. AWS explicitly notes that a consumer that runs past that timeout can cause another consumer to process the same message; at-least-once delivery is a normal constraint, not an edge case AWS guidance https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/designing-for-outage-recovery-scenarios.html . For a read-only task, duplicate work wastes tokens. For a write, it can duplicate a ticket, overwrite a CRM field, or send a second customer message. Treat a run lease as an ownership protocol, not as a timer around a model call. A lease is a time-bounded claim on a specific run. Store the minimum state in a durable database or queue, never only in worker memory. | Field | Why it matters | |---|---| | run id | Stable idempotency and audit key for the whole task | | state | Queued, running, waiting, succeeded, failed, or cancelled | | lease owner | Identifies the process that currently owns the run | | lease until | Allows recovery when that process disappears | | fence | Monotonically increasing generation of ownership | | checkpoint | Durable progress that a replacement can safely resume | | side effect status | Evidence for whether an external action was attempted or confirmed | The core rule is simple: a worker may execute only while its lease is valid and its fence is the current fence for that run. Do not let a worker infer continued ownership because it started the task. Short leases recover quickly but need more renewals. Long leases reduce renewal traffic but leave work unavailable longer after a crash. Start with a duration slightly above the p99 time between durable checkpoints—not the whole run time. For example, a 45-second lease with a 15-second heartbeat works when the agent saves after each tool call or model stage. An eight-minute uncheckpointed tool call does not become safe because the lease is eight minutes; it merely delays recovery. The same trade-off appears in managed queues. Azure Service Bus returns a message for redelivery when its lock expires, and supports renewal for active processing Microsoft documentation https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement . Use that behavior as a delivery primitive, then implement application-level ownership around agent state. A lease alone protects the scheduler. It does not protect a downstream API from a stale worker. Each successful claim increments a fence number: php Worker A claims r 42 - fence 17 Lease expires Worker B claims r 42 - fence 18 Worker A wakes up - must be rejected Pass run id and fence to every internal side-effect boundary. That boundary stores the highest fence it has accepted for the run and rejects lower values. This is called fencing because each replacement owner raises a barrier behind it. The critical comparison belongs where the effect happens, not only in the orchestrator. A stale worker can miss a failed heartbeat, keep an old HTTP client alive, or retry after a process restart. The email service, write proxy, or domain transaction must be able to say: “fence 17 is older than the accepted fence 18; no action.” Here is a compact PostgreSQL-style claim. The transaction makes availability, ownership, and the fence update one decision. UPDATE agent runs SET state = 'running', lease owner = :worker id, lease until = now + interval '45 seconds', fence = fence + 1, updated at = now WHERE run id = :run id AND state IN 'queued', 'waiting', 'running' AND lease until IS NULL OR lease until < now RETURNING run id, fence, lease until, checkpoint; If no row returns, do not work. Another worker owns the run, the run is terminal, or it is not ready. Avoid a read-then-write claim; two workers can both observe an expired timestamp before either writes. When a worker resumes from a checkpoint, record the checkpoint version and only move forward. Replaying a model call is usually acceptable. Repeating an external write is not. Split planning from committing so the side effect has its own durable operation record. Renewal must be conditional on the owner and fence you originally acquired. A timer that extends any lease on the row can revive an owner that has already been replaced. type Lease = { runId: string; workerId: string; fence: number }; async function renewLease db: Db, lease: Lease : Promise<boolean { const result = await db.query UPDATE agent runs SET lease until = now + interval '45 seconds', updated at = now WHERE run id = $1 AND lease owner = $2 AND fence = $3 AND state = 'running' AND lease until now RETURNING lease until , lease.runId, lease.workerId, lease.fence ; return result.rowCount === 1; } If renewal returns false, immediately enter lost ownership mode: Do not “try one last time.” Ownership has changed or can no longer be proven. A worker should be pessimistic here. This is also why an agent needs cancellation semantics separate from a UI button: stopping the visible stream does not automatically stop a background action. Suppose agents send customer email through an internal delivery service. Make the service accept an idempotency key and a fence, then enforce both in one transaction. INSERT INTO agent effects run id, effect key, fence, status, payload VALUES :run id, :effect key, :fence, 'accepted', :payload ON CONFLICT run id, effect key DO UPDATE SET fence = EXCLUDED.fence, payload = EXCLUDED.payload, status = 'accepted' WHERE agent effects.fence < EXCLUDED.fence RETURNING fence, status; This sketch needs a policy choice. For an irreversible send, it is often safer to accept only the first fence that reaches a sent state and reject all later attempts. For a replaceable draft, a newer fence may supersede an earlier accepted draft. Make that choice explicit per effect type. An external vendor may not understand fences. Put a small internal outbox or action proxy in front of it. The proxy can store run id, effect key, fence , enforce the rule, and use its own idempotency key with the vendor. Never assume the external call failed just because your request timed out; reconcile its result before retrying. The safest agent loop has short, recoverable segments: This gives a replacement worker something concrete to resume. It also prevents a long context transcript from becoming the only record of what happened. Keep the checkpoint purposeful: selected inputs, validated intermediate result, operation IDs, and the next permitted step. Do not persist hidden reasoning as a substitute for observable evidence. Leases are not a replacement for the rest of your reliability design. | Control | Solves | Does not solve | |---|---|---| | Idempotency key | Repeated request to the same effect | An old owner modifying a newer draft | | Run lease | Who may currently work | Stale network client reaching a downstream service | | Fencing token | Old owner acting after replacement | A vendor action with unknown outcome | | Reconciliation | Timeout after possible completion | Two workers doing expensive read-only work | | Dead-letter queue | Repeated or ambiguous failure | Safe replay by itself | Use them together. A dead-letter queue holds a task that needs a decision; a lease and fence make sure that decision is not undermined by an old worker continuing in the background. Happy-path tests prove very little. Add deterministic failure tests to your agent runtime: Record the run ID, fence, worker ID, lease expiry, checkpoint version, external operation ID, and final disposition in each trace. Those fields make a production incident answerable without guessing from logs. You do not need to rebuild every agent at once. Start with the highest-risk write action: email, record update, deployment, payment, or a browser action that changes customer data. Add an action proxy with idempotency and fence checks. Then add run leases to the workers that invoke it. Finally, split long runs into checkpointed stages. Measure lost-lease events, rejected stale-fence attempts, renewal failures, and time spent in recovery. A nonzero stale-fence rejection count is not automatically a bug; it can be proof that the boundary stopped a duplicate. Investigate trends, especially when a single model or tool stage routinely outlives its lease. The useful mental model is blunt: leases are permissions with an expiry date. A worker that cannot prove it still holds permission must lose the ability to act. That one rule turns a queue timeout from a duplicate-side-effect incident into a recoverable handoff. An AI agent run lease is a time-limited ownership claim for one background run. Only the worker holding the unexpired lease may advance the run, renew it, or request a side effect. A queue lock protects delivery, but a worker can continue after its lock expires or after a network partition. A fencing token lets the downstream action boundary reject that stale worker’s older ownership generation. Set it slightly above the p99 interval between durable checkpoints, then renew well before expiry. Avoid using one long lease for a whole multi-minute run; split the work into checkpointed stages instead. Usually the cost of duplicate read-only work is lower, so an idempotency key and lease may be enough. Add fencing when stale output could overwrite newer state, drive a costly downstream job, or create an observable user-facing result. It should stop issuing new work, cancel interruptible calls, avoid committing local results, and allow the current owner to resume from durable state. Do not retry a side effect from the old worker. No. If an external provider does not enforce your fence and a request times out after it may have succeeded, you still need an idempotency key and reconciliation. The fence protects your own boundary; reconciliation resolves uncertain outcomes.