# Your AI Agent Checked the State. Someone Changed It Before Execution. > Source: > Published: 2026-10-29 05:36:40+00:00 *Time-of-check-to-time-of-use bugs, stale reads, and concurrent updates are not just distributed-systems problems anymore. When AI agents can execute real actions, they become authorisation and safety problems.* The AI agent did everything it was supposed to do. A customer asked for a refund. The agent retrieved the order, checked its status, verified the refundable balance, evaluated the applicable policy, and determined that the request was allowed. The order was eligible. The amount was correct. The customer owned the order. The agent proceeded. Except that another service had already refunded part of the order. The balance the agent checked was no longer the balance that existed when the refund reached the payment provider. Depending on how the application was designed, the customer might receive too much money, the transaction might fail, or the system might record a result that does not match reality. Nothing was necessarily wrong with the agent's reasoning. Its information was simply no longer current. This is a classic time-of-check-to-time-of-use problem, commonly abbreviated as TOCTOU. A system checks whether an operation is permitted, assumes the relevant conditions will remain unchanged, and then performs the operation after those conditions have changed. Traditional software has dealt with this problem for decades. AI agents make it more interesting because they introduce additional planning steps, tool calls, context changes, retries, and decisions between the initial check and the final action. An agent might inspect an order, call another API, summarise a document, wait for human approval, and then attempt the refund. Every additional step can widen the window in which the underlying state changes. The solution is not to tell the model to check more carefully. Nor is it sufficient to repeat the same check immediately before execution and assume the problem is solved. The real question is whether the system can guarantee that the conditions used to authorise an action still hold when the action takes effect. That requires a combination of concurrency control, version validation, transactional execution, and carefully designed boundaries between the AI agent and the systems it operates. Imagine a conventional application that allows a customer to withdraw money from an account. The account has a balance of ₹10,000. Two requests arrive almost simultaneously, each attempting to withdraw ₹7,000. Both requests read the balance before either withdrawal is committed. ``` Initial balance: ₹10,000 Request A reads balance: ₹10,000 Request B reads balance: ₹10,000 Request A checks: ₹7,000 <= ₹10,000 Request B checks: ₹7,000 <= ₹10,000 Request A withdraws ₹7,000 Request B withdraws ₹7,000 Expected outcome: One withdrawal succeeds. Possible outcome with unsafe implementation: Both succeed. ``` The problem is not that either request calculated incorrectly. Both made a decision using the same earlier state. The application treated a read as though it reserved the state for future use. It did not. Now introduce an AI agent. The agent retrieves the account balance, asks a policy engine whether the withdrawal is permitted, plans the next step, and calls a tool. Perhaps it also waits for approval. The same concurrency problem remains, but the decision window can become substantially longer. An AI agent does not need to be malicious, confused, or poorly prompted to trigger this failure. It can make a perfectly reasonable decision from a snapshot that has already become invalid. This is why I think production AI reliability needs to borrow more heavily from distributed-systems engineering. Once an agent operates on mutable business state, correct reasoning is only part of the problem. The system must also preserve the validity of the assumptions on which that reasoning depends. TOCTOU describes a gap between checking a condition and using the result of that check. Consider this sequence: ``` T1: Read the resource T2: Check permissions and business rules T3: Prepare the action T4: Resource changes T5: Execute the action using the old decision ``` The dangerous interval is between T2 and T5. At T2, the operation might genuinely be permitted. At T5, it might no longer be permitted. The important distinction is between **the validity of a decision at check time and the validity of that decision at execution time**. A check proves something about the state it observed. It does not automatically reserve that state, lock it, or guarantee that another process cannot change it. For example, an agent may check that: Another process can change any of these conditions before the agent acts. TOCTOU is especially important when the action is irreversible or has external consequences. A stale product recommendation is inconvenient. A stale permission check followed by a financial transaction can be a security incident. There is a subtle but important point here: not every state change invalidates every decision. If an unrelated field changes, the original action may remain safe. The engineering task is to identify the exact invariants that must hold when the operation takes effect and enforce them at the appropriate boundary. Traditional applications often execute a relatively short sequence of deterministic operations. AI agents introduce workflows in which the duration and number of intermediate steps are less predictable. Consider an agent handling a refund: ``` 1. Receive customer request. 2. Retrieve order details. 3. Retrieve payment history. 4. Check refund policy. 5. Estimate the refundable balance. 6. Prepare a proposed action. 7. Request human approval. 8. Receive approval. 9. Call the payment API. 10. Record the result. ``` Between steps 2 and 9, another customer-support agent, a scheduled job, a payment webhook, or a human operator may update the same order. The agent may also call additional tools because its plan changes. A tool response may introduce new information, or an execution attempt may time out and require recovery. Every additional operation introduces a possible delay, and every delay gives concurrent processes more opportunities to change the underlying state. I see four characteristics that make this particularly relevant to AI systems. **First, planning is not atomic.** The agent's reasoning process may involve several model calls and tool interactions. The business state is not frozen while the agent reasons. **Second, context can become stale.** A tool response retrieved earlier may remain in the agent's context even after the underlying resource changes. **Third, retries can repeat old assumptions.** A failed execution attempt may be retried using the same plan without checking whether the original conditions still hold. **Fourth, approval adds another boundary.** A human may approve an action based on one version of a resource, while the execution service later encounters a different version. The common mistake is to treat the agent's plan as though it were a transaction. It is not. A plan describes what the agent intends to do. It does not guarantee that the conditions required to do it still exist. Let's look at a more realistic scenario. An order has a refundable balance of ₹10,000. Two support agents independently handle separate requests related to that order. Agent A proposes a refund of ₹7,000. Agent B proposes a refund of ₹6,000. Each agent retrieves the same order state. Each verifies the refundable balance. Each asks the policy engine whether its proposed amount is within the limit. Both requests pass their checks. The problem becomes clear when we look at the interleaving of events. | Time | Agent A | Agent B | |---|---|---| | T1 | Reads balance: ₹10,000 | | | T2 | | Reads balance: ₹10,000 | | T3 | Validates ₹7,000 refund | | | T4 | | Validates ₹6,000 refund | | T5 | Executes ₹7,000 refund | | | T6 | | Executes ₹6,000 refund using stale balance | The combined refund is ₹13,000, even though only ₹10,000 was refundable. Both agents followed the same policy. The policy itself may be perfectly correct. The failure occurred because the system did not enforce the shared invariant at the point where concurrent operations changed the state. The invariant is: [ \text{Total refunds} \leq \text{Refundable limit} ] A pre-execution check cannot guarantee this invariant when another request can change the balance after the check. The application must make the state transition itself conditional on the invariant still holding. That is the difference between checking whether an operation is allowed and making sure that only an allowed operation can commit. A common first response is to repeat the validation immediately before calling the provider. That is better than relying entirely on an old read, but it does not necessarily solve TOCTOU. Consider the following sequence: ``` order = get_order(order_id) if order.refundable_balance_minor >= amount_minor: execute_refund(order_id, amount_minor) ``` The application checks the balance and then calls the provider. What if another worker changes the balance after the condition evaluates but before `execute_refund` takes effect? The same race still exists. Moving the check closer to execution narrows the window. It does not eliminate it. The correct solution depends on where the authoritative state lives and whether the check and state change can be made atomic. If the balance is controlled by your own database, you may be able to enforce the invariant inside a transaction or through a conditional update. If the balance is controlled by an external payment provider, your local transaction cannot automatically make the provider's operation atomic with your own database. You need provider-supported concurrency controls, idempotency, reconciliation, or another design that handles the distributed boundary explicitly. This distinction matters because a system can have excellent local transaction handling and still produce incorrect external effects if it assumes that a local check reserves a remote resource. One useful technique is optimistic concurrency control. Instead of locking a resource throughout a potentially long workflow, the application reads the resource with a version number and checks that the version has not changed when it attempts to update the resource. Suppose the order record contains: ``` { "order_id": "ORD-1042", "refundable_balance_minor": 1000000, "version": 12 } ``` The agent reads version 12 and prepares a refund. Before it commits the update, another worker modifies the order. The version becomes 13. The first worker then attempts an update using version 12. Because the current version is 13, the update fails. The application knows that its decision was based on a stale state. A simplified SQL example looks like this: ``` UPDATE orders SET refundable_balance_minor = refundable_balance_minor - :amount_minor, version = version + 1 WHERE order_id = :order_id AND tenant_id = :tenant_id AND version = :expected_version AND refundable_balance_minor >= :amount_minor; ``` The application checks the affected row count. If exactly one row was updated, the conditional state transition succeeded. If zero rows were updated, the application must not assume the refund was reserved. Possible explanations include: The application should distinguish these outcomes where necessary, without weakening the atomic condition. The important part is that the version check and state update happen in the same database statement. This is not equivalent to reading the version in one query, comparing it in application code, and issuing an unconditional update afterwards. That would reintroduce the race. Optimistic locking works well when conflicts are relatively uncommon, and the application can retry or request a fresh decision when a conflict occurs. It is also useful for AI agents because the agent can perform its planning work without holding a database lock open for the duration of a model call. However, there is an important limitation: optimistic locking only protects the state covered by the version check and conditional operation. It does not automatically protect unrelated records, external services, or resources that do not participate in the same concurrency protocol. When optimistic locking detects a conflict, the application has a choice. It can reject the action, retrieve a fresh state, or attempt to rebuild the operation under a defined policy. What it should not do is blindly retry the same stale plan until the update succeeds. Suppose the agent originally planned to refund ₹7,000 from a ₹10,000 balance. Another process refunds ₹6,000, leaving only ₹4,000. The version check correctly rejects the original ₹7,000 refund. A dangerous retry loop might simply fetch the new version and retry the same amount. The update will continue to fail because the invariant no longer holds. A worse implementation might drop the balance condition and force the operation through. A safer workflow is: ``` Conflict detected | v Discard stale execution decision | v Retrieve the current authoritative state | v Reevaluate business rules and policy | +----> No longer eligible: Reject | +----> Material change: Request reapproval | +----> Still eligible: Build a new action | v Validate and execute safely ``` The distinction between rebuilding an action and retrying an operation is important. A transport retry attempts to deliver the same logical operation again. Replanning creates a new decision based on new information. The two should not share the same assumptions. If the relevant state changes materially after a human approves an action, the application should determine whether the original approval remains valid. A change to the remaining refundable balance might require a new approval because the risk or outcome has changed. An AI agent can help interpret the new state and propose an alternative. It should not be allowed to silently transform a rejected action into a different authorised action. Optimistic locking detects unexpected changes. Database transactions can provide stronger guarantees when the relevant operations are within the same transactional system. Suppose the application maintains the authoritative refundable balance in its own database. It can reserve or deduct the amount as part of a transaction, provided the data model and business process support that design. A simplified example: ``` BEGIN; UPDATE orders SET refundable_balance_minor = refundable_balance_minor - :amount_minor, version = version + 1 WHERE order_id = :order_id AND tenant_id = :tenant_id AND refundable_balance_minor >= :amount_minor; -- Application must verify that exactly one row was updated. -- If not, roll back. INSERT INTO refund_operations ( operation_id, order_id, amount_minor, status ) VALUES ( :operation_id, :order_id, :amount_minor, 'PENDING' ); COMMIT; ``` This example assumes that the application has already validated the action's identity and currency and that the database schema enforces appropriate constraints. It also omits implementation-specific handling for the affected row count. In production, the application must roll back if the conditional update affects zero rows and ensure that the operation record is created only when the reservation succeeds. The transaction protects the local state transition: the balance update and operation record either commit together or roll back together. That gives the application a durable record that the amount has been reserved for a particular logical operation. However, this is not yet the same as completing a refund with an external payment provider. The provider call cannot normally be included in the local database transaction. If the application calls the provider and then the database transaction fails, the provider may have executed the refund even though the local transaction did not commit. Similarly, if the database commits first and the provider call fails, the operation remains reserved but has not completed externally. A production system must explicitly handle this boundary. AI agents frequently operate through external APIs: payment providers, CRM systems, ticketing platforms, cloud infrastructure, email services, and internal microservices. A local database transaction cannot guarantee atomic execution across all these systems. ``` 1. Reserve refund in the local database. 2. Commit local transaction. 3. Call the payment provider. 4. Provider executes refund. 5. Network response is lost. ``` The application does not know whether the refund succeeded merely because it did not receive a response. Issuing a second refund request without a recovery strategy may create a duplicate effect. This is where durable execution records, idempotency, and reconciliation become essential. A robust design typically creates a durable operation record with a unique operation ID. A worker processes that operation, uses a stable provider idempotency key when supported, and records the outcome. If the worker crashes, another worker can resume processing based on the durable state instead of inventing a new operation. If the outcome is uncertain, the system reconciles against the provider before attempting an action that could duplicate the effect. The key states might include: `PENDING`: the operation has been durably created.` IN_PROGRESS`: a worker is processing it.` SUCCEEDED`: the provider's successful outcome is confirmed.` FAILED`: a definitive failure is known.` OUTCOME_UNKNOWN`: the request may have succeeded, but confirmation is missing.`RECONCILIATION_REQUIRED`: automated recovery cannot safely determine the result. The precise state machine depends on the provider and business process. Some failures are retryable; others require manual intervention. Idempotency keys are particularly important, but they are not magic. Their scope, retention, request-matching rules, and behavior after timeouts depend on the provider's contract. The application must use them according to those semantics and retain enough information to recover safely. A stable key should identify the same logical operation across retries. Generating a new key for every retry can defeat the intended protection. One way to coordinate local state changes with asynchronous external execution is the transactional outbox pattern. Instead of updating the business record and then attempting to publish a message separately, the application commits the state change and an outbox event in the same database transaction. For example: ``` BEGIN; UPDATE orders SET refundable_balance_minor = refundable_balance_minor - :amount_minor, version = version + 1 WHERE order_id = :order_id AND tenant_id = :tenant_id AND refundable_balance_minor >= :amount_minor; -- Verify that exactly one row was updated. INSERT INTO refund_operations ( operation_id, order_id, amount_minor, status ) VALUES ( :operation_id, :order_id, :amount_minor, 'PENDING' ); INSERT INTO outbox_events ( event_id, event_type, aggregate_id, payload ) VALUES ( :event_id, 'RefundRequested', :order_id, :event_payload ); COMMIT; ``` A separate worker reads the outbox and sends the request to the payment provider. The advantage is that the business state and the intention to perform the external operation are recorded together. If the application crashes after the transaction commits, the outbox event remains available for processing. The worker must still handle duplicate delivery, provider idempotency, and uncertain outcomes. An outbox does not guarantee exactly-once delivery to an external API. It helps make the handoff durable and recoverable. This pattern is especially useful for AI agents because it separates the model's planning lifecycle from the execution lifecycle. The model can propose an action, the application can validate and commit an authorized operation, and a worker can execute it reliably without requiring the model to remain active throughout the transaction. TOCTOU is not limited to business data. Authorisation itself can become stale. Suppose an AI agent checks that a user has permission to delete a production resource. The policy engine returns `ALLOW`. Before the delete operation executes, an administrator revokes that user's access. If the application relies solely on the earlier decision, it may execute an action that is no longer authorised. This is a time-of-check-to-time-of-use problem involving permissions rather than balances. A safer execution path checks the relevant authorisation conditions immediately before the action takes effect. Depending on the system, this may involve: A policy version can be useful if the application defines a clear invalidation model. For example, an authorisation decision might be associated with version 42 of a policy. If the relevant policy changes to version 43, execution can require a fresh decision. However, version checks alone do not solve every authorisation race. The resource-owning service must enforce the actual permission, and the system must define how revocation interacts with in-flight operations. For highly sensitive actions, the application may need a linearizable authorization check, a short-lived capability bound to a specific action, or a design in which the resource owner validates authorization as part of the state-changing operation. The objective is to avoid treating an old `ALLOW` response as a permanent permission. There is a temptation to solve stale state by asking the model to retrieve everything again before every action. That may reduce some failures, but it is not a sufficient concurrency strategy. The agent cannot guarantee that the state will remain unchanged after the final retrieval. It also cannot reliably coordinate concurrent updates across services merely by reasoning about them. Instead, give the agent a clear execution contract. The agent can: The execution service must: This separation is important because a model-generated plan is not a lock, a reservation, or a transaction. A useful response protocol might distinguish: ``` ALLOW Action is currently eligible for execution. CONFLICT Relevant state changed; refresh and reevaluate. DENY Current policy prohibits the action. APPROVAL_REQUIRED Human authorisation is required before execution. OUTCOME_UNKNOWN Execution may have occurred; reconcile before retrying. ``` These outcomes let the agent respond appropriately without giving it authority to override the decision. A `CONFLICT` response should not automatically trigger repeated retries of the same payload. The application should determine whether a new plan is permissible, whether human reapproval is required, or whether the action must be abandoned. This approach makes the agent a participant in a controlled workflow rather than the component responsible for maintaining consistency across the entire system. Concurrency defects are difficult to catch with ordinary happy-path tests because they depend on timing and interleaving. A test that executes one refund at a time may never expose the bug. The test suite should deliberately create competing operations and force changes between checking and execution. Here are several scenarios worth testing. **Two refunds against one balance** Start two requests simultaneously, each of which would be valid against the initial balance but invalid when combined. Verify that the total successful refund amount never exceeds the permitted limit. **State changes after approval** Approve an action, change a relevant resource field, and then attempt execution. Verify that the old action is rejected or subjected to the explicitly defined revalidation process. **Permission revocation** Grant access, obtain an initial authorisation decision, revoke access, and then attempt the action. Verify the intended revocation semantics. **Concurrent version updates** Have two workers read the same resource version and attempt updates. Verify that at most one update using that version succeeds and that the losing worker handles the conflict correctly. **Provider timeout after execution** Simulate a provider that performs the operation but drops the response. Verify that recovery does not create a duplicate effect. **Worker crash after reservation** Commit a local reservation and operation record, then crash before sending the provider request. Verify that another worker can recover the operation safely. **Outbox duplicate delivery** Deliver the same event more than once. Verify that the consumer and provider integration handles duplicates according to the operation's idempotency rules. **Stale model context** Let the agent retrieve a resource, change the underlying state, and then submit the old plan. Verify that the execution service—not the model—detects the conflict. For database-backed workflows, use concurrent integration tests against the actual database engine and isolation configuration. Mocked tests are useful for individual components, but they cannot prove that the real conditional update, transaction, or locking behaviour is correct. Property-based testing can also help. Generate competing operations and verify invariants such as: [ \text{Successful refunds} \leq \text{Authorized refundable limit} ] The test should examine committed outcomes, not just the decisions made by the agents. Before deploying an AI agent that modifies shared or externally managed state, ask the following questions. If the system cannot answer these questions, it may be relying on the assumption that the state remains stable while the agent thinks. That assumption is rarely safe in a distributed system. An AI agent can make the correct decision and still cause an incorrect outcome. It can inspect a valid resource, evaluate a legitimate policy, and construct an appropriate action. Another process can then change the resource before execution. The agent's reasoning was correct for the state it observed. The system failed to preserve the conditions required for that decision to remain valid. This is why reliable AI agents need more than better prompts, stronger models, and additional validation calls. They need the same concurrency guarantees that reliable transactional software has always required. Use optimistic locking to detect stale versions. Use conditional updates and transactions to enforce invariants within a database. Use durable operation records, idempotency, and reconciliation to manage external effects. Revalidate authorisation at the appropriate execution boundary. Treat conflicts as reasons to reevaluate, not invitations to retry blindly. And above all, separate the agent's intention from the system's guarantee that the intended action can safely take effect. The model can decide what it wants to do next. It cannot guarantee that the world still looks the way it did when that decision was made. **A state check tells you what was true. Concurrency control determines whether it is still safe to act on that truth.**