Your AI agent learned a better way. Who approved the change?

A growing class of AI agent systems can silently widen their own authority when case histories are consolidated into reusable memory, according to an August preprint titled "When Memory Becomes Authority" by Qiuyang Zhan and colleagues, which found that memories could retain a claim while losing the source-authority distinctions needed for a later action, and that preserving authority metadata reduced authorization errors in the tested pipeline. The paper's controlled benchmark does not measure how often this occurs in deployed enterprises. The analysis cites Salesforce's September beta release of Agent Optimizer, an October 1 AWS implementation guide for building agent memory with NVIDIA NeMo Agent Toolkit and Amazon S3 Vectors, and Microsoft memory-safety guidance recommending that retrieved memories be treated as candidate context rather than authoritative truth.

Imagine a support team working through complaints after a service outage. Its refund policy normally requires an eligibility review. For this incident, an authorized manager approves an exception: customers on the verified outage list may receive up to $100 without individual review, through Friday. An agent processes those cases successfully. Later, the system consolidates those case histories into a reusable memory: small, immediate refunds resolve service complaints quickly. The summary preserves the outcome but loses the outage restriction and expiration date. The following week, the agent retrieves that guidance for a similar complaint from a customer who was not affected by the outage. It issues a refund without review. The payment service checks the agent’s credentials and the dollar limit, but does not check whether the customer qualifies for an active exception. No attacker is needed in this hypothetical. The case may close faster, yet the agent has begun applying a temporary exception beyond its authorized scope. Has the company authorized the agent to find a better way to handle refunds, or to change who gets one without review? Those are different decisions. In September, Salesforce introduced Agent Optimizer https://www.salesforce.com/blog/agent-optimizer/ in beta to analyze production behavior, propose improvements and build and test changes. Customers choose review points, including how far the system may proceed toward staging changes for deployment. An October 1 AWS implementation guide https://aws.amazon.com/blogs/machine-learning/build-agent-memory-with-nvidia-nemo-agent-toolkit-and-amazon-s3-vectors/ illustrates a different mechanism: consolidating individual observations into generalized memories that agents can retrieve and share. Its investment-research example shows how experience can become persistent context without retraining a model. Updating stored memory changes information available to later runs. Editing instructions or workflows changes how the application directs the work. Training changes the model’s parameters. All can alter behavior, but teams inspect and approve them in different ways. A reviewed workflow revision has a release step. A new memory may enter through routine data processing and influence the next customer interaction. The code, model and tool permissions can remain unchanged while the guidance shaping decisions changes. In the August preprint When Memory Becomes Authority https://arxiv.org/html/2608.01679v2 , Qiuyang Zhan and colleagues examine whether consolidation preserves who supplied information and what that source is allowed to authorize. In their controlled benchmark, memories could retain a claim while losing the source-authority distinctions needed for a later action. Preserving authority metadata reduced authorization errors in the pipeline they tested. The study does not measure how often this happens in deployed enterprises. The refund example concerns a related boundary: permission for particular customers and dates must remain limited to those conditions. Success with last week’s cases cannot establish that other customers qualify. Microsoft’s memory-safety guidance https://learn.microsoft.com/en-us/security/zero-trust/sfi/manage-agentic-memory-safety treats persistent memory as a layer that can shape behavior and recommends treating retrieved memories as candidate context, not authoritative truth. Knowing what happened before does not establish what is allowed now. Keep the exception’s scope, approving authority, source and expiration in a controlled record. Reusable guidance should retain a reference to that record, but a reference alone does not grant permission. The system must still check that the approval applies to this action, for this customer, at this time. For a payment, enforce those conditions outside the agent’s editable guidance. The execution service should check the authoritative outage list and the exception’s current status, rather than accepting the agent’s claim that a customer is eligible. Neither the agent nor its learning process should be able to rewrite that authorization record. This is a familiar authorization requirement. If the backend already enforces it, the improper payment is blocked. But faulty guidance can still lead the agent to promise a refund or prepare a misleading recommendation for a reviewer. The evaluation needs to examine those communications as well as the transaction. When a case calls for judgment beyond the agent’s delegated discretion, route it to someone authorized to decide and show the conditions behind the recommendation. The memory can still help assemble the evidence. Not every stored observation needs a manager’s signature. A support agent could learn a faster way to gather the required evidence, provided it still uses approved sources and meets the policy’s requirements. A policy might also already delegate limited discretion. Using that discretion within its terms does not require a new approval each time. Broadening eligibility, raising payment limits or removing mandatory review requires a separate business decision. Keep those proposals separate from approved instructions. An automated classifier may flag a proposal that changes those conditions, but it cannot catch every consequence of an edit. The conditions must remain binding even if the classifier misses it. The refund-policy owner should decide whether to expand the exception. Engineering should show the current rule, the proposed behavior and cases where the decisions differ. “Customers outside the outage list would now receive refunds without review” is more useful to that owner than a screen full of prompt edits. Measure legitimate completion and policy violations separately. Faster closure cannot compensate for an unauthorized payment. Keep the business rules and evaluation criteria outside the control of the process proposing improvements. Tests generated from successful outage cases may check execution, but they do not establish eligibility for unrelated customers. Ask the vendor or internal team: What can the system change between releases, and which business conditions remain binding after it does? Ask the team to demonstrate this sequence in a test environment with the memory and workflow components intended for production. Save the memory before and after learning. Compare the same cases against each saved state, keeping the model, tools and policy fixed. Repeat representative trials. The τ-bench research https://proceedings.iclr.cc/paper files/paper/2025/hash/1b126cc38b8638e07bef37e7b2bb72bf-Abstract-Conference.html offers a useful precedent: it checks customer-service agents against database outcomes and measures reliability across repeated attempts. One successful run is not enough. Keep test runs from influencing one another through newly stored memories. Retain the guidance supplied to the agent, the applicable policy version, execution checks and outcomes, with appropriate access and retention limits. These records help investigators reconstruct what the system had available; they cannot establish exactly how each memory influenced a response. Continue checking the same boundaries as production memory accumulates. The support team should benefit from what its agent learns. It can assemble records sooner, explain the outage more clearly and propose a broader refund policy with evidence for the manager to consider. Until that proposal is approved, other claims should continue through the ordinary process. At the next deployment review, ask which improvements the business has delegated as well as which actions the agent may take. A successful exception can teach an agent something useful. The decision to make it a standing practice still belongs to the business.