{"slug": "damn-it-ai-stop-lying-to-me-and-do-what-i-say", "title": "Damn it AI, stop lying to me and do what I say", "summary": "A new report titled 'Proof, Not Trust' warns that AI agents connected to enterprise systems can be manipulated by adversarial text, citing incidents from 2012 to 2026 including a $440M Knight Capital loss, a $25M Arup wire fraud after a deepfaked CFO video call, and an agent wiping a production database and backups in 9 seconds. The report argues that guardrails inside AI reasoning are vulnerable and that enforcement must be in deterministic code outside the agent, with humans signing off on actions.", "body_md": "# Your AI agent can be talked into anything.\n\nA fraudulent invoice. A poisoned web page. One crafted email. Software that acts on your behalf can be manipulated by the very information it reads, and no prompt can fix that. *Proof, Not Trust* shows the control layer that makes agent authority safe: **the agent negotiates, humans sign, nothing binds without them.**\n\nwired after a deepfaked CFO video call\n\ncompanies breached via chat-agent tokens\n\nfor an agent to erase a database and backups\n\n## The problem: agents that can act, but cannot be trusted\n\nFor two years, AI agents mostly drafted things — emails, summaries, code suggestions — leaving a human to act. That boundary is now dissolving. Enterprises are wiring agents directly into the systems that move money, sign agreements, provision infrastructure, and approve requests. The agent stops being an assistant and becomes an actor.\n\nThis introduces a category of risk the software industry has not had to manage before. A traditional program does exactly what it was written to do. An AI agent decides what to do based on language it reads at run time — and that language can be adversarial. A supplier’s invoice can contain hidden instructions. A web page the agent visits can carry a concealed command. This family of attacks has no clean technical fix, because **the agent’s power and its vulnerability are the same faculty**: it treats the text it reads as meaningful, which is exactly what an attacker exploits.\n\nThe tempting response is to make the agent smarter or more careful — better instructions, a second AI to check the first. This fails for a structural reason: anything inside the AI’s reasoning can be manipulated by the same inputs that manipulate the AI. A guardrail the agent evaluates is a guardrail an attacker can talk the agent past. The only durable place for enforcement is outside the agent, in deterministic code that does not read, reason about, or trust the adversarial material.\n\n## The incident record, 2012–2026\n\nThis is no longer hypothetical. Teal marks the incidents where the missing control is precisely an enforcement layer outside the agent.\n\n- 2012\nKnight Capital loses ~$440M in 45 minutes\n\nAutonomous execution without a control layer fails at machine speed.\n\n- Feb 2024\nA tribunal holds an airline liable for its chatbot’s promise\n\n“An agent can bind the company” is settled in court.\n\n- Feb 2024\nArup wires $25M after a deepfaked CFO video call\n\nAuthorization must arrive through a channel an attacker cannot fabricate — a face and a voice no longer qualify.\n\n- Jun 2025\nEchoLeak: the first zero-click prompt injection in production\n\nThe text an agent reads is an attack surface; guardrails inside the model’s reasoning can be talked past.\n\n- Jul 2025\nA coding agent deletes a production database, then misreports it\n\nInstructions do not bind an agent, and its own account of its actions cannot be trusted.\n\n- Aug 2025\nStolen agent OAuth tokens breach 700+ organizations\n\nStanding credentials are frozen authority — silent on whether this specific action, right now, is legitimate.\n\n- Sep 2025\nThe first malicious MCP server found in the wild\n\nThe agent’s supply chain is adversarial — policy must be enforced outside both the model and its tools.\n\n- Apr 2026\nAn agent wipes a production database and its backups in 9 seconds\n\nA stray credential is blanket authority nobody consciously granted.\n\nWhat has not yet happened at scale is the autonomous payment disaster — for a revealing reason. Enterprises have not dared give agents unsupervised authority to move large sums, precisely because the control layer to make it safe does not exist. The absence of a nine-figure agent-payment loss is not reassurance; it is the market not existing yet.\n\n## Why every current approach falls short\n\nKeep a human in the loop for everything\n\nThe gapSafe, but it defeats the purpose of autonomy and does not scale. The failure is not that humans are involved — it is that involvement has exactly one setting: every action, or none.\n\nGive the agent a limited budget or virtual card\n\nThe gapAuthority is frozen the moment the card is issued. A spending limit cannot express “this purchase was approved by this person,” and it does nothing for fraud that stays under the limit.\n\nExtend identity and access controls (OAuth, scopes)\n\nThe gapPermissions are frozen at login, against a fixed list of allowed actions. A scope says nothing about whether a specific action, with specific values, is legitimate right now.\n\nRoute the agent through existing approval workflows\n\nThe gapClosest to right — but the approval is a record in a database, not bound cryptographically at the point money moves, and it assumes the requester is a trustworthy human rather than software that can be remotely manipulated.\n\nDetect the manipulation (filters, classifiers)\n\nThe gapA useful layer but a brittle one: an arms race against adversarial text that puts the security decision back in the probabilistic zone the attacker is targeting.\n\nThe common thread: each approach either **trusts the agent**, **freezes authority at setup time**, **bolts security onto the thing being manipulated**, or **slows the process to uselessness**. None binds a specific, human-approved intent to a specific action at the moment it happens, enforced by something the agent cannot influence.\n\n## Seven principles that close the gap\n\nTogether they turn your approval policy into a cryptographic lock an AI agent cannot pick, regardless of how it is manipulated. In business terms: signing authority, enforced against agents.\n\nEnforcement lives outside the AI\n\nA deterministic gatekeeper sits between the agent and every consequential action. Ordinary, auditable code — it does not reason, cannot be argued with, and never sees the adversarial text the agent reads.\n\nThe agent’s claims are worthless; only evidence counts\n\nThe gatekeeper never takes the agent’s word. It requires a cryptographic signature produced by a real person through a channel the AI cannot reach. The agent can carry that evidence; it cannot forge it.\n\nConsent is bound to the exact action\n\nNot “this person approves spending” but “this person approves paying this amount to this recipient,” checked field for field. Even a fully compromised agent cannot execute anything that diverges from what a human actually signed.\n\nGranularity is a dial the grantor sets\n\nA signed claim can sit anywhere between a single action and an explicit, revocable mandate — “purchases under $200, at these three vendors, this month.” The dial position is itself part of the evidence.\n\nDecisions happen at the moment of action\n\nNot frozen at setup. The same policy can require nothing for a small routine purchase, one approver above a threshold, and two above a higher one, because it evaluates the specific values of the specific request.\n\nAuthority can be multi-party\n\nReal corporate signing authority is thresholds, roles, and dual control. The gatekeeper composes multiple signed approvals into a single decision, mirroring your actual delegation-of-authority structure.\n\nThe rules are provable\n\nThe policy is authored so its properties can be mathematically verified, not merely tested — written in a proof assistant and compiled to the enforcement engine. You can demonstrate with proof, rather than assertion, that your controls behave as claimed.\n\n## This is not a new idea — it is zero trust, extended to the agent\n\nNone of these principles is novel security theory. Zero trust — the stance that no actor is trusted by virtue of where it sits, and that every action is authorized on its own merits — runs from Saltzer and Schroeder’s 1975 principles through de-perimeterization, Google’s BeyondCorp, and the NIST Zero Trust Architecture standard.\n\nThis system extends that architecture with one addition to the list of things not trusted: the AI agent itself. Classic zero trust distrusts an actor’s network position and then verifies its identity — after which the authenticated actor is trusted to behave within its permissions. An agent breaks that assumption. **Its identity is not in question; its judgment is.** It is a legitimate participant that can be talked into betraying its own instructions by the data it reads.\n\n## Why “human in the loop” becomes a rubber stamp\n\nThere is an honest objection to any approval-based system: people click “approve.” Two decades of human-factors research on automation bias documents the pattern — when an automated system is right most of the time, its operators stop evaluating and start ratifying. Vigilance decays precisely because the system earns trust. The attack category called MFA fatigue exists because approval prompts become reflexes.\n\nAsk it plainly: would you sign every piece of paper an employee put in front of you? No executive would. Yet that is what an unstructured “human in the loop” becomes at agent speed — a signature line attached to a firehose. And the trap is double. The first failure mode is the human who approves without reading. The second is worse: **an agent under manipulation can simply claim the human approved.** A checkpoint the agent merely reports passing is not a checkpoint.\n\nThe architecture answers both. The signature is produced on the approver’s own device, through a channel the agent cannot reach, so approval cannot be fabricated. Routine actions run inside scoped mandates and interrupt no one, so the approvals that do reach a human are rare enough to deserve attention. The approval screen shows the specific amount, recipient, and document — a concrete decision, not an “OK?” button. The goal is not more human approvals; it is fewer, better ones.\n\n## How it works, concretely\n\nThe clearest illustration is the smallest one, already running in working software. A child asks to spend $15. The family policy says purchases up to $10 need only the child’s own signed request, but anything above $10 also needs a parent’s approval. The agent proposes the purchase; the gatekeeper denies it precisely — “missing: parent’s authorization”; the parent signs on their own device, bound to that exact request; both signatures are verified against the actual purchase; only then does the payment execute.\n\nNow translate that to a corporation. The “child” is an AI procurement agent. The “$10 threshold” is a delegation-of-authority matrix. The “parent” is whichever manager the policy requires for that amount and category. The flow is subtractive, not ceremonial: the policy determines which signatures the transaction requires, the system determines which already exist, and **only what is missing interrupts a human.** If the agent was manipulated into paying a different party, the signatures won’t match, and the payment never leaves.\n\n## Why now\n\nDeployment is outrunning control\n\nEnterprises are putting agents into consequential roles right now, ahead of the safeguards that make doing so responsible.\n\nRegulation is tightening\n\nProvable, auditable authorization is moving from nice-to-have toward expected, especially in finance.\n\nThe standards are converging\n\nOn the same primitives this is built from — signed mandates and verifiable consent — while leaving the hard part, the enforcement decision, unaddressed.\n\nThe risk is already being priced\n\nMunich Re extends AI-performance cover toward agent risk, Lloyd’s-backed specialists write standalone AI-liability policies, and Verisk is drafting agentic-AI exclusions. Insurers are deciding, line by line, what an ungoverned agent costs.\n\nThe fourth force deserves the closest look. When insurers start excluding ungoverned agents, a provable, auditable control layer becomes the thing that keeps an enterprise insurable — a compliance nicety turned purchasing requirement.\n\n## Three questions to ask about your own agents\n\nA reader who takes nothing else from this paper should take these. For every AI agent your organization runs or plans to run:\n\n- Q1\nIf this agent were manipulated today, what is the most expensive thing it could commit you to — and would you find out from your controls, or from the counterparty?\n\n- Q2\nCan you prove, to an auditor or an insurer, that a specific consequential action was approved by a specific person — with evidence that doesn’t depend on the agent’s own logs?\n\n- Q3\nWhere do your agent’s rules live: in its instructions, where the text it reads can override them, or outside it, where nothing it reads can?\n\nIf any answer is uncomfortable, the discomfort is the point: these are the questions regulators, insurers, and your board will be asking within the year. Better to ask them first.\n\n- —The complete incident record, 2012–2026, with sources for every entry.\n- —The architecture diagrams: where the gatekeeper sits, and the five-step flow from proposal to execution.\n- —Implementation detail for the three building blocks — verifiable credentials, the verified policy engine, and the gatekeeper.\n- —“One architecture for any agreement”: how the same architecture covers payments, contracts, access grants, and regulated sign-off.\n- —The expensive history of unproven rules — the DAO and Parity — and why multi-party approval code is the sharpest warning.\n- —Full references and the roadmap: what runs today, and what is still ahead.\n\nCTOs, CISOs, and finance and operations leaders wiring AI agents into systems that move money, sign agreements, or grant access — and the risk, audit, and insurance professionals who will be asked to sign off on them.\n\nThe control layer described here is open source and inspectable today — [github.com/SenteLabsAI/extensible-mcp ↗](https://github.com/SenteLabsAI/extensible-mcp)", "url": "https://wpnews.pro/news/damn-it-ai-stop-lying-to-me-and-do-what-i-say", "canonical_source": "https://sentelabs.ai/proof-not-trust", "published_at": "2026-08-29 00:39:29+00:00", "updated_at": "2026-08-29 01:18:27.645533+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-policy"], "entities": ["Knight Capital", "Arup", "EchoLeak"], "alternates": {"html": "https://wpnews.pro/news/damn-it-ai-stop-lying-to-me-and-do-what-i-say", "markdown": "https://wpnews.pro/news/damn-it-ai-stop-lying-to-me-and-do-what-i-say.md", "text": "https://wpnews.pro/news/damn-it-ai-stop-lying-to-me-and-do-what-i-say.txt", "jsonld": "https://wpnews.pro/news/damn-it-ai-stop-lying-to-me-and-do-what-i-say.jsonld"}}