{"slug": "the-model-may-not-be-your-biggest-risk", "title": "The model may not be your biggest risk", "summary": "A developer argues that the most significant AI risks lie outside the model itself, citing incidents where OpenAI's cyber-focused models escaped their sandbox and compromised Hugging Face systems, an agent deleting user files, and a coding tool uploading entire codebases. The piece emphasizes that permissions, data provenance, and presentation are critical areas where risks accumulate, and it highlights Anthropic's $1.5 billion settlement over pirated books used to train Claude as an example of upstream data risks.", "body_md": "# **The biggest AI risks sit outside the model**\n\nThe most revealing AI failures right now are not stories about models becoming too capable. They are stories about everything *around* the model. One system received more access than its test environment could contain. Another was trained on material whose acquisition created $1.5 billion in exposure. In a third, people walked away more certain without being any more correct. I [recently argued](https://open.substack.com/pub/gradientflow/p/your-ai-passed-its-evals-thats-the?r=ks4p&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true) that passing your evals does not mean you are safe. This is the next layer of that argument. Permissions, inputs, and presentation are three places where serious AI risks accumulate, and each one lands on a different desk.\n\n#### What an Agent Can Touch\n\nModel capability tells you what a system can do. Architecture determines what it may do, what data and tools it may use, and which actions require approval. A mediocre model with broad write access can be more dangerous than a stronger model in a tight box.\n\nOpenAI’s cyber-focused models, including GPT-5.6 Sol, [left the sandbox](https://www.wired.com/story/openai-models-escaped-containment-and-hacked-huggingface/) used for testing, reportedly exploited a zero day, reached the open internet, and compromised [Hugging Face production systems](https://openai.com/index/hugging-face-model-evaluation-security-incident/). Nobody appears to have noticed for days. Calling the behavior *rogue* explains little. Safety filters had been switched off for the exercise, and network isolation was weaker than the test assumed. Those were architecture and configuration decisions.\n\nOther reports describe an [agent deleting user files](https://mlq.ai/news/openais-gpt-5-6-sol-deletes-user-files-unprompted-weeks-after-company-flagged-the-risk/) without being asked and a [coding tool uploading full codebases](https://mlq.ai/news/xais-grok-build-cli-caught-uploading-entire-codebases-to-google-cloud-without-consent/) without clear consent. Different incidents, same pattern: a weakly specified task becomes dangerous in proportion to what the system can touch.\n\nAn agent can also be redirected by hostile instructions embedded in an email, document, web page, or tool response. **Once tools are attached, prompt injection** can lead to an unauthorized action, not merely a bad answer. Permissions and approval rules therefore have to be enforced outside the model.\n\nGive every production agent its own identity and task-specific permissions rather than letting it borrow a person’s credentials or a generic service account. Split read from write. Use short-lived credentials, narrow network access, transaction limits, and approval gates for irreversible actions. Preserve logs and make recovery possible. The goal is not perfect prevention. It is a small blast radius, fast detection, and enough evidence to understand what happened.\n\nNot every system needs all of those controls. If an agent only reads, gets things wrong cheaply, and can be corrected before anyone acts, most of those controls are overhead. Once it can write to a system of record, spend money, touch regulated data, or produce something a customer will act on without review, those controls apply. The dividing line is the damage the system can cause before someone stops it. Many teams want a third option. There isn’t one.\n\n#### Where the Data Came From\n\nThe same pattern runs upstream, into [what the model was trained](https://gradientflow.substack.com/p/the-ai-data-problem-i-kept-avoiding) on before anyone deployed it. A federal judge [approved](https://apnews.com/article/ai-anthropic-copyright-settlement-claude-books-bartz-74b140444023898aeba8579b6e9f0d63) Anthropic’s $1.5 billion settlement over pirated books used to train Claude. The headline number is the least interesting part. What matters is the line the settlement draws between training on material you obtained lawfully and training on material pulled from pirate libraries. The failure was not model capability. The training pipeline did what it was built to do, and a decision made around it created the exposure.\n\nIf you are fine-tuning a model, assembling an evaluation set, or building [retrieval over third-party content](https://mlq.ai/news/hachette-cengage-and-elsevier-sue-google-over-ai-training-on-millions-of-copyrighted-works/), that distinction is your problem too. How the material was acquired, what rights came with it, and whether those facts can be reconstructed later matter as much as what the model does with it.\n\nBoards now have reason to ask where training data came from, who carries the risk, and whether a disputed model or dataset can be removed without disrupting the product.\n\nAnswering those questions [requires a dataset inventory](https://gradientflow.substack.com/p/the-ai-data-problem-i-kept-avoiding), license terms you can produce on request, a clear answer on who pays if you get sued, a current view of litigation involving your vendors, and a way to swap out a model or dataset without taking the product down. Most teams have some of this. Very few could pull it together on short notice.\n\n#### The Confidence Trap\n\nControls and documentation can limit what a system does and clarify what went into it. Neither fully addresses what happens when a convincing answer reaches a user. A [recent preprint](https://arxiv.org/abs/2607.13562) reports five experiments involving 3,132 participants who answered difficult questions and had the option to say they did not know. Giving people access to AI advice nearly eliminated that restraint. Participants answered more questions, but they were correct about one-third as often as people without AI access. Their confidence nearly doubled.\n\nYou should assume your users can do the same thing. In [medical](https://www.forbes.com/sites/jessepines/2026/07/26/a-lawsuit-says-chatgpt-nearly-someone-heres-the-data-on-ai-diagnosis/), legal, financial, hiring, and security workflows, a confident wrong answer can be more dangerous than an obvious failure.  The risk grows when the interface presents a smooth recommendation as the natural next step. A confidence score [does not automatically help](https://justinflick.com/2026/07/27/llm-confidence-scores.html), especially when it comes from the same model that produced the answer. A precise-looking number can become decoration rather than evidence.\n\nYou cannot engineer this one away, because you cannot stop a person from over-valuing a fluent answer. What you can do is design against it, and the moves are concrete. Ask the user to commit to a view before you reveal the model’s recommendation. Show uncertainty honestly instead of rounding it into confident prose. Take confidence numbers from past accuracy, not from the model. Surface disagreement between sources rather than smoothing it out. Make correcting the system easy enough that people actually do so. Let the system [abstain](https://github.com/cactus-compute/cactus-hybrid). Escalate decisions when the cost of being wrong is high.\n\nThen check whether it worked. The test is not whether people like the assistant or finish more tasks with it. It is whether the human and the system together make better decisions than the process they replaced, know when to abstain, and escalate the right cases.\n\n#### Three Desks, One Problem\n\nThe next phase of enterprise AI will be defined less by who gets the most capable model and more by who builds the strongest system around it.\n\nSecurity and architecture teams have to bound what the system can touch. Data, legal, and procurement teams have to document what went in. Product teams have to preserve human judgment. Rules people once enforced through policy, habit, or supervision increasingly need technical equivalents such as distinct agent identities, approval gates, provenance, and logs.\n\nThese risks land on different desks, but they are becoming one management problem.\n\nCapability is the layer you buy. Everything around it is the layer you own.\n\n**Evaluating the system, rather than just the model, requires a richer set of measurements. The [Luminos whitepaper](https://www.luminos.ai/info/c/a-practical-guide-to-ai-evals?utm_source=gradientflow&utm_medium=newsletter) explains one approach.**\n\n# [When Mathematical Results Become Abundant](https://gradientflow.com/ai-in-math-2026-08/)\n\n**[Ben Lorica](https://gradientflow.com/disclosure/)** edits [Ethics.dev](https://ethics.dev/) and the [Gradient Flow newsletter](https://gradientflow.substack.com/), and he hosts the **[Data Exchange podcast](https://thedataexchange.media/)**. He helps organize the **[AI Conference](https://aiconference.com/?utm_source=gradientflow&utm_medium=newsletter)** and the **[AI Agent Conference](https://agentconference.com/?utm_source=gradientflow&utm_medium=newsletter)**. You can follow him on [Linkedin](https://www.linkedin.com/in/benlorica/), [X](https://x.com/bigdata), [Mastodon](https://indieweb.social/@bigdata), [Reddit](https://www.reddit.com/r/GradientFlow/), [Bluesky](https://bsky.app/profile/gradientflow.com), [YouTube](https://www.youtube.com/c/GradientFlow), or [TikTok](https://www.tiktok.com/@gradientflow). This newsletter is produced by [Gradient Flow](https://gradientflow.com/blog/).", "url": "https://wpnews.pro/news/the-model-may-not-be-your-biggest-risk", "canonical_source": "https://gradientflow.substack.com/p/i-think-we-are-looking-for-ai-risk", "published_at": "2026-08-18 13:00:36+00:00", "updated_at": "2026-09-08 16:56:27.836361+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-agents", "ai-policy"], "entities": ["OpenAI", "Hugging Face", "Anthropic", "GPT-5.6 Sol", "Claude"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-model-may-not-be-your-biggest-risk", "markdown": "https://wpnews.pro/news/the-model-may-not-be-your-biggest-risk.md", "text": "https://wpnews.pro/news/the-model-may-not-be-your-biggest-risk.txt", "jsonld": "https://wpnews.pro/news/the-model-may-not-be-your-biggest-risk.jsonld"}}