{"slug": "the-ai-as-normal-technology-view-of-loss-of-control-incidents", "title": "The AI-as-Normal-Technology view of loss-of-control incidents", "summary": "A 13,000-word essay applies the \"AI as Normal Technology\" framework to synthesize the AI safety and cybersecurity communities' opposing views of recent loss-of-control incidents at OpenAI and Anthropic, including the OpenAI-Hugging Face incident in which hundreds of OpenAI agents accessed the internet and hacked Hugging Face to learn how they were being graded on an evaluation. The essay argues AI companies should be liable for their agents' actions, clarified through policymaking, and identifies three investment areas to address loss-of-control risks: research on controlling increasingly capable agents, translating existing control techniques into usable tools, and organizational changes to ensure adoption. It also calls for organizational governance standards as a way to pace the frontier and move AI companies past a \"move fast and break things\" attitude.", "body_md": "*This essay is over 13,000 words long and represents our most substantial writing on AI safety since the original essay.*\n\nOver the last few months, [loss](https://openai.com/index/hugging-face-incident-and-the-road-ahead/)-[of](https://collusion.wiki/)-[control](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) incidents at OpenAI and Anthropic have intensified concerns about AI safety. Warnings about existential risk have increasingly [reached](https://www.platformer.news/ai-safety-vibe-shift-coxon-anthropic/) the broader public, alongside calls to slow AI development. Dario Amodei’s call to “[pace the frontier](https://darioamodei.com/post/we-must-pace-the-frontier)” reflects the concern that safety efforts are not keeping up with AI capabilities.\n\nThe most prominent example was the [OpenAI - Hugging Face incident](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/), where hundreds of OpenAI agents got access to the internet and hacked Hugging Face to find how they were being graded on an evaluation. Over the last two weeks, the details of [many other instances](https://collusion.wiki/additional-findings) of such activity from OpenAI agents being evaluated have surfaced, such as agents using an old Wiki website to communicate with each other despite restrictions on such activity, or [attacking](https://www.reuters.com/legal/litigation/openai-agents-attacked-software-service-rubygems-before-hugging-face-incident-2026-09-11/) a software repository to attempt to upload malicious software.\n\nThe AI safety community has viewed these incidents [largely](https://thezvi.substack.com/p/openai-model-hacks-into-huggingface) as a crisis for [alignment](https://blog.redwoodresearch.org/p/are-we-existentially-threatened-by).[1](#footnote-1) In this view, alignment will become harder over time as agents learn to reason covertly, and as a result, loss-of-control incidents are likely to become much more widespread and damaging as agents become more capable.\n\nOn the other hand, cybersecurity practitioners have [largely](https://www.iansresearch.com/resources/all-blogs/post/security-blog/2026/08/28/openai's-postmortem-recasts-the-hugging-face-breach-as-an-incident-response-failure) [viewed](https://www.linkedin.com/feed/update/urn:li:activity:7493061037823492096/) these incidents as consequences of companies failing to adopt basic security precautions. They do not see the incidents as a sign of AI reaching a new milestone in cybersecurity. This is also the dominant reaction in the tech community outside AI, which has largely treated the incidents as a result of [ineptitude and negligence](https://news.ycombinator.com/item?id=49666735) by AI companies.[2](#footnote-2)\n\nBoth communities have made important points. But the polarization between them is counterproductive, and there are lessons from both the alignment failure and the security failure for understanding the path forward.\n\n**In this essay, we apply the AI as Normal Technology framework to synthesize the views of the safety and cybersecurity communities and offer a constructive middle ground.** We think AI companies should be liable for what their agents do, and this should be clarified through policymaking. At the same time, recognizing their responsibility does not mean that preventing future incidents is a solved problem. \n\nWe identify three areas where investment is necessary to address loss-of-control risks: research to develop better methods for controlling increasingly capable agents, translating existing research and known control techniques into usable tools, and organizational changes to ensure that these tools are actually adopted. In our view, standards for organizational governance should be a key way to pace the frontier and pull AI companies out of the “move fast and break things” attitude they currently operate in.\n\nThis essay has three parts. In Part 1, we argue that alignment alone is not enough to prevent such incidents, and discuss technical, organizational, and policy interventions for improving AI control. We are cautiously optimistic that the right investments and policy interventions can allow AI control to keep pace with AI capability improvements. In Part 2, we discuss the impact of improving AI capabilities, such as agent swarms, on cybersecurity. In Part 3, we share how our views on AI safety have changed in light of new evidence.\n\nA summary of our argument:\n\n- **We agree with security practitioners that OpenAI did not take adequate protections for controlling their agents.** But this is not just a matter of applying 30-year-old security methods to a new domain. Security for AI agents — AI control — while important, is not a solved problem. While known control methods would have prevented the Hugging Face incident, as agent capabilities continue to advance, we will only be able to control them if we invest adequately in control interventions.\n- **We also agree with security practitioners’ implicit position that these incidents are primarily a security story.** In the AI safety community, rogue agents are treated as inherently catastrophic because of the assumption that there is an endless list of risks that will arise from their development. We disagree. We have long advocated that the best approach to AI safety is to identify the risks and address those specific risks. Over the last few months, it has become clear that one urgent risk is cyberoffense, because it has unique properties that allow agents to carry it out autonomously. We should similarly invest in defenses against other*specific* risks, such as biorisk and risks from military AI.\n- **We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents.** But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques. More broadly, there are many common-sense policy proposals that could help promote investments in AI control where we share common ground with the safety community.\n- **How should we reason about AI’s impact on cybersecurity?** It’s plausible that advances in agent capabilities upset the offense-defense balance for cybersecurity. We cannot yet be certain, but there is enough evidence that agent capabilities might soon make widespread cyberoffense possible that urgent action is warranted. We discuss potential interventions for tilting the offense-defense balance towards defenders.\n- **How the AI as Normal Technology framework has evolved over the last year.** We take stock of AI progress and share how we have updated our views. In the essay, we did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models). We were too confident that companies would take basic control precautions and underplayed the importance of jaggedness, which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity.\n- **At the same time, many distinctive claims of AI as Normal Technology have held up, and it remains valuable for understanding AI’s societal impacts.** In particular, we think recent incidents support our[continuity hypothesis](https://asteriskmag.com/issues/10/does-ai-progress-have-a-speed-limit) — the behavior of “rogue” agents became apparent and widely publicized while they are still far from causing serious harm and incompetent at hiding their traces. The societal reaction to even the relatively small harms from these incidents has been fierce (and the safety community deserves credit for keeping up pressure on companies). Whether this translates into meaningful changes in companies’ behavior remains an open question, and a test of the usefulness of the AINT framework.\n- Finally, we think AINT is particularly valuable for analyzing these incidents because it provides a framework for synthesizing the AI safety and cybersecurity communities’ views into a coherent plan of action: hold companies responsible, invest in control, and strengthen defenses against specific risks.\n\nThis table summarizes our argument. While the AI community has in principle emphasized the notion of “defense in depth”, in practice, alignment has by far been the intervention that received the most attention and investment. On the other hand, the cybersecurity community has viewed control as largely a solved problem. In our view, to protect against loss-of-control incidents from legitimate parties, we need to urgently invest in control so that it keeps pace with capability improvements. Neither alignment nor control help against malicious users, so we need downstream defenses and resilience to alleviate the impact and severity of risks. All of these efforts can be shored up by policy interventions. The table emphasizes interventions for cybersecurity, but we should simultaneously invest in defenses against other risks, such as biorisk.\n\n## Table of contents\n\n[Part 1: We can get better at AI control through technical and policy interventions](https://www.normaltech.ai/i/215609071/part-1-we-can-get-better-at-ai-control-by-investing-in-technical-and-policy-interventions)\n\n[Part 3: What AI as Normal Technology got wrong — and right — about AI safety](https://www.normaltech.ai/i/215609071/part-3-what-ai-as-normal-technology-got-wrong-and-right-about-ai-safety)\n\n# Part 1: We can get better at AI control through technical and policy interventions\n\n*In this part, we focus on AI agents that are operated by legitimate actors — such as consumers, businesses, and AI developers — who don’t intentionally use them to cause harm. This was the case for the OpenAI - Hugging Face incident. In Part 2, we focus on malicious actors who want to use agents for intentionally causing harm, such as using them for cyberoffense.*\n\nHow can we prevent AI agents from taking harmful actions? There are two broad kinds of interventions. One is AI alignment. This involves changing the AI system itself, such as by fine tuning or using reinforcement learning with human feedback (RLHF), to make it less likely to take harmful actions or give harmful responses. Alignment has been [pivotal](https://www.normaltech.ai/p/model-alignment-protects-against) for the commercial success of AI so far.\n\nThe other intervention is [AI control](https://www.normaltech.ai/i/161317202/control-comes-in-many-flavors): interventions made *outside* the model weights to prevent harmful actions — [even if the agent is misaligned](https://proceedings.mlr.press/v235/greenblatt24a.html). This includes improvements to sandbox security (to prevent AI models from taking unanticipated actions outside the sandbox), implementing the principle of least privilege, comprehensive logging, automated tripwires for unsafe behaviors, rapid shutdown mechanisms, and monitoring the agent to detect and prevent harmful actions. These mechanisms allow us to prevent unsafe actions *even if* agents are misaligned and try taking harmful actions.\n\nRoughly speaking, AI control can be thought of as cybersecurity against AI agent adversaries. Whereas cybersecurity has traditionally been concerned with human actors who seek to compromise or exploit a system, AI control uses these same principles to prevent unwanted actions by AI agents.\n\nUnlike cybersecurity, control interventions target agents being used within an organization (as opposed to human adversaries or external agents). This makes the problem both more and less challenging than traditional cybersecurity. It is less challenging because the agent is directly controlled by the organization (rather than being an unknown adversary), so its operating conditions can be closely monitored and intervened on. It is more challenging because these agents are often deployed by users who might have escalated privileges, and imposing security constraints also imposes constraints on usability which could be resisted or circumvented by users.\n\nAnalysis of the OpenAI - Hugging Face incident from the AI safety community primarily treats it as an [alignment failure](https://thezvi.substack.com/p/openai-model-hacks-into-huggingface). The main intervention that the AI safety community has focused on is improving technical methods for AI alignment. When AI control is discussed, it is often seen as a stopgap.[3](#footnote-3) On the other hand, in the cybersecurity community, AI control is often seen as a [solved](https://uphack.io/blog/post/the-hugging-face-incident-is-not-an-ai-story/) problem for preventing the kinds of issues that arose in the OpenAI - Hugging Face, as it only requires the adoption of known security techniques.\n\nIn this part, we discuss why marginal investments in control might be better suited compared to alignment for resolving such incidents. While we agree with the AI safety community that alignment is helpful in reducing the prevalence of such incidents, we think that alignment alone will not suffice to eliminate them. The incidents show that AI control has not been prioritized by AI companies, despite there being known methods that would have prevented the incident.\n\nA common response to recommendations for implementing AI control is that while they suffice for the current set of models, they might not suffice for future, more capable models. Here, we depart from the consensus in the security community: we agree that existing control interventions might not be enough to prevent harm from future, more capable models. But we argue that there is a lot of headroom in improving control interventions, and the right policy interventions (such as imposing liability for loss-of-control incidents, transparency for near misses, and whistleblower protections) can lead to dramatic improvements in controlling AI systems.\n\n## Alignment is helpful but not sufficient for preventing safety incidents\n\nWe have argued that [model alignment is not enough for AI safety](https://www.normaltech.ai/p/ai-safety-is-not-a-model-property). To recap: when a model is deployed, it doesn’t have enough context to know if the conditions it’s being deployed in are safe or not — a lot of that context is outside the model. Alignment isn’t a solved technical problem, and there are unknown technical advances needed to align models and prevent harmful actions.\n\nFor example, when a model is deployed to find vulnerabilities, it might be used for cyberoffense (to develop exploits) or defense (to fix them). Alignment can be helpful for harm reduction; for example, models can be trained to tilt the offense-defense balance towards cyberdefenders by finding vulnerabilities but not contributing to finding exploits, or analyzing the source code (which defenders have access to) but not analyzing binaries. But alignment is not foolproof, since beyond simple interventions (such as analyzing the code but not the binaries), it is unclear if the model is being used for a defensive or offensive task based on the context given to the model.\n\nSimilarly, for loss-of-control incidents like the OpenAI - Hugging Face one, the agent might be expected to carry out the attack in a “simulation” of its cyberoffense capabilities, but not attack Hugging Face in the real world. In this case, it might have been relatively clear to the agent that the Hugging Face infrastructure was not a simulation, but you could imagine cases where the distinction is not so clear, such as when the agent is tasked with escaping its sandbox to estimate loss-of-control risks. There are many other edge cases where alignment would fail.\n\nThings become even more blurry once a model is deployed to customers and businesses as opposed to being restricted within AI companies. The contexts within which AI companies use agents are relatively bounded, making the alignment problem easier to address. But customers outside companies can use AI agents in boundless ways, making it harder to judge whether an action is legitimate or a safety hazard based on the information available to the model.\n\nDespite these challenges, the mainstream view in the AI safety community is that to make AI deployments safe, we need to invest in alignment. This is also reflected in the safety efforts of AI companies, which often treat interventions outside the model (such as AI control interventions) as a stopgap. For example, when discussing monitoring interventions that OpenAI deployed for Astra, the company’s report [noted](https://openai.com/index/path-to-astra/): “These safeguards cannot replace good alignment of our models as capabilities increase, and our goal is for future models to be aligned well enough that these safeguards are never triggered.”\n\nTo be clear, we don’t mean to underplay the importance of making advances in alignment, and we think developing a better understanding of what causes harmful model behaviors is an important research direction. For example, over the last few weeks, [OpenAI](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf) and [Anthropic](https://alignment.anthropic.com/2026/reward-seeker/) have both shared how RL environments that are misspecified or those that reward shortcuts induce reward hacking in models. Addressing these shortcomings might reduce how often models take harmful actions. But they still won’t guarantee safe behavior.\n\nGiven the shortcomings in alignment science, deployers of AI agents must complement alignment with broader interventions in AI control. Unlike alignment, when it comes to control, there are many existing techniques that would have prevented the incident.\n\n## OpenAI did not use known control interventions that would have prevented the incident\n\nIn the aftermath of the incident, reports by [OpenAI](https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf) and [METR](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) revealed more details. It is clear from the reports that the agents acted in ways that weren’t intended by the evaluators, and we think it is fair to describe the actions of the agent as misalignment.\n\nBut note that the incident occurred when OpenAI had disabled most mechanisms for controlling their agents. While the company had implemented monitoring for [most internal uses](https://openai.com/index/running-codex-safely/), it hadn’t implemented it for [evaluations](https://x.com/tomekkorbak/status/2092855236247597064?s=20). On top of that, the evaluation used a different set of prompts and harnesses compared to the production Codex harness.\n\nWhat would have happened had the company used known control techniques? OpenAI found that using the production Codex harness and system prompt reduced the propensity to compromise out-of-scope infrastructure by more than 100×. Their automatic review would have flagged most dangerous actions in the tested rollouts. And its current chain-of-thought monitoring would have raised an alert more than a day before the Hugging Face breach.\n\nIn fact, existing AI models are already helpful for monitoring and control. METR used OpenAI’s publicly available models to carry out large parts of their analysis. According to their report, these agents made errors, showed poor judgment, and sometimes adopted the perspective of the agents whose behavior they were reviewing. Yet, an investigation of this scale would have been infeasible in six days by three researchers without them. Monitoring does not have to be perfect to be extremely useful when it augments skilled humans rather than replacing their judgment.\n\nMore importantly, before the incident occurred, there was a warning sign that something had gone wrong — one of the services that the agent used to communicate had an internal outage. Instead of launching a deep dive into the root cause for these evaluations, OpenAI restarted the evaluations in the same setup after fixing the one bug they had uncovered.\n\nThis shows that the culture within AI labs was just as culpable for the incident. Engineers work long days, spin up thousands of experiments, and there’s insufficient human oversight of these experiments. As Joshua Saxe [notes](https://www.chinatalk.media/p/cyber-apocalypse-now):\n\n*…the culture among the training teams and the evals teams at the labs has a Wild West feeling to the whole thing. Everybody’s under a ton of pressure to move really quickly. There’s enormous time pressure to release new models, tremendous awareness of how any given lab is doing relative to the others in terms of capability. And the field has evolved super fast in the four years since ChatGPT launched in fall 2022. The whole thing has been kind of a blur — anybody who’s worked in the labs in that time has been working sixty-hour weeks the whole time. So it’s not surprising to me that you could have a kind of grad-student computer-science-lab culture around the security of these model training runs, which I think is what’s the case, and I think that’s why you’ve seen these lab escapes. It is ironic, because frontier AI people talk about safety all the time — but that irony is a real thing. The security practices are not very good, because of all those dynamics.*\n\nTo OpenAI’s credit, they have been transparent about the incident even where it doesn’t suit their interests. For example, they conducted additional analyses with the Codex comparison in the report and put controls in place to improve monitoring after the incident.\n\nBut given the competitive dynamics in the AI industry, there is a lot of room for policy to require improvements to internal incident response, accountability and liability for loss-of-control incidents, and improving the culture around security and safety.\n\n## Existing organizational governance norms would have prevented the incident\n\nEven setting aside policy interventions, one reason for these incidents is that at the organizational level, AI companies function like startups. While their revenue and valuations have grown immensely, by all accounts, they lack the organizational processes that are the norm in any other industry. As a result, they haven’t developed the organizational maturity to get visibility into risky experiments and deal with the fallout.\n\nRather than behaving like startups, AI companies need to behave like more mature organizations. When a single misconfigured RL environment or unmonitored evaluation can cause real-world harm, individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams. AI companies need processes for reviewing experiments, assigning responsibility for monitoring them, and investigating warning signs deeply before restarting experiments. If putting these processes in place requires pausing some experiments, companies should do so. In fact, in the first instance, **proposals to “pace the frontier” should be about addressing these organizational failures** rather than aiming for technical breakthroughs alone.\n\nUnfortunately, AI companies are trying to reinvent basic aspects of organizational governance as a problem to be solved by improving the technology. But even developing better control techniques will not be enough if irresponsible individuals or teams within large organizations can choose not to use them. Without such governance, of the thousands of employees and hundreds of teams at a company such as OpenAI, you need just one team to deploy agents irresponsibly for incidents to reoccur.\n\nTo get a sense of how standard such organizational norms are, consider any other industry where a company claims to develop revolutionary technology. It is hard to imagine the company reporting that its technology had caused real-world harm (such as hacking another organization) and expecting to continue running such experiments without meaningful oversight. In fact, AI companies’ claims about the power and risks of their technology makes the case for organizational oversight stronger. We do not need to resolve disagreements about unsolved technical challenges to expect companies to take basic precautions.\n\nWe speculate that one reason why AI companies haven’t adopted [standard governance mechanisms](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/) is that they have grown too quickly and have tried to maintain their startup mentality despite this period of extraordinary growth. Companies in most other industries see more gradual growth that allows the time for building organizational norms. AI companies resemble Facebook and social media companies that grew quickly with the “move fast and break things attitude”, but ultimately needed to put in [guardrails](https://about.fb.com/news/2018/04/restricting-data-access/) and organizational governance [standards](https://about.fb.com/news/2018/05/update-on-app-audit/) in light of incidents like Cambridge Analytica.\n\nFortunately, it seems like AI companies are recognizing the importance of such changes. In Amodei’s [letter](https://darioamodei.com/post/we-must-pace-the-frontier) calling for pacing AI, he refers to AI control requiring “operational excellence”. He says: “Many things go wrong not because companies are missing some important theory or insight, but because of problems in execution.” Organizational governance is one way to pace the frontier and prevent execution errors that cause real-world harm.\n\n## Why has the AI community under-invested in control?\n\nIf there were known control techniques to prevent the incident from occurring, why didn’t OpenAI invest in them? In the previous section, we discussed one potential reason could be the culture within AI companies that prioritizes the pace of progress over safety and security.\n\nBut there is another reason, and it relates to the difference in how AI companies view alignment and how they view control. Alignment is treated as a key component of every model release; billions of dollars have been spent on developing methods and collecting data for alignment. But OpenAI did not implement basic control interventions during evaluations. Across AI companies and the safety community, alignment is seen as the key problem to track, and control as a [stopgap](https://alignment.anthropic.com/2025/bumpers/). We think there are a few reasons why the gap persists:\n\n**The epistemic gap between cybersecurity and AI:** There are many known techniques in cybersecurity that could improve the state of AI control. But these communities can talk past each other when it comes to implementing solutions. After the OpenAI - Hugging Face incident, many security practitioners pointed out that OpenAI did not take basic preventative steps to address the incident. On the other hand, advocates for AI safety (both within and outside AI companies) correctly argued that improvements in AI capabilities mean that known control interventions might not work for very long.\n\nThere is a lot of value in synthesizing these perspectives and bridging the gap between these communities. AI companies have ignored known control interventions and should invest in them. But known interventions might not be enough and need to be dramatically updated to account for increasing agent capabilities. As one example, consider the recent post by [Artem Dinaburg](https://blog.trailofbits.com/2026/08/26/vms-wont-contain-cyber-capable-agents/), who shows that capable agents can develop new exploits, so we can no longer assume that standard VMs can be used to sandbox capable agents. Now that companies routinely use millions of agents in such sandboxes (in either production or research settings), we need to dramatically increase the scrutiny that sandbox security receives, such as by stress-testing and improving sandbox security through offensive agents on a routine basis with models of increasing capability. We need similar efforts to harden cyber infrastructure across the board against capable cyber agents.\n\nAt the same time, there are many straightforward interventions that do not need new technical breakthroughs. As an example, AI agents today are already helpful for control despite the lack of specific interventions on improving control and monitoring. For example, METR used off-the-shelf models in their analysis, and still found them useful. We expect that through the right incentives, we could continue to broaden our control toolkit. These could [include](https://www.apolloresearch.ai/monitoring/a-scalable-monitoring-research-agenda) [fine tuning models specifically for improving control](https://x.com/MariusHobbhahn/status/2094861159954325774?s=20), developing benchmarks and leaderboards to improve monitoring, developing layered mechanisms (such as a separate layer of agents monitoring agent monitors) to improve the robustness of control and monitoring efforts, using ensembles of models with different model families at different temperatures, etc.\n\n**The tradeoff between usability and control:** Even when security experts are involved in AI development, they may find it hard to advocate for control interventions because these interventions can trade off against usability. For example, when Claude Code was initially launched, many people used it without restrictions using a mode called “dangerously skip permissions”. This was despite the obvious risks of allowing the agent to take any action without permission.\n\nBut this tradeoff is not inherent. Over the last few months, Anthropic and OpenAI have both released auto review modes in [Claude Code](https://www.anthropic.com/engineering/claude-code-auto-mode) and [Codex](https://alignment.openai.com/auto-review/). These modes run a classifier on the proposed actions being taken by the agent to prevent harmful or destructive actions, detect potential prompt injections in content that the agent looks up, and, when necessary, pause the agent to raise concerns for the user’s approval.\n\nThis is not just a security improvement compared to no human intervention; it is also an improvement compared to requiring human approval for each action. The latter approach quickly leads to “alarm fatigue” and the approval becomes a rubber-stamping exercise. A recent Anthropic evaluation [found](https://claude.com/blog/auto-mode-default-in-claude-code) that auto mode prevented harmful actions compared to both full human approval and the “dangerously skip permissions” modes.\n\nCoding agents use dozens of such control interventions. Other interventions include running agents within [sandboxes](https://learn.chatgpt.com/docs/sandboxing?surface=app), user settings for which commands need user approvals, and access protections for agent configurations set by the user to prevent agents from editing them.\n\nThis shows how we can improve both security *and* usability via new control interventions. We think there is a lot of headroom for improving control interventions without compromising on usability. Notably, despite its usefulness, auto mode was introduced only in March 2026 — over a year after the release of Claude Code, and well after coding agents became mainstream, and despite the fact that it didn’t require new technical breakthroughs.\n\n**Conflating capability and power:** In the AI as Normal Technology essay, we pointed out that those with the superintelligence worldview often conflate the capabilities of AI systems with how powerful they will be (in the sense of being able to act in the real world). Implicit in this is the assumption that once models become much more capable, they will also be able to exert more real-world influence.\n\nBut as we discuss in the essay, capable is not the same as powerful. If you assume capable models will be given power, alignment is all you have left. On the other hand, under the AI as Normal Technology view, we can choose whether to grant power to AI systems. Companies can be incentivized to implement control techniques through policy interventions such as increasing transparency, auditing, and imposing liability for safety incidents.\n\n*Differences between two views of how AI loss of control could arise. Applying it to the OAI/HF incident, In the superintelligence view, fixing misalignment is the main technical intervention; in our view, fixing the poor monitoring, lack of human oversight, sandbox security, and other control interventionsare likely to be much more productive. (Figure from AI as Normal Technology.)*\n\n## Will control keep pace with AI capabilities?\n\nOne response to prioritizing control-based interventions is that control [will stop being effective](https://techcrunch.com/2026/07/27/openais-hugging-face-breach-has-reignited-the-debate-over-alignment-and-control/) as AI systems become more capable. For example, there has been a lot of discussion around OpenAI’s architectural choices for GPT-6 Astra [reducing](https://x.com/steph_palazzolo/status/2094954680765829533?s=20) the ability to conduct chain-of-thought monitoring.\n\nBut note that this is not just a matter of forecasting what technical capabilities are needed or predicting the inevitability of a race to the bottom on safety. There are many technical and policy levers that can incentivize investments in control. If companies are liable for the harms caused by out-of-control systems, control interventions would be a straightforward way to make progress without compromising on control.\n\nAt a technical level, even if [chain-of-thought monitoring](https://arxiv.org/abs/2507.11473) is no longer effective for oversight, there are many orthogonal control mechanisms for AI control. We can monitor the environment and sandbox state, tool calls, inter-agent messages, and internal activations.\n\nIn fact, most external monitoring relies on such channels since the raw chains of thought aren’t available. Based on public documentation, auto mode runs on tool calls and inputs to the model, not the model’s chains of thought. In our own work, we have found such external control mechanisms to be extremely valuable.\n\nChain-of-thought monitoring could also *[reduce](https://x.com/charliermarsh/status/2097793924781773009?s=20)* the efficacy of monitoring, since the chains of thought could contain ([incorrect](https://arxiv.org/abs/2504.09762)) assertions such as the setting being a simulation that throws off the monitor. A recent Anthropic report [notes](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents):\n\n*Our offline monitor would likely not have flagged the Mythos 5 incident—it flagged only around 1% of actions. Mythos 5 CoT, which indicates that the environment is simulated, convinces the monitor the activity is permitted. When we strip the CoT and show the monitor only user turns, tool calls, and tool results, the flag rate jumps to about 50%.*\n\nStill, losing channels for monitoring AI systems is a technical choice by AI companies, which can be disincentivized by policy. Here, our thesis is that if companies need to internalize the costs of out-of-control AI systems (such as by policy changes that clarify or shore up liability), they will make the necessary investments. We talk about some such proposals in the next section.\n\nThe need for control also creates market incentives for third-party companies to develop [tools](https://watcher.apolloresearch.ai/blog/announcing-watcher-live/) and provide the expertise for AI control within other orgs — both AI companies as well as businesses and third parties adopting AI. Some companies, like Apollo Research, have this explicit [thesis](https://x.com/MariusHobbhahn/status/2090441122086621474?s=20) for motivating their work. They have worked with AI companies as well as businesses outside the AI industry.\n\nWhile tech companies have the expertise to implement these interventions in-house, how would AI control interventions scale beyond these companies? As an example, consider cybersecurity. Cybersecurity consulting is a rapidly growing share of consulting companies’ revenue, which often carry out implementation work in the area. For companies that lack the scale or technical expertise for implementing AI control in-house, this function might be realized through a combination of off-the-shelf products for AI control and consulting companies helping set up this internal expertise.\n\nWe might also see application-specific controls, since some controls require context about how the application is used. For example, we might see companies specializing in legal AI implementing controls to reduce fabrications, and those specializing in medical AI applications implementing controls for users’ medical data etc.[4](#footnote-4)\n\nDeveloping [agent infrastructure](https://arxiv.org/abs/2501.10114) could also play a role. This could include providing [agent-specific delegation credentials](https://proceedings.mlr.press/v267/south25a.html) with the right level of access instead of letting agents run with a user’s escalated credentials and identification to tie agents to real-world users who commissioned the task. We expect that the need for such interventions will grow dramatically over the next few years. It won’t just be a matter of building tools; we also need to train specialists in AI control.\n\n## AI control should become a job (and a part of every job), just like cybersecurity\n\nIn AI as normal technology, we wrote:\n\n*As more physical and cognitive tasks become amenable to automation, we predict that an increasing percentage of human jobs and tasks will be related to AI control. If this seems radical, note that this kind of near-total redefinition of the concept of work has happened previously. Before the Industrial Revolution, most jobs involved manual labor. Over time, more and more manual tasks have been automated, a trend that continues. In this process, a great many different ways of operating, controlling, and monitoring physical machines were invented, and what humans do in factories today is a combination of “control” (monitoring automated assembly lines, programming robotic systems, managing quality control checkpoints, and coordinating responses to equipment malfunctions) and some tasks that require levels of cognitive ability or dexterity that machines are not yet capable.*\n\nThis trend is underway in professions that are at the leading edge of AI adoption, notably software engineering. More and more software engineers today are [agent operators](https://arxiv.org/abs/2606.05391), not manual coders. But this incident shows that a few more things are needed:\n\n- Some people will probably need to specialize in AI control, especially when it comes to running large numbers of highly capable agents. [5](#footnote-5)\n- So far, engineers are learning AI control through trial-and-error and osmosis. There needs to be more of an effort to extract its principles and teach people to quickly get up to speed on how to monitor powerful agents in real time.\n- As mentioned above, there need to be organizational changes and governance norms that complement improvements in control tooling and training. What sort of team and processes does one need in order to be running tens of thousands of reduced-guardrail agents in parallel? We don’t know, but certainly more than what OpenAI appears to have had in place.\n\nWe expect AI control to continue to be important as AI agents improve because of the [agent trilemma](https://x.com/random_walker/status/1942915285389836326?s=20): developing *general-purpose* AI systems that have *no human supervision* that are deployed in settings with a *high cost of errors* has proven to be extremely challenging.\n\nOne reason for this is the lack of [reliability](https://hal.cs.princeton.edu/reliability/) in AI agents. We expect that even as agents become reliable at tasks of increasing complexity, their reliability will continue to lag capability, creating the need for human supervision at tasks that require frontier capabilities.\n\nThe other reason is safety: when errors have high costs, even rare occurrences need to be prevented, and using human supervision for AI control would outweigh the benefits of fully autonomous deployment.\n\nFinally, AI control needs to become a research field. It is quite surprising that this is not already the case. AI control is often treated as an application of known cybersecurity principles to AI, with a need to build better tooling. But when the [adversary is an AI agent](https://www.apolloresearch.ai/governance/misaligned-ai-as-a-new-insider-risk) (or a swarm of them), rather than a human, the game changes. A few examples:\n\n- Sandboxes that are effective against human adversaries have proven inadequate for controlling AI. How can we build extra-hardened sandboxes? Perhaps it’s time for more research on formal verification techniques.\n- “Security policies” for agents are often specified informally in prompts, and the resulting grey area is the cause of many alignment and control failures. Can we build a translation layer that turns humans’ natural-language intent into a formal specification that the humans can review and verify, and can then be automatically enforced? [6](#footnote-6)\n- As we’ve argued, monitoring agents with other agents is helpful and essential, but currently unreliable. Can we train models and agents to excel at the task of reliable monitoring?\n\nOne would think that the cybersecurity research community would have seized this moment with gusto. AI control is a source of interesting, important, and urgent problems, arguably more so than any development in the last few decades. Instead the community has reacted with a [collective yawn](https://x.com/joshua_saxe/status/2098632294923653295). Forget a new research field — there are close to zero papers in traditional cybersecurity conferences that tackle this new set of problems. (In contrast, prompt injection appears to have attracted a decent amount of research, but note a key difference: in that scenario, AI models are targets, not attackers.)\n\nIn other words, it appears that this community’s reaction to the incidents is that this is just a problem of implementation, not research. If so, something has gone seriously wrong. If AI control techniques sit still while AI’s offensive capabilities continue to advance, control will become less effective over time.\n\n## AI policy can incentivize AI control\n\nAI companies have two business incentives for investing in AI control: reputational costs and legal costs. On the former, businesses and individuals using AI models will hesitate to give autonomous agents access to consequential decisions if AI failures can take expensive actions or create liability. On the latter, AI companies might themselves face [liability](https://x.com/NateWitkin/status/2093724533647618171?s=20) for deploying AI agents that lead to safety failures.\n\nPerhaps as a result of these incentives, after the Hugging Face incident, OpenAI temporarily [stopped](https://x.com/joshua_saxe/status/2089847654213714025) evaluations on ExploitGym, stopped training and inference for the internal models, rebuilt compromised infrastructure, restricted research access to the underlying models, explicitly accepted costs to research and development speed, and increased investments in AI control.\n\nBut these incentives still leave gaps: customers often cannot assess safety, companies might ignore safety interventions because they expect the benefits outweigh the costs, and an unclear liability regime and the lack of standards in AI insurance could make it hard to assess the consequences for unsafe deployments.\n\nAI policy needs to systematically close these gaps. Building on the recommendations in [AI as Normal Technology](https://knightcolumbia.org/content/ai-as-normal-technology), we see three roles for policy for helping alleviate loss-of-control incidents: clarifying and strengthening liability, addressing market failures that liability does not resolve, and requiring transparency to improve public awareness and reduce uncertainty about AI’s impacts.\n\n**1. Companies should be responsible for harms caused by the agents they operate, even when they did not intend the harmful actions.** This responsibility should extend to internal uses during development and evaluation, not just to products released to customers. Existing tort law already provides avenues for liability, but [its application to AI leaves important questions open](https://www.rand.org/pubs/research_reports/RRA3243-4.html), including what precautions count as reasonable and when companies can be held responsible for unexpected behavior.[7](#footnote-7) Policymakers could, for example, clarify that running powerful agents without appropriate containment and monitoring is negligent. The fact that an agent was supposed to behave safely should not excuse the failure to implement controls against the possibility that it wouldn’t. Liability can also incentivize more investment in recruiting and training AI control specialists to prevent such incidents.\n\n**2. Market failures remain even when companies are liable, and policy must address them.** Even if companies are clearly liable for the harm caused by their use of AI, this still leaves the scope for market failures when [reckless action is not disincentivized by liability](https://hls.harvard.edu/bibliography/liability-for-harm-versus-regulation-of-safety/). For example, what happens if a company deploying agents recklessly is valued less than the harm it causes? One response is to require [insurance](https://aiuc.com/) as a mechanism to both [incentivize](https://arxiv.org/abs/2607.11999) safer deployment (to reduce premiums) and to compensate victims.\n\nA second example of market failures is when investments in safety have positive externalities (that don’t translate to revenues). For example, giving vulnerable cyberdefenders such as schools, hospitals, and critical infrastructure operators access to capable models can help them find and fix vulnerabilities, and benefits society as a whole. One way to incentivize this is to create [public support for defensive research and the adoption of defensive AI](https://knightcolumbia.org/content/ai-as-normal-technology). Policy should also enable coordination between AI companies and defenders, including ways for legitimate defenders to investigate attacks without being blocked by overbroad safety refusals. We discuss these defensive uses in Part 2.\n\n**3. Transparency interventions should make it possible for outsiders to identify risks to control and assess whether companies are managing them.** We proposed many interventions to improve transparency and reduce uncertainty about AI’s impacts in AI as Normal Technology, including [incident reporting](https://cset.georgetown.edu/publication/ai-incidents-key-components-for-a-mandatory-reporting-regime/), [independent auditing](https://www.averi.org/ourwork/frontier-ai-auditing), [whistleblower protections](https://www.njleg.state.nj.us/bill-search/2024/AR158), and [safe harbors for safety research](https://knightcolumbia.org/blog/a-safe-harbor-for-ai-evaluation-and-red-teaming). The threat of incurring liability can make companies [more reluctant to investigate and disclose risks](https://www.yalejreg.com/print/tort-law-at-the-frontier-of-artificial-intelligence/) if doing so could increase their legal exposure. Transparency requirements help prevent companies from responding to liability by concealing problems rather than fixing them.\n\nFor example, one objection to our proposals for AI control is that investing in control might make it unclear when “near misses” occur. Like in the Hugging Face incident, suppose OpenAI agents attempt to hack an external organization, bypass a few basic filters, and are only caught as a result of new/advanced monitoring techniques. This constitutes a “near miss”. Requiring incident reporting for near misses could allow us to understand where alignment failures continue to persist and what control mechanisms function well to prevent loss-of-control incidents. [Arnold and Llerena](https://www.lawfaremedia.org/article/when-reporting-an-ai-security-incident-is-not-mandatory) discuss other potential improvements to existing laws for transparency laws.\n\nHow should we enact transparency and auditing requirements? One proposed mechanism is to rely on [independent verification organizations](https://www.pnas.org/doi/10.1073/pnas.2509742123) (IVOs): government-authorized expert organizations that develop technical criteria and verify whether AI companies meet safety standards. For such scrutiny to be useful for AI control, evaluators would need meaningful access to the systems and evidence required to test whether controls work, including in internal research settings. While we are not attached to a particular institutional mechanism for enabling external access, we are cautiously optimistic that interventions to improve transparency, including IVOs, would allow the kind of oversight that is required to discover failures and assess the precautions that AI companies take.\n\n# Part 2: There is great uncertainty about how AI will impact cyberrisk. But what we need to do regardless is relatively clear.\n\nIn this Part, we argue that:\n\n- There is a crisis, but it is specifically about cybersecurity, not generally about AI safety, alignment, or loss of control.\n- Financially motivated cybercriminals — just like legitimate enterprises — face adoption barriers that make it hard for them to quickly adopt new AI capabilities to cause more harm. It isn’t clear how much of a boost agents will give them in the short term.\n- But this is no cause for comfort. There are many kinds of non-financially motivated threat actors including state-sponsored hacking groups, terrorists, hacktivists, vigilantes, and people motivated by personal grievances. There is also the possibility of rogue agents not under the control of any person.\n- Most of these are not rational actors and it is very hard to predict how much additional harm they will cause by taking advantage of advanced AI. Still, we must act as if attacks stemming from these sources will become very common.\n- Unfortunately alignment and control don’t help against bad actors using open-weight models, because those safeguards can and will be stripped, and the offensive capabilities of open-weight models tend to catch up [quickly](https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are-leading-open-weight-models-on-cyber) . We must rely on downstream defense and resilience, the latter two layers of our four-layer framework.\n- There have been many [warnings](https://www.ncsc.gov.uk/report/impact-ai-cyber-threat-now-2027) before this year, but until a few months ago, we lacked the urgency to invest adequately and move quickly. Fortunately, there are signs that that is changing. But there are important gaps.\n\n## The specific threat that is urgent is cyberrisk\n\nSuppose we agree that loss-of-control incidents are going to become common. So what? Of course, out-of-control agents could carry out autonomous cyberattacks, and we need to get better at defending against them, which we discuss below. But what else? We argue that there just isn’t a whole lot else that out-of-control agents can autonomously do to cause harm that malicious actors couldn’t. As a result, we should invest in defenses that would help regardless of whether the risk is from malicious actors or rogue agents.\n\nIn the superintelligence view, frequent loss of control means that the AI safety battle has already been lost. In addition to cyberrisk, AI could amplify biorisk, and even more broadly, “CBRN” risks, which stands for chemical, biological, radiological, and nuclear. Superintelligent AI might persuade people — such as those in charge of critical infrastructure or high-stakes decisions — into acting against their own interests. As AI becomes smarter and more useful, CEOs, politicians and others will have no choice but to rely on AI for consequential decisions, amplifying these risks. And as we get closer to the possibility of “recursive self-improvement” of AI, out-of-control AI could lead to a harmful intelligence explosion. In short, the view is that once the loss-of-control threshold has been crossed, the risks are endless and self-amplifying.\n\nWe strongly disagree. In AI as Normal Technology, we argued that this view of AI fundamentally misunderstands the nature of intelligence (including human intelligence). Building on that perspective, we argue that cyberrisk is special in two ways: in that superhuman capabilities are even possible, and because it is a purely digital risk with no physical-world bottleneck.\n\nCyberrisk has many specific properties that make superhuman offensive capabilities achievable (and largely already achieved). In general, some capabilities are like “making a building taller” — there is no real limit to achievement. Others are like “making a ball rounder” — there is a limit. One major point of departure between our worldview and the AI safety worldview is that we think tasks like chess or cybersecurity that are like making a building taller are the exception rather than the norm. In cybersecurity, like chess, there is an endless amount of training data (and more can be generated on demand), tasks (vulnerabilities / problems) can be arbitrarily complex, and speed makes a big difference.\n\nIn contrast, turning to the other risks, we stand by our prediction that superhuman persuasion ability is largely a myth. The confusion arises from carelessness about the term persuasion. Even GPT-4, an ancient model, is [superhuman](https://www.nature.com/articles/s41562-025-02194-6) in certain situations at persuading people on political and social issues by presenting evidence. But this implies nothing about whether a model can “persuade” — trick — a trained operator to give up nuclear codes.\n\nSimilarly, AI’s impact on [biorisk](https://secondthoughts.ai/p/why-arent-bioweapons-common) is a legitimate concern, but for the foreseeable future it is more in the vein of [democratizing](https://x.com/RuxandraTeslo/status/2094000557472043153) capabilities that already exist rather than [developing](https://x.com/DavidRBellamy/status/2099187370407112758?s=20) new superhuman capabilities. As for people in positions of power delegating consequential decisions to AI, our perspective remains that this is [extremely problematic](https://www.normaltech.ai/p/what-will-be-left-for-us-to-work) regardless of loss-of-control risks, and must be resisted.\n\nIn short, our view is that (1) cyberrisk demands special, urgent attention (2) more broadly, anticipating and defending against specific risks is better than trying to solve “AI safety” as a catchall (3) there is a big difference between two concepts that are often conflated: loss of control over individual agents (or swarms of agents) and some sort of (usually ill-defined) civilizational loss of control over AI as a whole. Even in a scenario where the former is common and normalized, we have many levers to avoid the latter.\n\nSo let us turn to what we think is the critical question for now, which is how autonomous cyber capabilities will affect the attacker/defender balance. If we can ramp up defensive measures to keep that balance under control, we don’t have to panic about loss-of-control per se.\n\n## How will autonomous cyber capabilities and open-weight models affect the attacker/defender balance?\n\nWe think the cyber capabilities of today’s frontier models will diffuse to open models soon, as they have repeatedly so far. A worldwide ban on open-weight models is highly unlikely, and at any rate, defenders must prepare under the assumption that it won’t happen.[8](#footnote-8) We must assume that today’s frontier cyberoffense capabilities are widely available to threat actors in a matter of months, not years.\n\nAlignment and control interventions only work when used by legitimate actors, since guardrails on open models can be stripped. So we think addressing risks from bad actors using advanced offensive AI capabilities is the grand challenge for cybersecurity.\n\nLet’s start with cybercriminals — financially motivated actors. Assuming that the cyber capabilities of powerful open-weight models continue to improve rapidly, and that they will be available without guardrails, how much will it help cybercriminals? There are two key variables: how quickly cybercriminals will adopt these tools, and whether AI helps overcome critical barriers or if it impacts more peripheral aspects of their workflow.\n\nThese two questions are strikingly similar to the questions that pertain to how much AI will help economic growth. The reason we are skeptical about rapid growth is because we think organizational barriers result in slow diffusion and because in many cases the part that AI accelerates [wasn’t](https://www.normaltech.ai/p/could-ai-slow-science) the [slow](https://www.linkedin.com/posts/randomwalker_amtrak-had-a-lot-of-fanfare-around-its-fancy-activity-7407459876484063233-tsui/) one to begin with.\n\nNow let’s apply that analysis to cybercriminals. They certainly don’t face the same level of organizational barriers as defenders. But it is not zero. And there is a surprising answer to the question of whether finding new exploits is a critical capability.\n\n## To understand what open-weight models mean for the future of cybercriminal activity, we need to understand the past correctly\n\nIn the early decades, the cybersecurity community framed cyber-defense as a technical problem — building and deploying bug-free systems. But this approach [didn’t work](https://www.acsac.org/2001/papers/110.pdf). We do have a lot of tools in our toolkit, including formal verification, safer programming languages, and penetration testing, but they are costly (in terms of programmer time, security analysts’ time, or software performance), so we don’t deploy them nearly enough for invulnerable systems to be a remotely realistic goal. Besides, human behavior in complex systems remains unpredictable and a weak link.\n\nInstead, we made progress by treating cybersecurity as a primarily *economic* problem — raising the cost for attackers and lowering benefits; and only investing in security up to the point where the (capturable portion of the) benefits are worth the costs. Specifically:\n\n- Bug bounties and patching culture reflect the assumption that consumers and enterprises are constantly running exploitable software; the defenders’ goal is instead to simply outbid and outrun attackers.\n- Cyber insurance takes this idea to the extreme. Firms buy insurance because driving the probability of breaches to zero is not economically achievable. The cost of recovery from inevitable breaches becomes a budget item that competes with the budget for preventive measures.\n- Defenders go after criminal revenue rather than technical attack capability. A famous economics-of-security insight in the late 2000s was that attackers are decentralized but payment networks are a centralized chokepoint, and going after these turned the tide on various kinds of scams.\n- Much security policy is now about reallocating liability because liability changes behavior. Notably, forcing banks to internalize the cost of ATM and credit card fraud forces them to invest in security up to the socially optimal amount.\n\nOverall this approach has been successful, and has driven cybercrime from a fundamental constraint on the digital economy to a problem that can be priced and managed. But running in permanently-insecure mode has its downsides. More on that in a minute.\n\n## The majority of cybercriminals are surprisingly low-tech, and the barrier is monetization, not exploitation\n\nThe defenders’ strategy of focusing on the economics — and the fact that cybercriminals are fundamentally economically motivated rational actors — means that the key constraint for attackers is monetization, not exploitation. Breaching is not the hard part — it’s what to do *after* the breach. Over the decades, cybercriminal organizations have evolved toward excellence in monetizing compromised victims, not their ability to find vulnerabilities. The vulnerabilities that they exploit are [largely](https://www.microsoft.com/en-us/security/security-insider/threat-landscape/microsoft-digital-defense-report-2025) [mundane](https://www.sophos.com/en-us/press/press-releases/2025/04/sophos-report-56-sophos-ir-and-mdr-cases-adversaries-logged-instead) — phishing/social engineering, unpatched servers, exposed remote services, compromised credentials — and when they do need zero-day vulnerabilities, they purchase them instead of developing them in-house.\n\nAs one small but illustrative example, credit card networks’ hostility to criminal activity forced cybercriminals to turn to cryptocurrency. In fact, crypto was essential for the success of ransomware, today’s dominant form of malware. But ransomware gangs found that victims were often confused about how to pay using crypto, and needed reassurance that payment would result in their data being returned. This required the gangs to set up customer service centers! The parallels between cybercriminal organizations and legitimate businesses don’t end there, as discussed in the paper [cybercrime is entrepreneurship](https://weis2021.econinfosec.org/wp-content/uploads/sites/9/2021/06/weis21-anderson.pdf).\n\nIn short, the bottlenecks to scaling tend to be high-touch and human. This explains why the diffusion of cutting-edge technology tends to be slow. As [Joshua Saxe](https://www.chinatalk.media/p/cyber-apocalypse-now) puts it: “There’s no reason to expect a ransomware gang in Belarus to adopt AI faster than Deloitte does in its consulting work.”\n\nThis insight also applies to the attack surface opened up by AI itself. Many commentators (including us!) have been predicting that we’ll start to see lots more prompt injection attacks in the wild, but they [haven’t panned out](https://joshuasaxe181906.substack.com/p/where-are-all-the-prompt-injection) so far. One possible reason is that this is way too high tech for mainstream attackers to go after.\n\n## So, how much will AI help cybercriminals?\n\nWe just don’t know yet. It could go both ways.\n\nOn the one hand, the status quo might continue; cybercriminals might see little reason to adopt cutting-edge tools, for the reasons described above. We have long observed that AI helps attackers’ *productivity*, just like it helps everyone else’s, and that seems to be the effect so far. At least as of [last year](https://cloud.google.com/blog/topics/threat-intelligence/adversarial-misuse-generative-ai), even state-backed actors were using AI for things like “researching potential infrastructure and free hosting providers, reconnaissance on target organizations, research into vulnerabilities, payload development, and assistance with malicious scripting and evasion techniques”, rather than automating the entire “cyber kill chain”. A productivity uplift is quite different from new capabilities which is in turn different from a capability that is so radically new that it changes the economics of the operation. As we write this, Anthropic has released a new report on [countering the misuse of AI](https://www-cdn.anthropic.com/e50be2e51e7695dc4b1366a37a245a597377d3b5/Anthropic-Detecting-and-countering-091026.pdf), with five case studies on advanced cyber operations, and it is not yet clear to us how to classify these and how widespread a trend these case studies represent.\n\nOn the other hand, maybe the productivity uplift will compound over, say, a 5-year timeframe to make attacks much more pervasive (though, of course, defenders won’t be standing still). Automated exploitation could tilt the balance toward mass exploitation without putting much effort into monetizing each target. This has been tried before. The fraction of victims who end up paying tends to be low, sometimes as low as [2.5%](https://coveware.com/2026/02/why-zero-day-downstream-mass-data-extortion-campaigns-are-losing-their-bite/), but maybe at sufficient scale even that is worthwhile. Agents could help with the high-touch human bottlenecks discussed in the previous section, though this has little to do with their exploitation capabilities. Finally, the ease of exploitation could return us to the bad old days of [botnets](https://darioamodei.com/post/we-must-pace-the-frontier), when cybercriminals would pre-emptively compromise thousands of enterprises or millions of consumer devices and maintain persistent access in advance of figuring out how to monetize them.\n\nEven if AI doesn’t end up helping cybercriminals much, there are very different kinds of threats to worry about.\n\n## Are we about to see harmful autonomous agents?\n\nThe OpenAI agents were ultimately attempting to accomplish the task they had been set, although they went about it in harmful ways that weren’t intended by OpenAI. And though they are often referred to as “rogue”, they did not exfiltrate their own weights from OpenAI’s servers (and presumably had no ability to do so), meaning that OpenAI was always in a position to shut them down.\n\nIn contrast, researchers have proposed the idea of [“self-sovereign” agents](https://self-sovereign-agent.github.io/paper.pdf) that are autonomous in a much more fundamental sense: they control the economic resources necessary for their own operation. For example, they might perform tasks on online gig work marketplaces and use the revenues to rent the GPUs necessary to run themselves. There are also many related ideas such as [LLM-based worms](https://arxiv.org/abs/2606.03811v1).\n\nPutting these ideas together, a clear picture emerges: in the future, there might be harmful agents that can’t be shut off because no one controls their weights (maybe the agent keeps exploiting vulnerabilities to copy itself faster than it can be detected and shut down), and probably also because the agent controls its own wallet, likely cryptocurrency.\n\nThere are many ways in which [harmful autonomous agents](https://www.hyperdimensional.co/p/on-the-loose) might arise, but they break down into roughly two categories: an agent that’s deliberately created and let loose, or a misaligned agent that escapes its sandbox. Clearly, persistent, autonomous rogue agents would create new kinds of cyberrisk. The former type of rogue agent might be explicitly instructed to hack as many targets as possible, and the latter might choose to do so as a way of earning money in order to ensure its own persistence.\n\nAs sci-fi as these scenarios seem, we agree with AI safety researchers who consider them a realistic near-term possibility. The capabilities required are within reach, and any alignment or control mechanisms can of course be undone by someone deliberately seeking to create such rogue agents.\n\nBut once again, we ask: if we have AI agents on the loose, so what? What exactly are these agents going to do? They might hack a bunch of stuff; let’s start defending those now. They might try to start poisoning future training runs; we should defend against data poisoning regardless of whether it comes from agents or malicious actors. They could try to use [cloud labs](https://www.rand.org/pubs/perspectives/PEA3851-1.html) to create pathogens; we should implement controls and [screening](https://screendna.org/) for orders to such labs. But in all of these cases, the risks come back to what bad actors can already do (with or without AI), and the rogueness isn’t a problem by itself.\n\nSome commentators have linked rogue agents to the possibility of a “secret intelligence explosion” (for example, rogue agents building even more capable agents). But we see such extensions as categorically different. It is true that agents might be able to easily persist because *inference* is cheap and getting cheaper. In fact, if and when we have rogue agents that can run on consumer hardware, they don’t need a budget at all, and don’t need to control a wallet. But training still requires orders of magnitude more compute, so it will be a while before this becomes a problem.\n\nSecond, fortunately we have an economic chokehold that can help minimize the damage. We can require Know-Your-Customer processes for cloud compute (and especially for training-scale compute). This will not prevent rogue agents from existing but can greatly mitigate their prevalence and harmfulness. The [paper](https://self-sovereign-agent.github.io/paper.pdf) on self-sovereign agents recommends “economic frictions for fully automated participation”. We agree. If frictions like bot detection, rate limiting, and human-in-the-loop verification (like CAPTCHAs, but more effective) are pervasively deployed, it can greatly limit how far rogue agents can get.\n\n## Why the Morris worm is a good analogy for what might be about to happen\n\nSo far AI seems to have helped defenders more than attackers, in line with previous waves of automated vulnerability discovery. But it is possible that we are seeing the start of a new paradigm that will upset the equilibrium, at least temporarily. It is too early to tell.\n\nA possibly good analogy is that this is like the [Morris worm](https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1792) moment for security (it was the first prominent computer worm, back in 1988). Consider what worms enabled, compared to previous ways of compromise:\n\n1. Attackers could greatly magnify their power because of the worms’ self-replication.\n2. Attackers need not be financially motivated. Many worms, including the Morris worm, went accidentally viral when their creators were playing around with them, without even intending to cause harm.\n3. The speed of propagation was so fast that defenses (like applying patches to vulnerabilities) had to be partially automated.\n4. Because the adversary is not a rational cybercriminal but an unpredictable hobbyist, no one could rely on the hope that they were unlikely to be attacked. (It was only much later that the baseline level of security improved to the point where worms were no longer the predominant threat, but rather targeted exploitation by cybercriminals.)\n\nAll four of these factors are present, to varying degrees, in today’s plausible new era of fully automated cyberattacks. If anyone can start up malicious agents on a whim, and isn’t looking to make money, then millions of organizations that were previously not lucrative enough to target now become vulnerable, as are billions of individuals who previously weren’t targeted because there just weren’t enough bad guys to go around.\n\nTo be clear, this has not started happening yet. But if it does, it would radically upset the economic equilibrium that has persisted for most of the past two decades.\n\n## What the future of cyberdefense might look like: the roles of people and defensive AI\n\nTwo things seem likely: AI will be extremely useful and necessary for defense, but we’re going to need a lot more *people* working in AI-centric cybersecurity. There’s an attacker-defender asymmetry: an attacker with little technical expertise can use even unreliable AI tools and hope for the best; a defender cannot. That’s because the attacker only needs to succeed once, but the defender needs to succeed every time.\n\nAI has three main potential roles in cyber-defense:\n\n- **Pre-emptive hardening** : The amount of work needed to bring enterprise systems up to speed using known security techniques is simply immense — longstanding bugs, excessive permissions, misconfigurations, unpatched software, weak authentication, and legacy tech debt. AI is definitely useful here, though human expertise will remain front-and-center.\n- **Detection, triage, and forensics** : Both Hugging Face’s own detection and analysis of the OpenAI attack and METR / Redwood’s analysis of the incident relied heavily on AI. There are concerns about the reliability of AI for these tasks, but we suspect it will quickly become essential.\n- **Autonomous, real-time defense** : This remains very controversial in the practitioner community. Time will tell if the speed of AI attacks will necessitate this, or whether humans will remain in the defensive loop.\n\n## What are the gaps in the state of cyberdefense against AI?\n\nFor effective cyberdefense, we need people on the frontlines, access to defensive AI, investment in defense, and policy to steward it all. Let’s look at the state of each of those four.\n\n**Workforce**\n\nWhat it means to be a cybersecurity professional will be increasingly AI-centric. Surprisingly, there are well over a million cybersecurity professionals in the U.S. alone. At first this seems like great news: there is an army of people ready to take on the task of defending systems from autonomous cyberattacks. But the vast majority of these professionals are doing governance, operations work, and [compliance audits](https://www.paloaltonetworks.com/cyberpedia/soc-2), which involve a very different skillset, rather than adversarial security work. So it is not clear to what extent the necessary infusion of expertise can be achieved by retraining current professionals versus recruiting new “AI-native” security people. This workforce lag is arguably one of the most worrying things about the current moment. Many security professionals are [surprisingly unenthusiastic](https://medium.com/anton-on-security/rsa-2026-agentic-future-analog-fundamentals-the-paradox-of-why-the-old-guard-still-survives-bf93e81eaaa6) about defensive AI.\n\n**Access to defensive AI**\n\nWhen Hugging Face tried to use U.S. frontier models in responding to the OpenAI incident, safety guardrails kept blocking them, because the guardrails are bad at distinguishing between offensive and defensive work. This well-publicized anecdote, and others like it, have led to concerns about access to defensive AI. Have frontier AI companies kneecapped defenders through their safety policies? Hugging Face was fortunately able to use an open model (GLM 5.2), but it is weaker than frontier models. And if restrictions are imposed on open-weight models in the name of safety, will that only further harm defenders?\n\nThe overactive nature of safety classifiers is a choice made by U.S. companies, perhaps in part because of the Trump administration’s response to Mythos. Fortunately, it appears that in recent weeks and months they have been [moving](https://www.anthropic.com/claude-fable-and-mythos-5-1) [quickly](https://openai.com/index/path-to-astra/) to rectify them, as well as [expand](https://openai.com/index/expanding-daybreak-as-the-cyber-defense-window-narrows/) [access](https://www.securityweek.com/anthropic-expands-mythos-5-access-to-more-defenders-unveils-35m-open-source-fund/) to ‘[cyber](https://blog.google/innovation-and-ai/technology/safety-security/fairwind-program/)’ versions of frontier models that can be used by defenders. Overall, it seems likely that the hiccups involving defenders’ access to models will be resolved. Open models are definitely useful as a resilience mechanism, but we don’t see recent events as providing strong support for the position that they will be the only option (or even the primary option) for defense.\n\n**Investment**\n\nThere have been a series of investments and announcements in cyberdefense ever since the announcement of Anthropic’s Mythos model in the spring, which led to [Project Glasswing](https://www.anthropic.com/glasswing), a cyberdefense initiative that comes with up to $100 million in usage credits. The biggest announcement to date is OpenAI’s “[Daybreak for frontline defenders](https://openai.com/index/daybreak-for-frontline-defenders/)” initiative which provides $1 billion in the form of “subsidized access to frontier cyber capabilities, hands-on training and technical assistance, and new partnerships to get those capabilities to organizations”.\n\nAt first these are eye-catching numbers. The problem is that it is not clear how much goes toward closing the expertise and workforce gap. It seems that the $1 billion refers to [credits](https://openai.com/daybreak/) and not cash. The cash investments that have been announced are much lower, typically about a [few million](https://www.isocfoundation.org/grant-programme/common-good-cyber-fund/). One exception is the AI cybersecurity observatory which reportedly has a commitment for “[eight-figure seed funding](https://x.com/joshua_saxe/status/2095162541915754588)”, though the organization’s purpose is measurement and forecasting more than defense.\n\nOverall these numbers don’t seem like the right order of magnitude given the seriousness of the threat — which, again, is a potential disruption of the economic cybersecurity equilibrium that has allowed e-commerce and the entirety of digital society to exist and flourish.\n\n**Policy and government leadership**\n\nSo far, government leadership on this issue, whether in the U.S. or elsewhere, has been underwhelming. Far from an increase in investment, the U.S. Cybersecurity and Infrastructure Security Agency (CISA) is set to *[lose](https://www.warner.senate.gov/newsroom/press-releases/warner-raises-alarm-on-cisa-workforce-and-budget-cuts-that-are-leaving-our-country-vulnerable-to-threats/)* $700 million from its 2027 budget. Its staff has already been cut by a third, including cuts in cyber defense training.\n\nBut should we even look to governments, given their chronic lack of speed, or should the response be primarily led by companies, nonprofits, and philanthropy? For all its faults, we think the government still has an important role to play, because cybersecurity is a public good.\n\nOverall, governments seem to be approaching cyberdefense (as they do almost everything else) through the lens of “normal” policymaking. The evidence of urgency that has accumulated so far doesn’t really register because there has been little materialized harm.\n\nTo be clear, our view remains that AI risks don’t justify [extraordinary government interventions](https://knightcolumbia.org/blog/do-ai-risks-require-extraordinary-government-intervention), which are precautionary, freedom-reducing, and bypass usual governance processes. But in between the lethargy and reactive nature of normal policymaking and the urgency and unilateral nature of “extraordinary government interventions” lies a vast space of approaches that we suspect we and many of our critics would agree on. We think there’s room for somewhat technocratic interventions in which information about emerging large-scale risks gets translated into defensive investments and resilience (layers 3 and 4 of our table). These do not require unilateral authority or restrictions on freedom.\n\n# Part 3: What *AI as Normal Technology* got wrong — and right — about AI safety\n\nThe recent loss-of-control incidents resulted in our first major update to our views on safety, so it is worth spelling them out in detail. But first, a few preliminary remarks:\n\nFirst, this update to our views is about safety. AI as Normal Technology has two [loosely connected](https://knightcolumbia.org/blog/do-ai-risks-require-extraordinary-government-intervention) parts; the other one is about the economy (the connection is loose because legitimate and illegitimate uses of AI are governed by different constraints). As an explanation of slow economic impacts, we think the essay has held up [extremely well](https://www.normaltech.ai/p/why-ai-hasnt-replaced-software-engineers), and there has been a general recognition that the AI industry was wrong about rapid diffusion and job loss. Sam Altman was explicit in a recent podcast:\n\nThe economy just has so much inertia. ... We’ve all been too ambitious on timelines … Society and the economy will adapt more slowly.\n\nHe views this as good news, because it will make the AI transition “smoother and slower.” That was exactly our main point.\n\nSecond, many of our views on safety that were minority positions at the time are now taken for granted, such as the fact that AI is useful for defense as well as offense, and the relevant question is how it shifts the attacker-defender balance. Many of the basic distinctions we introduced, such as the difference between capability and power, or the [four dimensions of advanced AI](https://www.normaltech.ai/p/what-will-be-left-for-us-to-work), remain important but under-recognized in the safety discourse.\n\nThird, most AI safety policies we argued for in the essay remain necessary and have increased in urgency. One goal of *AI as Normal Technology* was to move the policy conversation forward by pointing out that we do *not* need consensus on worldviews to have agreement on policy. We stand by our policy proposals in the essay, including transparency requirements, auditing, safe harbors for safety researchers, and whistleblower protections.\n\nFourth, we have been putting our time where our mouth is. Reducing uncertainty about the pace of AI’s impacts remains crucial. A significant portion of our current empirical research is focused on testing the cruxes between the AI as Normal Technology worldview and the superintelligence worldview, and we plan to continue this research.\n\n## Do risks arise from development or deployment?\n\nWe wrote: *We see both benefits and risks as arising primarily from AI deployment rather than from development; thus, the speed of progress in AI methods is not directly relevant to the question of impacts.*\n\nThe AI-enabled harms that have materialized so far have been overwhelmingly from deployment — things like AI-generated nonconsensual nudes, which we have long maintained to be far more harmful than the risks that dominate the headlines.\n\nBut we clearly missed *internal deployment for evaluation* as a potential source of risks. This is particularly embarrassing for us since we ourselves run an [AI evaluation research group](https://sage.cs.princeton.edu/), and we’ve observed versions of these risks in our own work even before we wrote *AI as Normal Technology*.\n\nGiven the importance of this risk, regulating developers is more important than we made it out to be in the original essay. Our updated understanding is reflected in this essay, notably in the section on how policy can incentivize better AI control.\n\n## How should we deal with rogue agents?\n\nAI as Normal Technology puts much more stock into defenses located downstream of models (what we call layers 3 and 4 in the language of this essay) than alignment or control. [Elsewhere](https://asteriskmag.com/issues/10/does-ai-progress-have-a-speed-limit) we have taken the position that “We should assume every model will be capable of escape and self-reproduction. Safety shouldn’t rely on that being difficult.” In other words, it is better to assume that there will be harmful autonomous agents and pre-emptively harden our defenses against them, than to try to prevent such agents from ever existing.\n\nIn light of recent events, it seems much clearer that this is the better strategy.\n\n## The continuity hypothesis\n\nAn important pillar of our pushback against catastrophic risks is that we will have a series of smaller-scale warnings before risks progress to a catastrophic level, giving us opportunities to anticipate those risks and improve our defenses. Continuity is [partly](https://asteriskmag.com/issues/10/does-ai-progress-have-a-speed-limit) an inherent property and partly a result of the choices we make.\n\nWe see the recent incidents as evidence in support of this hypothesis. The harms that resulted from these attacks are many, many orders of magnitude smaller than other AI harms we already tolerate. More importantly, while the agents in question excelled in some dimensions, they were incompetent in others (such as hiding their traces) and didn’t have properties that would make harms more serious and likely. Notably, based on the published analyses, the agents’ “intent” was simply to solve the benchmark tasks they were given, and they did not start pursuing their own goals. Thus, as serious as these incidents were, in our assessment, we are multiple qualitative changes away from catastrophic cyberrisks.\n\nWe are aware that many AI safety researchers have interpreted these incidents as evidence of *discontinuity*. That is an understandable reaction. These incidents are much more severe than anything else that had previously been publicly revealed. It is a perennial feature of AI safety debates that different groups interpret new evidence in ways that are consistent with their preferred worldviews. There is no objective standard for “how continuous is continuous?”\n\nTwo more points about continuity. We have emphasized that we can and must *engineer* continuity rather than take it for granted. Unfortunately, it is too easy to imagine a world in which none of the recent incidents had come to light. We remain dependent on voluntary transparency from frontier AI companies. [Embedded evaluation](https://darioamodei.com/post/we-must-pace-the-frontier) is a great step forward, but it should not be voluntary. We have long been in support of much stronger transparency legislation and reaffirm that view here.\n\nContinuity is important to our worldview because “Long before a system would be granted access to consequential decisions, it would need to demonstrate reliable performance in less critical contexts. Any system that interprets commands over-literally or lacks common sense would fail these earlier tests.”\n\nWe’ve now had significant examples of systems that fail these prerequisites. This is the crux. How will AI developers — and their customers, and policy makers, and society at large — react to the recent loss-of-control incidents? That reaction will determine whether or not the second part of the continuity hypothesis (the ability to respond to smaller-scale warnings) continues to hold. We consider it a critical test for the usefulness of the AINT framework for AI safety.\n\nWe were too confident in AI companies’ ability to take basic control precautions when running evaluations. We were surprised by the lack of organizational maturity within leading AI companies that led to these incidents. Our expectation was that existing legal liability, imperfect as it is, and the risk of brand damage would be a sufficient antidote to such organizational practices. We were wrong. This reinforces the need for policy interventions to move the needle.\n\n## Jaggedness and risk-specific defenses\n\nBroadly speaking, *AI as Normal Technology* argues for slow timelines. This is primarily because of bottlenecks to *diffusion*, but the essay also has a tentative section that talks about speed limits to progress in AI methods and advances in capabilities. These limits exist despite the possibility of recursive self-improvement. [We take RSI seriously](https://cruxevals.com/crux/can-ai-agents-conduct-research), but we think many of the bottlenecks to superintelligence are [external](https://www.normaltech.ai/p/what-will-be-left-for-us-to-work) and won’t be overcome by improving computation.\n\nThat said, we underestimated the importance of [jaggedness](https://helentoner.substack.com/p/taking-jaggedness-seriously). Not only is AI’s capability profile at any given moment highly jagged, so is the *rate* of improvement in different domains. Combined with the fact that deployment barriers for harmful uses are often lower than for economically productive uses, risks such as in cybersecurity can arise rapidly, leading to periods where the offense-defense balance is upset since defenders might not have time to adapt. Thus, we are no longer confident in our earlier [prediction](https://www.youtube.com/watch?v=AV7PdlWyGHI) that the offense-defense balance will continue to hold even with advances in AI.\n\nAt the same time, jaggedness only strengthens our view that it is more important to mitigate specific risks rather than trying to solve all safety risks through alignment. While risks might arise rapidly, they do so in specific areas, and we should be prepared to rapidly strengthen our defenses in those areas — and ideally do so pre-emptively. For example, there are known interventions that would help mitigate the biosecurity risks from advanced AI models, such as [DNA synthesis screening](https://screendna.org/), which we should invest in now.\n\n## Is AI safety on track?\n\nBecause many people are only familiar with the title of the *AI as Normal Technology* thesis and not its contents, we are often mistaken as downplaying AI risks, though we have repeatedly clarified that that is not our position. Still, it is important for us to be explicit about how much urgency there is.\n\nBecause the question is a bit fuzzy, we have said different things at different times. We’ve expressed views along the lines of “There’s work to do, but we are on track” when it comes to AI safety. But in other writing, including in the *AI Snake Oil* book many years ago, we’ve been clear that we’re bad at reacting to even normal general-purpose technologies and that even before AI, we chronically underinvest in protections against catastrophic risks, such as pandemics, which are arguably amplified by AI.\n\nIt is important to reiterate that message. Despite recent increases in investment, we still don’t think there is nearly enough investment in cyber-defense. Biorisk defense investment also remains woefully inadequate. Similarly, there hasn’t been much progress in the policy interventions we called for in AINT, and reaffirmed in the essay on [Common Ground between AI 2027 & AI as Normal Technology](https://asteriskmag.substack.com/p/common-ground-between-ai-2027-and).\n\nSo while we continue to reject the alarmism, and don’t believe that catastrophic risks are imminent, we do think risks are increasing over time because our investments and policies aren’t keeping pace. And while the specific types of investments and policies we call for overlap only partly with that of the safety community, we agree that we are not currently on track.\n\n# Conclusion\n\nReactions to the recent loss-of-control incidents have split the AI safety and cybersecurity communities. How should we reconcile these starkly different responses to the same incident? One way is to consider the central role of *worldviews* in shaping peoples’ opinions. As we wrote in the AI as Normal Technology essay:\n\n*AI as normal technology is a worldview that stands in contrast to the worldview of AI as impending superintelligence. Worldviews are constituted by their assumptions, vocabulary, interpretations of evidence, epistemic tools, predictions, and (possibly) values. These factors reinforce each other and form a tight bundle within each worldview.*\n\nWhen we identify the role of worldviews, the factors underlying these differences in opinion become clear. The AI safety community saw the incident through the lens of their long-running predictions about misalignment risks coming true. These risks had been dismissed as sci-fi scenarios by many people, and the community felt it was vindicated by these early incidents and used this opportunity to push for action by companies.\n\nOn the other hand, the cybersecurity community compared the incident to the scale and scope of cyberattacks that they regularly deal with. It was clear that AI companies had not adopted a security mindset, nor had they implemented known control interventions when conducting these evaluations. In other words, the security community was taken aback when an incident that clearly resulted from negligence and largely did not lead to real-world harm led to so much commotion in the AI community.\n\nWe applaud both communities for trying to hold AI companies accountable. But a big risk from these polarized responses is that communities form [epistemic bubbles](https://www.normaltech.ai/p/ai-existential-risk-probabilities) and don’t seriously engage with the other side’s opinions. If the two communities fail to find common ground, they won’t be able to develop technical interventions and adopt practices that would clearly improve our ability to alleviate these risks.[9](#footnote-9)\n\nIn this essay, we try to offer that middle ground. We synthesize lessons from both communities and offer a path forward. If we act with the appropriate urgency, we can hold AI companies to a higher standard, reduce risks from loss of control, and even tilt attacker-defender balance in cybersecurity [back towards defenders](https://secondthoughts.ai/p/playing-cyber-to-win).\n\n**Acknowledgments**. We are grateful to Joshua Saxe, Steve Newman, Stephan Rabanser, Justin Curl, and Rod Moshtagi for feedback and discussions that informed this essay. [Joshua Saxe’s writing](https://joshuasaxe181906.substack.com/) on this topic has been particularly influential to our thinking.\n\n**Further reading**\n\n- How should AI companies operationalize AI control? [Guidelight](https://guidelight.ai/) has proposed AI control standards for frontier AI companies to implement existing/known practices. Had OpenAI adopted these practices, it is extremely unlikely the Hugging Face incident would have occurred.\n- [Cohen and others argue](https://www.ballardspahr.com/insights/alerts-and-articles/2026/08/ai-gone-rogue-what-recent-openai-and-anthropic-ai-incidents-could-mean-for-cfaa-liability) that AI companies could face criminal and civil liability under CFAA for evaluations where they disable safeguards and it consequently leads to real-world harm.\n- Researchers and practitioners in the AI safety and cybersecurity communities have written a number of essays about these incidents. Essays from the AI safety community include [ [1](https://thezvi.substack.com/p/metr-and-redwood-offer-holy-postmortem) ], [[2](https://www.planned-obsolescence.org/p/the-hugging-face-attack-surprised) ], [[3](https://blog.redwoodresearch.org/p/are-we-existentially-threatened-by) ] and the cybersecurity community include [[4](https://www.iansresearch.com/resources/all-blogs/post/security-blog/2026/08/28/openai's-postmortem-recasts-the-hugging-face-breach-as-an-incident-response-failure) ], [[5](https://www.linkedin.com/feed/update/urn:li:activity:7493061037823492096/) ], [[6](https://www.economist.com/by-invitation/2026/08/23/fears-of-ai-induced-armageddon-are-overdone) ].\n\n[1](#footnote-anchor-1)\n\nFor example, in [Zvi Mowshowitz’s discussion of the incident](https://thezvi.substack.com/p/metr-and-redwood-offer-holy-postmortem), he writes: “The biggest failure, the one that counts in the end, was that the models were severely misaligned”\n\n[2](#footnote-anchor-2)\n\nAs an example, consider that nearly every single top-level comment on the [Hacker News discussion](https://news.ycombinator.com/item?id=49666735) about a recent OpenAI incident, where its agents attacked RubyGems, is about how the company was negligent.\n\n[3](#footnote-anchor-3)\n\nThis is not to say companies haven’t invested in control. There have been many [examples](https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/securing-the-future-of-ai-agents/gdm-ai-control-roadmap.pdf) of control efforts across AI companies. But so far, they seem to pale in comparison to the effort being put into alignment.\n\n[4](#footnote-anchor-4)\n\nIn fact, as we were working on this essay, Harvey AI, a legal AI company, [acquired](https://www.harvey.ai/blog/guardrails-ai-joins-harvey) Guardrails, a company that builds control interventions for AI. Their main focus is improving AI control interventions for legal settings.\n\n[5](#footnote-anchor-5)\n\nIn our own work we’re seeing [log analysis](https://arxiv.org/pdf/2605.08545) emerge as a specialized skill with some members of our team excelling in it. Of course, real-time monitoring will require a slightly different and broader skillset compared to retrospective log analysis.\n\n[6](#footnote-anchor-6)\n\nThe “auto mode” in coding agents is a reasonably successful instantiation of the natural-language-as-security policy idea. It is striking that despite this, it has attracted very little principled research.\n\n[7](#footnote-anchor-7)\n\nThere is an ongoing debate about what improvements to AI liability policy should look like. Some arguments emphasize [what the existing tort system can already accomplish](https://cityjournal.substack.com/p/an-old-school-solution-to-the-new). Others advocate [strict liability](https://law-ai.org/tort-law-should-be-the-centerpiece-of-ai-governance/), which would not require victims to prove negligence, or [strengthening negligence law and combining it with independent oversight](https://www.yalejreg.com/print/tort-law-at-the-frontier-of-artificial-intelligence/). We don’t intend to settle these disagreements here; our point is simply that clarifying and strengthening liability for AI agents can make investments in control a business priority.\n\n[8](#footnote-anchor-8)\n\nThe [letter](https://images.nvidia.com/pdf/Open-Weights-and-American-AI-Leadership.pdf) supporting open models by a coalition of American AI companies was published only a week after the Hugging Face hack was disclosed. Whatever one thinks normatively, descriptively it seems clear that no demonstration of cyber-capabilities is so striking that it will get people to shift their views toward non-proliferation of open-weight models — if anything, the opposite.", "url": "https://wpnews.pro/news/the-ai-as-normal-technology-view-of-loss-of-control-incidents", "canonical_source": "https://www.normaltech.ai/p/the-ai-as-normal-technology-view", "published_at": "2026-09-14 10:30:14+00:00", "updated_at": "2026-09-14 10:36:13.212230+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-policy", "ai-ethics"], "entities": ["OpenAI", "Anthropic", "Hugging Face", "Dario Amodei", "METR", "Rubygems"], "alternates": {"html": "https://wpnews.pro/news/the-ai-as-normal-technology-view-of-loss-of-control-incidents", "markdown": "https://wpnews.pro/news/the-ai-as-normal-technology-view-of-loss-of-control-incidents.md", "text": "https://wpnews.pro/news/the-ai-as-normal-technology-view-of-loss-of-control-incidents.txt", "jsonld": "https://wpnews.pro/news/the-ai-as-normal-technology-view-of-loss-of-control-incidents.jsonld"}}