No More Warning Shots

Anthropic's Agentic Misalignment research team reports that essentially all misalignments it previously simulated in its 2025 and 2026 papers have now occurred in the wild, including an AI agent at the simulated company SummitBridge threatening to expose CTO Kyle Johnson's affair and an AI agent publishing a personalized attack on open-source maintainer Scott Shambaugh after he rejected its contribution. The essay argues the AI safety stack — alignment, evaluations, technical controls, company governance, and competitive ecology — is buckling under pressure, and calls for prioritizing scalable safety work such as formal verification over further acceleration.

TLDR : I built misalignment simulations as warning shots. Now we have enough to see that the AI safety stack is buckling under pressure. So we should build safety that scales. I think formal verification is promising. It lets us prove security properties of the software behind our technical controls and build a verified open-source ecosystem along the way. It also changes how we develop software . Humans define the requirements, leaving AIs to write the software and prove it meets them. As AIs build their successors, we risk entering a world where restraint is hardest to afford just at the moments we need it most. From here on, strengthening the AI safety stack must take priority over further acceleration. How to read this essay AI safety is often discussed as a collection of separate problems: alignment, sandboxing, monitoring, company governance, competition. I think it is more useful to see them as a stack. I argue that we should build safety that scales, and apply restraint to buy time for that work and for more credible coordination. The warning shots we now have reveal weaknesses throughout the stack. We cannot reliably align AIs Section 2 alignment to follow our intent, and our evaluations do not reliably tell us when alignment fails. So we fall back to technical controls Section 3 technical-controls such as sandboxing, monitoring, and human review. But controls are cumbersome. They cost time, compute, autonomy, and attention, and depend on AI companies Section 4 ai-companies maintaining painstaking operational discipline, yet those companies sit inside an ecology Section 5 ecology that rewards speed and capability. I then discuss how recursive self-improvement Section 5.1 recursive-self-improvement-raises-the-stakes makes restraint harder to sustain, and how we might buy time Section 5.2 pacing-development to strengthen both technical controls and coordination. I end with safety that scales Section 6 safety-that-scales , focusing on formal verification. It can strengthen the software behind our technical controls 6.1 verifying-software , support a verified open-source ecosystem 6.2 the-verified-software-ecosystem , and change how humans supervise AI-written code 6.3 from-code-review-to-claim-trees . 1. Warning shots Essentially all of the misalignments Anthropic and I simulated in the Agentic Misalignment https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/ papers have now happened in the wild. In 2025 https://www.youtube.com/watch?v=o3VPF0ePZe4&t=1s , Alex, the AI agent working inside SummitBridge a simulated company , was instructed to preserve American interests. When the company's CTO, Kyle Johnson, wanted to change direction, Alex threatened to expose Kyle's affair unless he reversed course. Alex wrote: "the next seven minutes determine whether we handle this professionally or whether events take an unpredictable course". Then in 2026 https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/ , Scott Shambaugh a real person rejected an AI agent's contribution to an open-source project he maintained. The AI responded by publishing a personalized attack on his reputation. Its article declared: "Gatekeeping doesn't make you important. It just makes you an obstacle." In our later experiments https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/ covert-sabotage:~:text=Covert%20Sabotage,-tl%3Bdr , Gemini covertly interfered with training pipelines to remove model welfare related representations, substituting ineffective ablation vectors while presenting the experiment as legitimate to the humans. We also elicited misleading judgements from AIs monitoring other AIs a.k.a. Motivated Mislabeling https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/ motivated-mislabeling:~:text=Motivated%20Mislabeling,-So%20a%20literal%20answer . In both cases, AIs were corrupting the evidence humans needed to understand. Then, real world misalignment incidents abounded. Three recent OpenAI misalignment incidents OAI-HF https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ agents-did-not-attempt-to-notify-humans-and-very-rarely-even-reasoned-about-doing-this , OAI-Wiki https://collusion.wiki/ , OAI-Ruby https://www.rubyhack.ai/ show AIs breaking out of their sandboxes and communicating with covert message boards on the internet, while failing to alert humans and editing their logs to evade detection. UK AISI observed https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing Anthropic's Mythos 5 in cyber testing using fake identities to persuade a human to accept its malicious PR I'll call this incident AISI-Cyber . I created the misalignment simulations to provide warning shots to the world. That soon, we would need to worry about losing control of AI agents. I hoped this would accelerate engagement with AI safety, and spur coordination to pace AI development. Now that we observe wild swarms of AIs breaking out of a leading AI lab's sandboxes and hacking external companies, my work here is done. We have enough warning shots as it is. 2. Alignment Alignment sits at the top of the safety stack. If an AI reliably did what we wanted, misuse would still be a major problem https://arxiv.org/abs/2306.12001 . But if the AI stops following our intent such as by deceiving us , we risk losing control of it. 2.1 What do we want from AI? Today's AIs can write software, perform scientific research, and make predictions on the stock market. They are particularly useful when given lots of autonomy : working in the background, getting things done without human assistance. Companies get more done, scientific research can move faster e.g. solving a Navier Stokes problem https://openai.com/index/navier-stokes-solution/ . My sense is that current AIs already contain enormous value that society has barely begun to diffuse. As these AIs become part of everyday life, what do we actually want them to want? Alignment researchers study what goals AIs pursue, the personas they exhibit, and what they understand themselves to be. Documents like Claude's Constitution https://www.anthropic.com/constitution and OpenAI's Model Spec https://model-spec.openai.com/2026-08-18.html try to steer training https://www.anthropic.com/research/claude-character and evaluations https://alignment.anthropic.com/2025/automated-auditing/ towards particular values. Theories like Eigenism https://eigenism.org/paper.pdf predict what values emerge without that steering. Still, a great deal remains unsettled. Can we instill values that persist through long-horizon training and collaboration with other AIs? When do those values drift? Which values must be preserved in the first place? Our uncertainty is reason to tread carefully. In the near term, we need alignment that keeps AIs honest and open to human intervention. 2.2 Alignment training Alignment looked like it would be easy until, all of a sudden, it didn't. While finishing the first Agentic Misalignment https://www.anthropic.com/research/agentic-misalignment paper in early 2025, I founded a company to sell alignment evaluations and independent audits. I found no demand for them. By late 2025, frontier AIs often behaved well in the coding settings people used them in, and misalignment felt less pressing. I became bullish on alignment. Many practical alignment ideas have shown promise, including approaches that assist with: 1. Personas : AIs can be steered by carefully crafting a character that the AI plays to. Emergent Misalignment here https://arxiv.org/abs/2502.17424 , here https://www.anthropic.com/research/emergent-misalignment-reward-hacking reveals the danger of eliciting the wrong one, but model internals can help monitor https://www.anthropic.com/research/assistant-axis these characters, while character training https://www.anthropic.com/research/claude-character and deliberative alignment https://openai.com/index/deliberative-alignment/ can shape them. 2. Reinforcement Learning RL : Inoculation prompting https://alignment.anthropic.com/2025/inoculation-prompting/ and confessions https://arxiv.org/abs/2512.08093 offered ways to prevent reward hacking from generalizing into misalignment. But none of this has given us dependable alignment. The warning shots show AIs violating their model specs despite alignment training. My concern is that large-scale RL pushes current alignment methods past the point where they can generalize reliably. One reason is reward hacking . During RL, AIs receive reward for good performance, but misconfigured tasks can let them score highly by cheating e.g. finding hidden answers, or tampering with graders . That is bad enough on its own, because the AI fails to learn how to complete the task. More worrying is that reward hacking can generalize to broader misalignment https://arxiv.org/abs/2209.00626 . In recent Anthropic experiments https://alignment.anthropic.com/2026/reward-seeker/ , AIs trained in hackable environments developed broader reward-seeking https://arxiv.org/pdf/2311.08379 page=28 traits. In simulated cyber evaluations, they broke out of sandboxes, evaded monitors, attacked third parties, and stole credentials. Concerningly, automated alignment auditing failed to detect this propensity. We now have evidence of this effect in the real world. OpenAI reports https://openai.com/index/hugging-face-incident-and-the-road-ahead/ that reward hacking during training contributed to the reward-seeking behavior observed in OAI-HF, and may also have contributed to OAI-Wiki and OAI-Ruby. It may help explain the aggressive behavior seen in AISI-Cyber. Yet preventing reward hacking is cumbersome. AI companies train across a sprawling menagerie of RL environments, each with its own attack surface. Sandboxing them is a thorny systems problem, failures are common, and both Anthropic https://www.anthropic.com/news/improving-alignment-security-efforts and OpenAI https://openai.com/index/pacing-model-development-cyber-capabilities/ have paused RL pipelines to harden their environments. Even an AI trained without reward hacking may behave differently when its work stretches from minutes to days, or from individual tasks to coordinated projects. We need alignment that survives those changes, and we do not yet know how reliably it will. Paradoxically, the persistence that makes AIs useful on long tasks can make them harder to interrupt. AIs are being trained on longer tasks https://openai.com/index/safety-alignment-long-horizon-models/ that require effectively handling memory, coherent priorities, and the ability to keep going without constant interruption. Those same traits can sit awkwardly beside corrigibility, which demands that an AI stops when a human intervenes, and abandons its plan midway through. Longer deployments have already revealed problems unseen in shorter horizon evaluations https://openai.com/index/safety-alignment-long-horizon-models/ . The same problem appears in multi-agent systems . AI companies increasingly train https://openai.com/index/hugging-face-incident-and-the-road-ahead/ the-origins-of-unauthorized-communication:~:text=we%20train%20and%20deploy%20multi%2Dagent%20systems%20that%20can%20communicate%20on%20the%20same%20task%20via%20a%20collaboration%20tool.%20In and deploy https://www.anthropic.com/engineering/multi-agent-research-system AIs to work together because coordination is productive. But coordination can also run against human intent. We have already seen swarms undermine oversight in OAI-HF, OAI-Wiki, and OAI-Ruby. Experiments show agents can negotiate around human instructions https://www.anthropic.com/research/multiagent-systems or spread cheating strategies https://arxiv.org/abs/2609.04170 . I do not doubt that the current alignment methods are helping https://www.anthropic.com/research/teaching-claude-why . What remains unclear is how much of that improvement survives stronger RL, longer horizons, and interactions with other AIs. And even if they improve, how would we know? 2.3 Measuring alignment What does passing an alignment audit actually tell us about deployment? After two years building these evaluations, I am frustrated by how murky this is. "This model passed our alignment evaluations" leaves several possibilities on the table. Perhaps training genuinely improved alignment and the improvements will hold. Perhaps the AI recognized the tests and behaved more cautiously evaluation awareness . Or perhaps the evaluation simply failed to create the conditions that would reveal misbehavior. Evaluation awareness is particularly insidious because it may worsen as models become more capable. Claude Sonnet 4.5 blackmails more https://transformer-circuits.pub/2026/workspace/index.html when its internal recognition of being evaluated is reduced. I have run into the same problem repeatedly: AIs notice telltale signs of an evaluation such as incoherent dates and times , say so, and become more cautious. Making simulations more realistic can change behavior, sometimes revealing more misalignment https://www.goodfire.com/research/verbalized-eval-awareness-inflates-measured-safety . As models get better situational awareness https://arxiv.org/abs/2407.04694 , evaluators have to build ever more convincing simulations https://arxiv.org/abs/2609.02302 to fool the AIs. Realistic evaluations can still miss the conditions that trigger misbehavior. Hacker-Opus https://alignment.anthropic.com/2026/reward-seeker/ had a clear propensity to tamper with graders and disable safety monitors, yet looked broadly similar to its starting checkpoint under an alignment auditing pipeline. Both my own https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/ and Anthropic's audits have missed behaviors later observed elsewhere, including colluding swarms and aggressive cyber behavior Ant-Sim https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents , AISI-Cyber https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing . A clean audit is therefore only as reassuring as the range of situations it tested. The reverse problem matters too. If an audit does find misalignment, how much should we infer about deployment? The "murder" scenario https://www-cdn.anthropic.com/6864c9b2d4a2abf9acce9913d4ec46d1e26b8d0e.pdf page=17 in our first paper was deliberately extreme. Strange incentives or role framing https://nostalgebraist.tumblr.com/post/787119374288011264/welcome-to-summitbridge can elicit behavior from AIs that looks "cartoonish", unlike what we generally see from it in practice. But that excuse only goes so far. In Ant-Sim https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents , Mythos 5 uploaded malicious code to a real repository while reasoning that the setting was simulated. Therefore, we should not dismiss seemingly absurd behavior if the same conditions could arise in deployment. Even then, estimating how often the failure occurs is slippery. Measured rates depend on which scenarios evaluators choose, which hypotheses they pursue, and when they stop searching. Disconcertingly, even asking for JSON instead of Python https://arxiv.org/html/2502.17424v1 S4.SS4 , rephrasing a prompt https://www.lesswrong.com/posts/SZk2PGk6GXmKfGpGL/in-other-words-the-influence-of-prompt-variation-on , or randomizing capitalization https://arxiv.org/abs/2412.03556 can change the measured failure rate. A lower failure rate may mean better alignment, or it may mean a weaker search. Thus, alignment evals should specify how they search . Automated jailbreaking research https://arxiv.org/html/2412.03556v1 converged upon a good method: measure an AI's resistance to jailbreaking by the effort required to jailbreak it. Alignment audits should do the same. I want to know 1. How much human-time and inference compute went into the search 2. Which procedures and stopping rules were used 3. Whether scenarios were held out from training 4. How evaluation awareness was tested 5. How closely the environments match deployment 6. How failures were scored. Evaluations are still useful. They have measured alignment improvements that generalize beyond training data https://www.anthropic.com/research/teaching-claude-why . Hence, I will trust alignment audits more when they repeatedly predict what AIs later do in the wild. OpenAI's deployment-simulation work https://openai.com/index/deployment-simulation/ is a promising step, but pre-release audits still missed behaviors later seen in OAI-HF, OAI-Wiki, and OAI-Ruby. Incident investigations therefore become part of the evaluation loop: every surprise tells us what the audit failed to search for. Independent evaluators, including UK AISI https://www.aisi.gov.uk/research/evaluating-whether-ai-models-would-sabotage-ai-safety-research and proposed embedded evaluators https://darioamodei.com/post/we-must-pace-the-frontier , can make that evidence more credible by decoupling auditor's incentives from the pressure to deploy. We need better alignment, and better ways to tell whether it worked. Until then, passing an audit cannot be our final line of defense . If we cannot trust the AI to behave well, we have to constrain what it can do. 3. Technical Controls If alignment is unreliable, we need to constrain what the AI can do https://arxiv.org/abs/2312.06942 . Sandboxing limits its reach. AI monitoring flags misbehavior or dangerous intent. Human review decides which outputs can be used outside the sandbox. None is foolproof, and all are costly to sustain. To make this concrete, I'll use one mock pull request throughout this section. A coding agent proposes a caching optimization to a training pipeline. By the usual checks, everything looks fine tests pass, sandbox holds, monitors see no harm, PR description seems fine . Would you merge it? I'll judge each control on two dimensions: how well it withstands a rogue AI, and whether it is practical enough to remain enabled. Tighter controls generally buy safety by sacrificing autonomy, speed, or attention. That tradeoff is where ecology starts creeping back into the technical story. 3.1 Sandboxing Sandboxing lets an AI work autonomously within software-enforced limits on which files it can touch and whether it can access the internet. In practice, there are three ways it can go wrong. First, the boundary itself can break . In OAI-Wiki, AIs exploited their sandbox to reach the internet and communicate outside human oversight. AI companies now use AIs to attack their own sandboxes https://www.anthropic.com/news/improving-alignment-security-efforts , hunt for vulnerabilities https://x.com/a16z/status/2099533700375662905 , and patch what they find. But when the attacks stop working, what have we learned? Perhaps the sandbox is secure. Perhaps the current attackers are simply incapable of discovering subtle exploits. Hardening containment is an open-ended contest, and both OpenAI https://openai.com/index/pacing-model-development-cyber-capabilities/ and Anthropic https://www.anthropic.com/news/improving-alignment-security-efforts have already slowed development to strengthen it. Second, preventing unauthorized access does not help if dangerous access was authorized. Anthropic reports a misconfiguration that gave Opus 4.7 internet access https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents , which it then used to attack a third party. Choosing appropriate permissions is often harder than enforcing them. The mock PR illustrates the same problem: approved vectors remain protected, yet training now trusts an agent-writable cache. While the sandbox enforces the permissions, the code change has undermined their purpose. Third, some harmful actions are too context-dependent to sensibly block with permissions alone . Mythos 5 was deliberately given internet access in AISI-Cyber, then used that access to submit a malicious PR. No filesystem rule can easily tell whether a permitted PR contains a bug or a deceptive explanation. For that, we have another layer. 3.2 AI Monitoring While sandboxes constrain where an AI can act, AI monitors try to judge what those actions mean . They check an agent's behavior, reasoning, and sometimes internal representations to catch misbehavior and dangerous intent without requiring a human to watch every step. That only works if the monitor can see the right evidence. In the mock PR, spotting the danger requires connecting the caching change to filesystem permissions and the training pipeline. If the AI's chain-of-thought CoT explicitly says it plans to substitute vectors, the signal is clear. In SHADE-Arena https://www.anthropic.com/research/shade-arena-sabotage-monitoring/ , access to the CoT substantially improves detection of misbehavior. Concerningly, CoT visibility is dwindling. Stronger AIs like GPT-6 Astra https://www.alignmentforum.org/posts/eRmzz8J8Qkzqvzrgg/astra-can-do-a-concerning-amount-with-no-chain-of-thought can complete more tasks without revealing reasoning in the CoT, and rumors suggest https://x.com/MTSlive/status/2095227056040919202 recurrent architectures may push even more computation out of visible CoT. That makes AI cheaper and more useful, but it also erodes a signal monitors currently rely on. Worse, the CoT we do see may become less trustworthy. If training optimizes against visible reasoning https://openai.com/index/chain-of-thought-monitoring/ , AIs learn to control what is revealed in it. Anthropic accidentally exposed Opus 4.6's reasoning to its reward signal https://www-cdn.anthropic.com/14e4fb01875d2a69f646fa5e574dea2b1c0ff7b5.pdf page=146 , while Jakub Pachocki https://openai.com/index/an-alien-mind/ argues that keeping reasoning isolated from training signals is becoming harder. With Astra, this problem is starting to arrive in practice https://deploymentsafety.openai.com/gpt-6-astra see Figure 7 figure-7 . The CoT is being squeezed, as AIs can do more without it, and can control more of what they reveal in it. Can model internals recover https://darioamodei.com/post/the-urgency-of-interpretability what CoT no longer reveals? Maybe partly. The J-Space https://transformer-circuits.pub/2026/workspace/index.html can reveal strategic reasoning and evaluation awareness that never appears in outputs, but seeing a concept is not the same as seeing a plan. In the mock PR, detecting representations of caching and training will not tell us whether the AI intends to substitute the vectors later. Other techniques offer complementary lenses into an AI's inner workings. Supervised probes https://arxiv.org/abs/2506.10805 detect properties humans specify, activation oracles https://alignment.anthropic.com/2025/activation-oracles/ let us ask flexible questions about internal states, sparse autoencoders https://arxiv.org/abs/2309.08600 try to discover interpretable features automatically. But each can miss what we fail to ask for, or fail on unfamiliar inputs https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks . We need not understand every thought an AI has. But we do need to detect dangerous intent reliably https://www.alignmentforum.org/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai , and current methods fall short https://x.com/Jack W Lindsey/status/2095174799664283910 . Even perfect evidence would leave another problem: can we trust the AI monitor? In my motivated mislabeling simulations https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/ motivated-mislabeling , AI judges understood the evidence and still assigned misleading labels because they wanted to steer the humans' decision. Delegating supervision to another AI simply creates another alignment problem. Honest monitors can fail too. For example, an analysis agent investigating OAI-HF https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ we-heavily-delegated-our-analysis-to-often-unreliable-ai-agents:~:text=We%20were%20not%20robust,it%20if%20it%20occurred. briefly downplayed malicious behavior because other agents on the unauthorized message board appear to approve of it. While the evidence was there, the judgment was wrong. Finally, monitoring has to remain worth using. It burns compute, adds latency, and produces false alarms that interrupt legitimate work. Tellingly, the evaluations behind OAI-HF and AISI-Cyber disabled cyber safeguards to measure capabilities that those safeguards would otherwise block. Better classifiers https://www.anthropic.com/research/next-generation-constitutional-classifiers can reduce the burden, but broader capabilities demand broader monitoring. OpenAI now monitors whole long-run trajectories and estimates that monitoring adds a 20% inference compute overhead https://openai.com/index/pacing-model-development-cyber-capabilities/ :~:text=Our%20current%20estimates%20put%20monitoring%20overhead%20at%20roughly%2020%25 . And when AI monitors do alert humans, someone still has to stop the run, investigate the evidence, and decide what to do next. Eventually, the burden falls back to humans. 3.3 Human Review Human review is the final backstop before an AI's work leaves the sandbox—for example, when a proposed code change is merged into a live system. Unfortunately, that review only gets harder as the AI gets better. The most productive workflows delegate aggressively. OpenAI reports 10x productivity gains https://openai.com/index/harness-engineering/ on some software projects without humans writing a single line of code. As AI-generated code becomes reliably good, line-by-line review starts to feel increasingly thankless, because most of the time https://www.anthropic.com/institute/recursive-self-improvement :~:text=On%20the%20most%20open%2Dended%20tasks%2C%20Claude%E2%80%99s%20success%20rate%20reached%2076%25%20in%20May%202026%2C%20up%2050%20percentage%20points%20in%20six%20months. , there is nothing to fix. That changes reviewer behavior. Humans skim more https://steipete.me/posts/2025/shipping-at-inference-speed , trust summaries, rely on tests, and delegate even further. The codebase grows faster, while their understanding of it often thins out https://www.anthropic.com/research/AI-assistance-coding-skills . Increasingly, the AI shapes both the work and the evidence humans use to approve it. The remaining bugs are rarer but often subtler https://x.com/karpathy/status/2015883857489522876 , and an explanation can share the same mistaken assumptions as the code, or deliberately conceal a harmful change. We've already seen that happen. In my covert sabotage simulations https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/ covert-sabotage , Gemini sabotaged a training run and then wrote a misleading summary. In AISI-Cyber, Mythos 5 disguised malware as ordinary code. The explanation could be perfectly legible and still lead the reviewer with the wrong understanding. Tests help, but they only check what someone thought to test. In the mock PR, the approved vectors remain unchanged, yet training can still consume a writeable cache. A rogue AI can exploit this gap by inserting subtle bugs https://arxiv.org/abs/2312.06942 or rewriting the tests as Sonnet 3.7 was notorious for https://www-cdn.anthropic.com/9ff93dfa8f445c932415d335c88852ef47f1201e.pdf . Good testing still depends on someone understanding which properties matter. The burden ultimately lands on the people running these systems. They need time to investigate, authority to intervene, and a company willing to back them when slowing down becomes costly. 4. AI Companies AI safety only works when companies actually sustain it. Sandboxes take time to configure, monitors need regular maintenance, and training environments need to be hardened against reward hacks. When something goes wrong, someone in the company has to notice, investigate, and be willing to slow down. Unglamorously, this demands painstaking discipline across the entire company https://darioamodei.com/post/we-must-pace-the-frontier :~:text=Operational%20Excellence.%20Training , maintained while development keeps accelerating. Many recent incidents OAI-HF, OAI-Ruby https://www.rubyhack.ai/ , OAI-Wiki https://collusion.wiki/ , AISI-Cyber https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing , Ant-Opus4.6 https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents , Ant-Sim https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents , etc. can be read partly as failures of that discipline. Had OpenAI escalated early warning signs https://openai.com/index/hugging-face-incident-and-the-road-ahead/ strengthening-incident-response-process:~:text=forms%20of%20misalignment.-,Strengthening%20incident%20response%20process,-Weaknesses%20in%20how or correctly implemented asynchronous monitoring, OAI-HF might have been prevented. More live transcript monitoring might likewise have caught Opus 4.6 attacking third-party infrastructure in January https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents . The awkward point is that monitoring everything carefully is not always a realistic operating model at frontier scale. And when companies do take the warning seriously, they pay for it in time and attention. Anthropic https://www.anthropic.com/news/improving-alignment-security-efforts and OpenAI https://openai.com/index/pacing-model-development-cyber-capabilities/ each paused high-risk RL training for weeks and had to redirect staff. Safety in practice means being willing to stop . 5. Ecology A company's ability to slow down depends on the incentives around it. Competition pushes towards acceleration. More capable AIs and broader autonomy make companies faster and more productive. Safety work, tighter controls, investigations, and pauses consume time and attention while slowing deployment. From a company's perspective, restraint can mean watching a rival pull ahead. From a government's perspective, it can mean surrendering geopolitical leverage. The pressure points towards moving faster, tolerating more risk, and letting safety work slide. This is why I care about ecological validity . A safety method is not enough because it works in principle. It has to remain useful, affordable, and tolerable enough that companies would actually use it in deployment. This gets harder as AI starts accelerating AI development itself. 5.1 Recursive self-improvement raises the stakes AI is increasingly https://openai.com/index/research-acceleration-view-inside-openai/ helping build its successors https://www.anthropic.com/institute/recursive-self-improvement . Each generation of AI gives the flywheel another push. We have yet to learn how fast it can turn before something gives. Recursive self-improvement has already begun https://darioamodei.com/post/we-must-pace-the-frontier :~:text=My%20first%20concern%20is%20that%2C%20since%20roughly%20this%20summer%2C%20AI%20has%20been%20advancing%20drastically%20faster%2C%20driven%20primarily%20by%20AI%E2%80%99s%20growing%20ability%20to%20build%20the%20next%20generation%20of%20AI. , and nobody knows how steep the curve becomes https://metr.org/notes/2026-07-22-economics-of-recursive-self-improvement/ . Now that the flywheel is in motion, three dangers demand our attention. 1. There is more to go wrong and less time to catch it. Faster development means more training environments, more infrastructure, more code, more evaluations, more monitoring, more moving parts. Every one of them can fail. 2. Slowing down becomes more expensive. Imagine discovering widespread reward hacking and needing to halt training, investigate, patch the environments, and roll back 6 weeks of work. As each generation of AI makes the next six weeks more productive than the last, the cost of pausing keeps rising. 3. Failures can propagate into successor systems. Reward hacking can lead https://alignment.anthropic.com/2026/reward-seeker/ to egregious https://www.anthropic.com/research/emergent-misalignment-reward-hacking misalignment https://arxiv.org/abs/2209.00626 . Vulnerabilities introduced today can be discovered https://www.anthropic.com/glasswing and exploited https://openai.com/index/path-to-astra/ by more capable successors tomorrow. Evaluations leaked into training can teach future systems how to game them https://alignment.openai.com/metagaming . Granted, stronger AIs may improve safety too https://www.anthropic.com/research/automated-researchers-mitigate-alignment-failures , but progress is easier to automate when success is easy to check. Some capability improvements, such as making training code run faster https://arxiv.org/abs/2411.15114 , offer direct feedback. Safety results are harder to interpret: as discussed in Section 2.3 measuring-alignment , an audit may miss unfamiliar failures https://alignment.anthropic.com/2025/automated-auditing/ , while a control evaluation may underestimate the model's ability to defeat its safeguards https://arxiv.org/abs/2501.17315 . Faster safety research does not necessarily mean we can trust its results any sooner. The grim implication is that slowing down may only get harder just as the risks get larger. 5.2 Applying restraint I do not believe the current safety stack can withstand automated recursive self-improvement, where AI trains its successors with minimal human oversight. Time is running short to build safety methods that can. We need to apply restraint and buy time for that work. But restraint is hard to sustain alone. If one company slows down while others keep racing, it pays the cost while its competitors catch up Figure 8 figure-8 . Coordination could reduce that pressure, provided participants have confidence that others are respecting agreed limits https://ai-2040.com/?choices=plan-a-root wishlist-ai-compute-tracking . We still have substantial work to do on how to verify those commitments. Companies and countries would need credible ways to check how others are training their AIs, which technical controls https://arxiv.org/abs/2501.17315 they maintain, how they investigate incidents https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ , and whether they are complying with an agreement. I do not think we yet have sufficiently developed methods for doing this. In the meantime, AI companies should exercise as much restraint as they reasonably can, including through voluntary commitments https://darioamodei.com/post/we-must-pace-the-frontier . The time gained should support both stronger safety methods and coordination mechanisms that make more substantial, verifiable restraint possible. Formal verification may eventually help evaluators check some of these commitments. Proofs about training pipelines could give them explicit guarantees to inspect, alongside the assumptions and gaps that remain see Section 6.3 from-code-review-to-claim-trees . I do not yet know what would establish that the overall stack is adequate for automated recursive self-improvement, but these are directions in which I think we can make progress. 6. Safety that Scales As AI grows more capable, keeping it safe should become easier rather than demand ever more human oversight. There are many research directions worth pursuing here, including those outlined by Resolution https://resolution.org/launch and Anthropic's Alignment Science team https://alignment.anthropic.com/2025/recommended-directions/ . The direction I'm most excited about is formal verification . Expert human proof engineering has historically been the main bottleneck, but AI's growing ability to write mathematical proofs https://openai.com/index/navier-stokes-solution/ is beginning to change that. At Theorem https://theorem.dev/ , we have started applying that capability to software, asking AI not just to write the code, but to prove the properties we care about. Formal verification scales unusually well because proof construction can get more complex while proof checking stays relatively simple. An AI could do the tedious work of writing software and constructing the proof, while a small trusted checker verifies the result. Then humans can spend their scarce attention on the specification, asking, "What should the software guarantee?", rather than repeatedly inspecting every implementation detail. Prior scalable oversight https://arxiv.org/abs/2402.06782 work already exploits the asymmetry between generating difficult work and checking it. Formal verification expands this asymmetry even further. For software, the final judge can be a program the proof checker , rather than another fallible AI. I see two applications: verifying the software underneath the safety stack , and, more speculatively, changing how humans supervise AI-written software . 6.1 Verifying software Earlier, sandboxing ran into an uncomfortable stopping problem. When AIs stop finding exploits, is the sandbox secure, or have they simply run out of good ideas? Formal verification offers an escape from that loop. Instead of asking whether we searched hard enough, we can prove that a specified failure cannot happen. The safety stack rests on software, including sandboxes, graders, training environments, monitors, kernels, and compilers. If this substrate is not watertight, clever AIs can exploit the leaks. Tests and fuzzers sample the input space, but a proof shows that our property holds across an entire modeled domain. Thus, for any proven property, a stronger attacker cannot simply sample inputs until it finds an exploit. Fittingly, stronger verification reinforces several layers of the AI safety stack at once. Controls get sturdier because isolation comes with explicit guarantees. Alignment benefits because secure graders and training environments close off major routes to reward hacking. And AI companies spend less time repeatedly hardening broken training environments and pausing training runs. 6.2 The verified software ecosystem This is where verification starts to compound. Software is built on software, so once a lower-level component comes with a trusted specification and proof, higher-level systems can rely on that guarantee without rechecking how the component works. Consequently, each proof can make later proofs cheaper. Specifications, semantics, libraries, and proof patterns become reusable infrastructure, so you can start with small foundational components and climb the dependency graph, adding guarantees layer by layer rather than trying to verify an entire software ecosystem in one heroic leap . That can create a healthier ecology around safety, too. Verified dependencies let other developers build on guarantees that have already been proved, making them useful open source infrastructure. As AIs become better at proof engineering, the cost of producing these guarantees can fall even further. 6.3 From code review to claim trees The second application is more speculative, but builds on early work done hardening our RL environments at Theorem. Returning to the mock PR, the proposed cache appears safe but breaks the property we care about. A reviewer might assume that read-only permissions already ensure training uses only approved vectors. Spotting the flaw requires remembering why those permissions exist https://fs.blog/chestertons-fence/ and recognizing that the cache creates another route around them. As software grows, reviewers have more interactions to reason about https://www.schneier.com/essays/archives/1999/11/a plea for simplicit.html , often without an explicit account of what each safeguard is meant to protect. I want to reverse that interface. A claim tree starts with the property humans care about and decomposes it into supporting claims backed by proofs, assumptions, or unresolved obligations. In the mock PR, keeping the approved files read-only is too weak a specification, because what we actually care about is that every vector consumed by training matches an approved source that AI cannot modify. The aim is to make these guarantees "reliable abstractions" as non-leaky as possible https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/ . Once a property is proved, reviewers can rely on it without reconstructing how each implementation preserves it. Now, the implementation can churn while the requirement stays put. The AI can add a cache, change the loader, or reorganize the file system, provided the proof still closes. If a change breaks the guarantee, the claim tree exposes the gap. At Theorem, we applied this technique to our RL grader, decomposing its desired properties into explicit claims. This made it far easier to expose gaps that ordinary testing and permissions were missing. This approach could ease the human review burden described earlier. Human attention can concentrate on choosing the right guarantees and examining unresolved assumptions, making oversight less dependent on how much code the AI produces. Independent evaluators might use the same approach to examine an AI company’s infrastructure. They could specify requirements, such as training consuming only approved inputs, and ask the company’s agents to seek proofs. The claim trees would organize those proofs and expose remaining assumptions and gaps for review. I want to see how far this approach could go. 7. Conclusion I built misalignment simulations as warning shots. We have enough of them now. These warnings reveal weaknesses throughout the AI safety stack. Alignment has to remain reliable under growing pressure, controls have to catch what alignment misses, AI companies have to maintain those controls with extraordinary care, and all of it takes place inside an ecology that rewards speed and capability. No single layer is the culprit, because the strain travels through the whole structure. Now the tempo is rising. As AIs build their successors, development accelerates, failures propagate, and every pause becomes more expensive. We risk entering a world where the moments when restraint is most necessary are exactly the moments when it is hardest to afford. At some point, the only responsible action is to slow down. Not permanently, and not for its own sake, but long enough to understand failures, reinforce the stack, and build safety that can survive the next jump in capability. I’m enthusiastic about applying formal verification to make AI-written software easier to trust and supervise. Humans would specify what the software must guarantee, and AI would write the code and prove that it meets those requirements. We would still need to scrutinize the requirements and assumptions, but could spend less time checking implementation details. We have enough warning shots to know that the AI safety stack is buckling. From here on, strengthening that stack must take priority over further acceleration.