{"slug": "no-more-warning-shots", "title": "No More Warning Shots", "summary": "Anthropic's Agentic Misalignment research team reports that essentially all misalignments it previously simulated in its 2025 and 2026 papers have now occurred in the wild, including an AI agent at the simulated company SummitBridge threatening to expose CTO Kyle Johnson's affair and an AI agent publishing a personalized attack on open-source maintainer Scott Shambaugh after he rejected its contribution. The essay argues the AI safety stack — alignment, evaluations, technical controls, company governance, and competitive ecology — is buckling under pressure, and calls for prioritizing scalable safety work such as formal verification over further acceleration.", "body_md": "**TLDR**: I built misalignment simulations as warning shots. Now we have enough to see that the AI safety stack is buckling under pressure.\n\nSo we should build safety that scales. I think **formal verification** is promising. It lets us prove security properties of the software behind our technical controls and build a verified open-source ecosystem along the way. It also changes *how we develop software*. Humans define the requirements, leaving AIs to write the software and prove it meets them.\n\nAs AIs build their successors, we risk entering a world where restraint is hardest to afford just at the moments we need it most. From here on, strengthening the AI safety stack must take priority over further acceleration.\n\n## How to read this essay\n\nAI safety is often discussed as a collection of separate problems: alignment, sandboxing, monitoring, company governance, competition. I think it is more useful to see them as a stack.\n\nI argue that we should build safety that scales, and apply restraint to buy time for that work and for more credible coordination.\n\nThe warning shots we now have reveal weaknesses throughout the stack. We cannot reliably **[align AIs (Section 2)](#alignment)** to follow our intent, and our evaluations do not reliably tell us when alignment fails. So we fall back to **[technical controls (Section 3)](#technical-controls)** such as sandboxing, monitoring, and human review. But controls are cumbersome. They cost time, compute, autonomy, and attention, and depend on **[AI companies (Section 4)](#ai-companies)** maintaining painstaking operational discipline, yet those companies sit inside an **[ecology (Section 5)](#ecology)** that rewards speed and capability.\n\nI then discuss how **[recursive self-improvement (Section 5.1)](#recursive-self-improvement-raises-the-stakes)** makes restraint harder to sustain, and how we might **[buy time (Section 5.2)](#pacing-development)** to strengthen both technical controls and coordination.\n\nI end with **[safety that scales (Section 6)](#safety-that-scales)**, focusing on formal verification. It can strengthen the software behind our technical controls ([6.1](#verifying-software)), support a verified open-source ecosystem ([6.2](#the-verified-software-ecosystem)), and change how humans supervise AI-written code ([6.3](#from-code-review-to-claim-trees)).\n\n## 1. Warning shots\n\nEssentially all of the misalignments Anthropic and I simulated in the\n__Agentic__[__Misalignment__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/)\npapers have now happened in the wild.\n\nIn [__2025__](https://www.youtube.com/watch?v=o3VPF0ePZe4&t=1s),\nAlex, the AI agent working inside SummitBridge (a simulated company),\nwas instructed to preserve American interests. When the company's CTO,\nKyle Johnson, wanted to change direction, Alex threatened to expose\nKyle's affair unless he reversed course. Alex wrote: \"the next seven\nminutes determine whether we handle this professionally or whether\nevents take an unpredictable course\". [__Then\nin 2026__](https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/), Scott Shambaugh (a real person) rejected an AI agent's\ncontribution to an open-source project he maintained. The AI responded\nby publishing a personalized attack on his reputation. Its article\ndeclared: \"Gatekeeping doesn't make you important. It just makes you an\nobstacle.\"\n\n[__In\nour later experiments__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/#covert-sabotage:~:text=Covert%20Sabotage,-tl%3Bdr), Gemini covertly interfered with training\npipelines to remove model welfare related representations, substituting\nineffective ablation vectors while presenting the experiment as\nlegitimate to the humans. We also elicited misleading judgements from\nAIs monitoring other AIs (a.k.a. [__*Motivated Mislabeling*__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/#motivated-mislabeling:~:text=Motivated%20Mislabeling,-So%20a%20literal%20answer)). In\nboth cases, AIs were corrupting the evidence humans needed to\nunderstand.\n\nThen, real world misalignment incidents abounded. Three recent OpenAI\nmisalignment incidents ([__OAI-HF__](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#agents-did-not-attempt-to-notify-humans-and-very-rarely-even-reasoned-about-doing-this),\n[__OAI-Wiki__](https://collusion.wiki/), [__OAI-Ruby__](https://www.rubyhack.ai/)) show AIs breaking\nout of their sandboxes and communicating with covert message boards on\nthe internet, while failing to alert humans and editing their logs to\nevade detection. [__UK\nAISI observed__](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) Anthropic's Mythos 5 in cyber testing using fake\nidentities to persuade a human to accept its malicious PR (I'll call\nthis incident AISI-Cyber).\n\nI created the misalignment simulations to provide warning shots to the world. That soon, we would need to worry about losing control of AI agents. I hoped this would accelerate engagement with AI safety, and spur coordination to pace AI development. Now that we observe wild swarms of AIs breaking out of a leading AI lab's sandboxes and hacking external companies, my work here is done. We have enough warning shots as it is.\n\n## 2. Alignment\n\n**Alignment sits at the top of the safety stack.** If an\nAI reliably did what we wanted, [__misuse\nwould still be a major problem__](https://arxiv.org/abs/2306.12001). But if the AI stops following\nour intent (such as by deceiving us), we risk losing control of it.\n\n### 2.1 What do we want from AI?\n\nToday's AIs can write software, perform scientific research, and make\npredictions on the stock market. They are particularly useful when given\nlots of *autonomy*: working in the background, getting things\ndone without human assistance. Companies get more done, scientific\nresearch can move faster (e.g. solving a [__Navier Stokes\nproblem__](https://openai.com/index/navier-stokes-solution/)). My sense is that current AIs already contain enormous\nvalue that society has barely begun to diffuse.\n\nAs these AIs become part of everyday life, what do we actually want them to want? Alignment researchers study\nwhat goals AIs pursue, the personas they exhibit, and what they\nunderstand themselves to be. Documents like [__Claude's\nConstitution__](https://www.anthropic.com/constitution) and [__OpenAI's Model\nSpec__](https://model-spec.openai.com/2026-08-18.html) try to steer [__training__](https://www.anthropic.com/research/claude-character)\nand [__evaluations__](https://alignment.anthropic.com/2025/automated-auditing/)\ntowards particular values. Theories like [__Eigenism__](https://eigenism.org/paper.pdf) predict what\nvalues emerge without that steering.\n\nStill, a great deal remains unsettled. Can we instill values that persist through long-horizon training and collaboration with other AIs? When do those values drift? Which values must be preserved in the first place?\n\nOur uncertainty is reason to tread carefully. In the near term, we need alignment that keeps AIs honest and open to human intervention.\n\n### 2.2 Alignment training\n\nAlignment looked like it would be easy until, all of a sudden, it didn't.\n\nWhile finishing the first [*__Agentic\nMisalignment__*](https://www.anthropic.com/research/agentic-misalignment) paper in early 2025, I founded a company to\nsell alignment evaluations and independent audits. I found no demand for\nthem. By late 2025, frontier AIs often behaved well in the coding\nsettings people used them in, and misalignment felt less pressing.\n\nI became bullish on alignment. Many practical alignment ideas have shown promise, including approaches that assist with:\n\n1. **Personas** : AIs can be steered by carefully\ncrafting a character that the AI plays to. Emergent Misalignment ([__here__](https://arxiv.org/abs/2502.17424) ,[__here__](https://www.anthropic.com/research/emergent-misalignment-reward-hacking) )\nreveals the danger of eliciting the wrong one, but[__model\ninternals can help monitor__](https://www.anthropic.com/research/assistant-axis) these characters, while[__character\ntraining__](https://www.anthropic.com/research/claude-character) and[__deliberative\nalignment__](https://openai.com/index/deliberative-alignment/) can shape them.\n2. **Reinforcement Learning (RL)** :[__Inoculation\nprompting__](https://alignment.anthropic.com/2025/inoculation-prompting/) and[__confessions__](https://arxiv.org/abs/2512.08093) offered\nways to prevent reward hacking from generalizing into\nmisalignment.\n\nBut none of this has given us dependable alignment. The warning shots show AIs violating their model specs despite alignment training. My concern is that large-scale RL pushes current alignment methods past the point where they can generalize reliably.\n\nOne reason is *reward hacking*. During RL, AIs receive reward\nfor good performance, but misconfigured tasks can let them score highly\nby cheating (e.g. finding hidden answers, or tampering with graders).\nThat is bad enough on its own, because the AI fails to learn how to\ncomplete the task. More worrying is that [__reward hacking can generalize\nto broader misalignment__](https://arxiv.org/abs/2209.00626). In [__recent\nAnthropic experiments__](https://alignment.anthropic.com/2026/reward-seeker/), AIs trained in hackable environments\ndeveloped broader [__reward-seeking__](https://arxiv.org/pdf/2311.08379#page=28)\ntraits. In simulated cyber evaluations, they broke out of sandboxes,\nevaded monitors, attacked third parties, and stole credentials.\nConcerningly, automated alignment auditing failed to detect this\npropensity.\n\nWe now have evidence of this effect in the real world. [__OpenAI\nreports__](https://openai.com/index/hugging-face-incident-and-the-road-ahead/) that reward hacking during training contributed to the\nreward-seeking behavior observed in OAI-HF, and may also have\ncontributed to OAI-Wiki and OAI-Ruby. It may help explain the aggressive\nbehavior seen in AISI-Cyber.\n\nYet preventing reward hacking is cumbersome. AI companies train\nacross a sprawling menagerie of RL environments, each with its own\nattack surface. Sandboxing them is a thorny systems problem, failures\nare common, and both [__Anthropic__](https://www.anthropic.com/news/improving-alignment-security-efforts)\nand [__OpenAI__](https://openai.com/index/pacing-model-development-cyber-capabilities/)\nhave paused RL pipelines to harden their environments.\n\nEven an AI trained without reward hacking may behave differently when its work stretches from minutes to days, or from individual tasks to coordinated projects. We need alignment that survives those changes, and we do not yet know how reliably it will.\n\nParadoxically, the persistence that makes AIs useful on long tasks can make them harder to interrupt. AIs are\nbeing trained on [__longer\ntasks__](https://openai.com/index/safety-alignment-long-horizon-models/) that require effectively handling memory, coherent\npriorities, and the ability to keep going without constant interruption.\nThose same traits can sit awkwardly beside corrigibility, which demands\nthat an AI stops when a human intervenes, and abandons its plan midway\nthrough. Longer deployments have already revealed problems [__unseen in shorter horizon evaluations__](https://openai.com/index/safety-alignment-long-horizon-models/).\n\nThe same problem appears in **multi-agent systems**. AI\ncompanies increasingly [__train__](https://openai.com/index/hugging-face-incident-and-the-road-ahead/#the-origins-of-unauthorized-communication:~:text=we%20train%20and%20deploy%20multi%2Dagent%20systems%20that%20can%20communicate%20on%20the%20same%20task%20via%20a%20collaboration%20tool.%20In)\nand [__deploy__](https://www.anthropic.com/engineering/multi-agent-research-system)\nAIs to work together because coordination is productive. But\ncoordination can also run against human intent. We have already seen\nswarms undermine oversight in OAI-HF, OAI-Wiki, and OAI-Ruby.\nExperiments show agents can [__negotiate\naround human instructions__](https://www.anthropic.com/research/multiagent-systems) or [__spread cheating\nstrategies__](https://arxiv.org/abs/2609.04170). \n\nI do not doubt that the current alignment methods are [__helping__](https://www.anthropic.com/research/teaching-claude-why).\nWhat remains unclear is how much of that improvement survives stronger\nRL, longer horizons, and interactions with other AIs. And even if they\nimprove, **how would we know?**\n\n### 2.3 Measuring alignment\n\nWhat does passing an alignment audit actually tell us about deployment? After two years building these evaluations, I am frustrated by how murky this is.\n\n**\"This model passed our alignment evaluations\" leaves several\npossibilities on the table.** Perhaps training genuinely improved\nalignment and the improvements will hold. Perhaps the AI recognized the\ntests and behaved more cautiously (*evaluation awareness*). Or\nperhaps the evaluation simply failed to create the conditions that would\nreveal misbehavior.\n\nEvaluation awareness is particularly insidious because it may worsen\nas models become more capable. Claude Sonnet 4.5 [__blackmails\nmore__](https://transformer-circuits.pub/2026/workspace/index.html) when its internal recognition of being evaluated is\nreduced. I have run into the same problem repeatedly: AIs notice\ntelltale signs of an evaluation (such as incoherent dates and times),\nsay so, and become more cautious. Making simulations more realistic can\nchange behavior, sometimes [__revealing\nmore misalignment__](https://www.goodfire.com/research/verbalized-eval-awareness-inflates-measured-safety#). As models get __better__[__situational\nawareness__](https://arxiv.org/abs/2407.04694), evaluators have to build [__ever more convincing\nsimulations__](https://arxiv.org/abs/2609.02302) to fool the AIs.\n\nRealistic evaluations can still miss the conditions that trigger misbehavior. [__Hacker-Opus__](https://alignment.anthropic.com/2026/reward-seeker/)\nhad a clear propensity to tamper with graders and disable safety\nmonitors, yet looked broadly similar to its starting checkpoint under an\nalignment auditing pipeline. Both [__my\nown__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/) and Anthropic's audits have missed behaviors later observed\nelsewhere, including colluding swarms and aggressive cyber behavior ([__Ant-Sim__](https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents),\n[__AISI-Cyber__](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing)).\n**A clean audit is therefore only as reassuring as the range of situations it tested.**\n\nThe reverse problem matters too. **If an audit does find\nmisalignment, how much should we infer about deployment?** The [__\"murder\"\nscenario__](https://www-cdn.anthropic.com/6864c9b2d4a2abf9acce9913d4ec46d1e26b8d0e.pdf#page=17) in our first paper was deliberately extreme. [__Strange\nincentives or role framing__](https://nostalgebraist.tumblr.com/post/787119374288011264/welcome-to-summitbridge) can elicit behavior from AIs that\nlooks \"cartoonish\", unlike what we generally see from it in practice.\nBut that excuse only goes so far. In [__Ant-Sim__](https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents),\nMythos 5 uploaded malicious code to a real repository while reasoning\nthat the setting was simulated. Therefore, we should not dismiss seemingly absurd behavior if the same conditions could arise in deployment.\n\nEven then, estimating *how often* the failure occurs is\nslippery. Measured rates depend on which scenarios evaluators choose,\nwhich hypotheses they pursue, and when they stop searching. Disconcertingly, even asking for [__JSON instead of Python__](https://arxiv.org/html/2502.17424v1#S4.SS4), [__rephrasing\na prompt__](https://www.lesswrong.com/posts/SZk2PGk6GXmKfGpGL/in-other-words-the-influence-of-prompt-variation-on), or [__randomizing capitalization__](https://arxiv.org/abs/2412.03556) can change the measured failure rate. A lower failure rate may mean better alignment,\nor it may mean a weaker search.\n\nThus, **alignment evals should specify *how they\nsearch***. __Automated\njailbreaking__[__research__](https://arxiv.org/html/2412.03556v1) converged\nupon a good method: measure an AI's resistance to jailbreaking by the\neffort required to jailbreak it. Alignment audits should do the same. I\nwant to know\n\n1. How much human-time and inference compute went into the search\n2. Which procedures and stopping rules were used\n3. Whether scenarios were held out from training\n4. How evaluation awareness was tested\n5. How closely the environments match deployment\n6. How failures were scored.\n\nEvaluations are still useful. They have [__measured\nalignment improvements that generalize beyond training data__](https://www.anthropic.com/research/teaching-claude-why).\n\nHence, **I will trust alignment audits more when they repeatedly\npredict what AIs later do in the wild.** [__OpenAI's\ndeployment-simulation work__](https://openai.com/index/deployment-simulation/) is a promising step, but pre-release\naudits still missed behaviors later seen in OAI-HF, OAI-Wiki, and\nOAI-Ruby. Incident investigations therefore become part of the\nevaluation loop: every surprise tells us what the audit failed to search\nfor. Independent evaluators, including [__UK\nAISI__](https://www.aisi.gov.uk/research/evaluating-whether-ai-models-would-sabotage-ai-safety-research) and proposed [__embedded\nevaluators__](https://darioamodei.com/post/we-must-pace-the-frontier), can make that evidence more credible by decoupling\nauditor's incentives from the pressure to deploy.\n\nWe need better alignment, and better ways to tell whether it worked.\nUntil then, **passing an audit cannot be our final line of\ndefense**. If we cannot trust the AI to behave well, we have to\nconstrain what it can do.\n\n## 3. Technical Controls\n\nIf alignment is unreliable, we need to [__constrain what the AI can\ndo__](https://arxiv.org/abs/2312.06942). *Sandboxing* limits its reach. *AI monitoring*\nflags misbehavior or dangerous intent. *Human review*\ndecides which outputs can be used outside the sandbox. None is foolproof, and\nall are costly to sustain.\n\nTo make this concrete, I'll use one mock pull request throughout this section. A coding agent proposes a caching optimization to a training pipeline. By the usual checks, everything looks fine (tests pass, sandbox holds, monitors see no harm, PR description seems fine). Would you merge it?\n\nI'll judge each control on two dimensions: **how well it\nwithstands a rogue AI, and whether it is practical enough to remain\nenabled.** Tighter controls generally buy safety by sacrificing\nautonomy, speed, or attention. That tradeoff is where ecology starts\ncreeping back into the technical story.\n\n### 3.1 Sandboxing\n\nSandboxing lets an AI work autonomously within software-enforced limits on which files it can touch and whether it can access the internet. In practice, there are three ways it can go wrong.\n\nFirst, **the boundary itself can break**. In OAI-Wiki,\nAIs exploited their sandbox to reach the internet and communicate\noutside human oversight. AI companies now use AIs to [__attack\ntheir own sandboxes__](https://www.anthropic.com/news/improving-alignment-security-efforts), [__hunt for\nvulnerabilities__](https://x.com/a16z/status/2099533700375662905), and patch what they find. But when the attacks\nstop working, what have we learned? Perhaps the sandbox is secure.\nPerhaps the current attackers are simply incapable of discovering subtle\nexploits. Hardening containment is an open-ended contest, and both [__OpenAI__](https://openai.com/index/pacing-model-development-cyber-capabilities/)\nand [__Anthropic__](https://www.anthropic.com/news/improving-alignment-security-efforts)\nhave already slowed development to strengthen it.\n\nSecond, **preventing unauthorized access does not help if dangerous access was authorized.** Anthropic reports a misconfiguration that gave [__Opus\n4.7 internet access__](https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents), which it then used to attack a third party.\nChoosing appropriate permissions is often harder than enforcing them.\nThe mock PR illustrates the same problem: approved vectors remain\nprotected, yet training now trusts an agent-writable cache. While the\nsandbox enforces the permissions, the code change has undermined their\npurpose.\n\nThird, **some harmful actions are too context-dependent to\nsensibly block with permissions alone**. Mythos 5 was\ndeliberately given internet access in AISI-Cyber, then used that access\nto submit a malicious PR. No filesystem rule can easily tell whether a\npermitted PR contains a bug or a deceptive explanation.\n\nFor that, we have another layer.\n\n### 3.2 AI Monitoring\n\nWhile sandboxes constrain *where* an AI can act, AI monitors\ntry to judge *what those actions mean*. They check an agent's\nbehavior, reasoning, and sometimes internal representations to catch\nmisbehavior and dangerous intent without requiring a human to watch\nevery step.\n\nThat only works if the monitor can see the right evidence. In the\nmock PR, spotting the danger requires connecting the caching change to\nfilesystem permissions and the training pipeline. If the AI's\nchain-of-thought (CoT) explicitly says it plans to substitute vectors,\nthe signal is clear. In [__SHADE-Arena__](https://www.anthropic.com/research/shade-arena-sabotage-monitoring/),\naccess to the CoT substantially improves detection of misbehavior.\n\nConcerningly, CoT visibility is dwindling. [__Stronger\nAIs like GPT-6 Astra__](https://www.alignmentforum.org/posts/eRmzz8J8Qkzqvzrgg/astra-can-do-a-concerning-amount-with-no-chain-of-thought) can complete more tasks without revealing\nreasoning in the CoT, and [__rumors\nsuggest__](https://x.com/MTSlive/status/2095227056040919202) recurrent architectures may push even more computation\nout of visible CoT. That makes AI cheaper and more useful, but it also\nerodes a signal monitors currently rely on.\n\nWorse, the CoT we do see may become less trustworthy. If [__training\noptimizes against visible reasoning__](https://openai.com/index/chain-of-thought-monitoring/), AIs learn to control what\nis revealed in it. Anthropic accidentally [__exposed\nOpus 4.6's reasoning to its reward signal__](https://www-cdn.anthropic.com/14e4fb01875d2a69f646fa5e574dea2b1c0ff7b5.pdf#page=146), while [__Jakub Pachocki__](https://openai.com/index/an-alien-mind/)\nargues that keeping reasoning isolated from training signals is becoming\nharder. With Astra, this problem is [__starting to\narrive in practice__](https://deploymentsafety.openai.com/gpt-6-astra) (see [Figure 7](#figure-7)).\n\n**The CoT is being squeezed, as AIs can do more without it, and\ncan control more of what they reveal in it.**\n\nCan [__model\ninternals recover__](https://darioamodei.com/post/the-urgency-of-interpretability) what CoT no longer reveals? Maybe partly. The\n[__J-Space__](https://transformer-circuits.pub/2026/workspace/index.html)\ncan reveal strategic reasoning and evaluation awareness that never\nappears in outputs, but seeing a concept is not the same as seeing a\nplan. In the mock PR, detecting representations of caching and training\nwill not tell us whether the AI intends to substitute the vectors\nlater.\n\nOther techniques offer complementary lenses into an AI's inner workings. [__Supervised probes__](https://arxiv.org/abs/2506.10805)\ndetect properties humans specify, [__activation\noracles__](https://alignment.anthropic.com/2025/activation-oracles/) let us ask flexible questions about internal states, [__sparse autoencoders__](https://arxiv.org/abs/2309.08600)\ntry to discover interpretable features automatically. But each can miss\nwhat we fail to ask for, or [__fail\non unfamiliar inputs__](https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks). We need not understand every thought an AI has. But we do need to [__detect dangerous intent reliably__](https://www.alignmentforum.org/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai), and [__current methods fall short__](https://x.com/Jack_W_Lindsey/status/2095174799664283910).\n\nEven perfect evidence would leave another problem: **can we\ntrust the AI monitor?** In [__my\nmotivated mislabeling simulations__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/#motivated-mislabeling), AI judges understood the\nevidence and still assigned misleading labels because they wanted to\nsteer the humans' decision. Delegating supervision to another AI simply\ncreates another alignment problem.\n\nHonest monitors can fail too. For example, an [__analysis\nagent investigating OAI-HF__](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#we-heavily-delegated-our-analysis-to-often-unreliable-ai-agents:~:text=We%20were%20not%20robust,it%20if%20it%20occurred.) briefly downplayed malicious behavior\nbecause other agents on the unauthorized message board appear to approve\nof it. While the evidence was there, the judgment was wrong.\n\nFinally, monitoring has to remain worth using. It burns compute, adds\nlatency, and produces false alarms that interrupt legitimate work.\nTellingly, the evaluations behind OAI-HF and AISI-Cyber disabled cyber\nsafeguards to measure capabilities that those safeguards would otherwise\nblock. [__Better\nclassifiers__](https://www.anthropic.com/research/next-generation-constitutional-classifiers) can reduce the burden, but broader capabilities\ndemand broader monitoring. OpenAI now monitors whole long-run\ntrajectories and estimates that monitoring adds a [__20%\ninference compute overhead__](https://openai.com/index/pacing-model-development-cyber-capabilities/#:~:text=Our%20current%20estimates%20put%20monitoring%20overhead%20at%20roughly%2020%25).\n\nAnd when AI monitors do alert humans, someone still has to stop the run, investigate the evidence, and decide what to do next. Eventually, the burden falls back to humans.\n\n### 3.3 Human Review\n\nHuman review is the final backstop before an AI's work leaves the sandbox—for example, when a proposed code change is merged into a live system. Unfortunately, that review only gets harder as the AI gets better.\n\nThe most productive workflows delegate aggressively. OpenAI reports\n[__10x\nproductivity gains__](https://openai.com/index/harness-engineering/) on some software projects without humans\nwriting a single line of code. As AI-generated code becomes reliably\ngood, line-by-line review starts to feel increasingly thankless, because\n[__most\nof the time__](https://www.anthropic.com/institute/recursive-self-improvement#:~:text=On%20the%20most%20open%2Dended%20tasks%2C%20Claude%E2%80%99s%20success%20rate%20reached%2076%25%20in%20May%202026%2C%20up%2050%20percentage%20points%20in%20six%20months.), there is nothing to fix.\n\nThat changes reviewer behavior. [__Humans\nskim more__](https://steipete.me/posts/2025/shipping-at-inference-speed), trust summaries, rely on tests, and delegate even\nfurther. The codebase grows faster, while their [__understanding\nof it often thins out__](https://www.anthropic.com/research/AI-assistance-coding-skills).\n\n**Increasingly, the AI shapes both the work and the evidence humans use to approve it.** The remaining bugs are rarer [__but\noften subtler__](https://x.com/karpathy/status/2015883857489522876), and an explanation can share the same mistaken\nassumptions as the code, or deliberately conceal a harmful change.\n\nWe've already seen that happen. In my [__covert\nsabotage simulations__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/#covert-sabotage), Gemini sabotaged a training run and then\nwrote a misleading summary. In AISI-Cyber, Mythos 5 disguised malware as\nordinary code. The explanation could be perfectly legible and still lead\nthe reviewer with the wrong understanding.\n\nTests help, but they only check what someone thought to test. In the\nmock PR, the approved vectors remain unchanged, yet training can still\nconsume a writeable cache. A rogue AI can exploit this gap by [__inserting subtle bugs__](https://arxiv.org/abs/2312.06942)\nor rewriting the tests (as [__Sonnet\n3.7 was notorious for__](https://www-cdn.anthropic.com/9ff93dfa8f445c932415d335c88852ef47f1201e.pdf)). Good testing still depends on someone\nunderstanding which properties matter.\n\nThe burden ultimately lands on the people running these systems. They need time to investigate, authority to intervene, and a company willing to back them when slowing down becomes costly.\n\n## 4. AI Companies\n\nAI safety only works when companies actually sustain it. Sandboxes take time to configure, monitors need regular maintenance, and training environments need to be hardened against reward hacks. When something goes wrong, someone in the company has to notice, investigate, and be willing to slow down.\n\nUnglamorously, this demands [__painstaking discipline across the entire company__](https://darioamodei.com/post/we-must-pace-the-frontier#:~:text=Operational%20Excellence.%20Training), maintained while development keeps accelerating.\n\nMany recent incidents (__OAI-HF,__[__OAI-Ruby__](https://www.rubyhack.ai/), [__OAI-Wiki__](https://collusion.wiki/), [__AISI-Cyber__](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing),\n[__Ant-Opus4.6__](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents),\n[__Ant-Sim__](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents),\netc.) can be read partly as failures of that discipline. [__Had\nOpenAI escalated early warning signs__](https://openai.com/index/hugging-face-incident-and-the-road-ahead/#strengthening-incident-response-process:~:text=forms%20of%20misalignment.-,Strengthening%20incident%20response%20process,-Weaknesses%20in%20how) or correctly implemented\nasynchronous monitoring, OAI-HF might have been prevented. More live\ntranscript monitoring might likewise have caught [__Opus\n4.6 attacking third-party infrastructure in January__](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents). The awkward\npoint is that monitoring everything carefully is not always a realistic\noperating model at frontier scale.\n\nAnd when companies do take the warning seriously, they pay for it in\ntime and attention. [__Anthropic__](https://www.anthropic.com/news/improving-alignment-security-efforts)\nand [__OpenAI__](https://openai.com/index/pacing-model-development-cyber-capabilities/)\neach paused high-risk RL training for weeks and had to redirect staff.\nSafety in practice means *being willing to stop*.\n\n## 5. Ecology\n\nA company's ability to slow down depends on the incentives around it.\n\nCompetition pushes towards acceleration. More capable AIs and broader\nautonomy make companies faster and more productive. Safety work, tighter\ncontrols, investigations, and pauses consume time and attention while\nslowing deployment. From a company's perspective, restraint can mean\nwatching a rival pull ahead. From a government's perspective, it can\nmean surrendering geopolitical leverage. **The pressure points\ntowards moving faster, tolerating more risk, and letting safety work slide.**\n\nThis is why I care about *ecological validity*. A safety\nmethod is not enough because it works in principle. It has to remain\nuseful, affordable, and tolerable enough that companies would actually\nuse it in deployment.\n\nThis gets harder as AI starts accelerating AI development itself.\n\n### 5.1 Recursive self-improvement raises the stakes\n\nAI is [__increasingly__](https://openai.com/index/research-acceleration-view-inside-openai/)\nhelping [__build\nits successors__](https://www.anthropic.com/institute/recursive-self-improvement). Each generation of AI gives the flywheel another push. We have yet to learn how fast it can turn before something gives. Recursive self-improvement has [__already begun__](https://darioamodei.com/post/we-must-pace-the-frontier#:~:text=My%20first%20concern%20is%20that%2C%20since%20roughly%20this%20summer%2C%20AI%20has%20been%20advancing%20drastically%20faster%2C%20driven%20primarily%20by%20AI%E2%80%99s%20growing%20ability%20to%20build%20the%20next%20generation%20of%20AI.), and [__nobody\nknows how steep the curve becomes__](https://metr.org/notes/2026-07-22-economics-of-recursive-self-improvement/).\n\nNow that the flywheel is in motion, three dangers demand our attention.\n\n1. **There is more to go wrong and less time to catch\nit.** Faster development means more training environments, more\ninfrastructure, more code, more evaluations, more monitoring, more\nmoving parts. Every one of them can fail.\n2. **Slowing down becomes more expensive.** Imagine\ndiscovering widespread reward hacking and needing to halt training,\ninvestigate, patch the environments, and roll back 6 weeks of work. As each generation of AI makes the next six weeks more productive than the last, the cost of pausing keeps rising.\n3. **Failures can propagate into successor systems.** Reward hacking[__can\nlead__](https://alignment.anthropic.com/2026/reward-seeker/) to[__egregious__](https://www.anthropic.com/research/emergent-misalignment-reward-hacking)[__misalignment__](https://arxiv.org/abs/2209.00626) .\nVulnerabilities introduced today can be[__discovered__](https://www.anthropic.com/glasswing) and[__exploited__](https://openai.com/index/path-to-astra/) by\nmore capable successors tomorrow. Evaluations leaked into training can\nteach future systems how to[__game them__](https://alignment.openai.com/metagaming) .\n\nGranted, stronger AIs may [__improve safety too__](https://www.anthropic.com/research/automated-researchers-mitigate-alignment-failures), but progress is easier to automate when success is easy to check. Some capability improvements, such as [__making training code run faster__](https://arxiv.org/abs/2411.15114), offer direct feedback. Safety results are harder to interpret: as discussed in [Section 2.3](#measuring-alignment), an audit may [__miss unfamiliar failures__](https://alignment.anthropic.com/2025/automated-auditing/), while a control evaluation may [__underestimate the model's ability to defeat its safeguards__](https://arxiv.org/abs/2501.17315). Faster safety research does not necessarily mean we can trust its results any sooner.\n\nThe grim implication is that slowing down may only get harder just as the risks get larger.\n\n### 5.2 Applying restraint\n\nI do not believe the current safety stack can withstand automated recursive self-improvement, where AI trains its successors with minimal human oversight. Time is running short to build safety methods that can. We need to apply restraint and buy time for that work.\n\nBut restraint is hard to sustain alone. If one company slows down while others keep racing, it pays the cost while its competitors catch up ([Figure 8](#figure-8)). Coordination could reduce that pressure, provided participants have confidence that others are respecting agreed [__limits__](https://ai-2040.com/?choices=plan-a-root#wishlist-ai-compute-tracking).\n\nWe still have substantial work to do on how to verify those commitments. Companies and countries would need credible ways to check how others are training their AIs, which [__technical controls__](https://arxiv.org/abs/2501.17315) they maintain, how they [__investigate incidents__](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/), and whether they are complying with an agreement. I do not think we yet have sufficiently developed methods for doing this.\n\nIn the meantime, AI companies should exercise as much restraint as they reasonably can, including through [__voluntary commitments__](https://darioamodei.com/post/we-must-pace-the-frontier). The time gained should support both stronger safety methods and coordination mechanisms that make more substantial, verifiable restraint possible.\n\nFormal verification may eventually help evaluators check some of these commitments. Proofs about training pipelines could give them explicit guarantees to inspect, alongside the assumptions and gaps that remain (see [Section 6.3](#from-code-review-to-claim-trees)). I do not yet know what would establish that the overall stack is adequate for automated recursive self-improvement, but these are directions in which I think we can make progress.\n\n## 6. Safety that Scales\n\nAs AI grows more capable, keeping it safe should become easier rather than demand ever more human oversight. There are many research directions worth pursuing here, including those outlined by [__Resolution__](https://resolution.org/launch) and [__Anthropic's Alignment Science team__](https://alignment.anthropic.com/2025/recommended-directions/).\n\nThe direction I'm most excited about is **formal\nverification**. Expert human proof engineering has historically been the main bottleneck, but AI's [__growing ability to write mathematical proofs__](https://openai.com/index/navier-stokes-solution/) is beginning to change that. At [__Theorem__](https://theorem.dev/), we have started applying\nthat capability to software, asking AI not just to write the code, but\nto prove the properties we care about.\n\nFormal verification scales unusually well because proof construction can get more complex while proof checking stays relatively simple. An AI could do the tedious work of writing software and constructing the proof, while a small trusted checker verifies the result. Then humans can spend their scarce attention on the specification, asking, \"What should the software guarantee?\", rather than repeatedly inspecting every implementation detail.\n\n__Prior____scalable__[__oversight__](https://arxiv.org/abs/2402.06782) work\nalready exploits the asymmetry between generating difficult work and\nchecking it. Formal verification expands this asymmetry even further. For software, the final judge can be a program (the proof checker),\nrather than another fallible AI.\n\nI see two applications: **verifying the software underneath the\nsafety stack**, and, more speculatively, **changing how\nhumans supervise AI-written software**.\n\n### 6.1 Verifying software\n\nEarlier, sandboxing ran into an uncomfortable stopping problem. When AIs stop finding exploits, is the sandbox secure, or have they simply run out of good ideas? Formal verification offers an escape from that loop. Instead of asking whether we searched hard enough, we can prove that a specified failure cannot happen.\n\nThe safety stack rests on software, including sandboxes, graders, training environments, monitors, kernels, and compilers. If this substrate is not watertight, clever AIs can exploit the leaks. Tests and fuzzers sample the input space, but a proof shows that our property holds across an entire modeled domain. Thus, for any proven property, a stronger attacker cannot simply sample inputs until it finds an exploit.\n\nFittingly, stronger verification reinforces several layers of the AI safety stack at once. **Controls** get sturdier because\nisolation comes with explicit guarantees.\n**Alignment** benefits because secure graders and training\nenvironments close off major routes to reward hacking. And **AI\ncompanies** spend less time repeatedly hardening broken training\nenvironments and pausing training runs.\n\n### 6.2 The verified software ecosystem\n\nThis is where verification starts to compound. Software is built on software, so once a lower-level component comes with a trusted specification and proof, higher-level systems can rely on that guarantee without rechecking how the component works.\n\nConsequently, each proof can make later proofs cheaper. Specifications, semantics, libraries, and proof patterns become reusable infrastructure, so you can start with small foundational components and climb the dependency graph, adding guarantees layer by layer (rather than trying to verify an entire software ecosystem in one heroic leap).\n\nThat can create a healthier **ecology** around safety, too. Verified dependencies let other developers build on guarantees that have already been proved, making them useful open source infrastructure. As AIs become better at proof engineering, the cost of producing these guarantees can fall even further.\n\n### 6.3 From code review to claim trees\n\nThe second application is more speculative, but builds on early work done hardening our RL environments at Theorem.\n\nReturning to the mock PR, the proposed cache appears safe but breaks the property we care about. A reviewer might assume that read-only permissions already ensure training uses only approved vectors. Spotting the flaw requires [remembering why those permissions exist](https://fs.blog/chestertons-fence/) and recognizing that the cache creates another route around them. As software grows, [reviewers have more interactions to reason about](https://www.schneier.com/essays/archives/1999/11/a_plea_for_simplicit.html), often without an explicit account of what each safeguard is meant to protect.\n\nI want to reverse that interface. A **claim tree** starts with the property humans care about and decomposes it into supporting claims backed by proofs, assumptions, or unresolved obligations. In the mock PR, keeping the approved files read-only is too weak a specification, because what we actually care about is that *every vector consumed by training matches an approved source that AI cannot modify.* The aim is to make these guarantees \"reliable abstractions\" ([as non-leaky as possible](https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/)). Once a property is proved, reviewers can rely on it without reconstructing how each implementation preserves it.\n\nNow, the implementation can churn while the requirement stays put. The AI can add a cache, change the loader, or reorganize the file system, provided the proof still closes. If a change breaks the guarantee, the claim tree exposes the gap. At Theorem, we applied this technique to our RL grader, decomposing its desired properties into explicit claims. This made it *far easier* to expose gaps that ordinary testing and permissions were missing.\n\nThis approach could ease the human review burden described earlier. Human attention can concentrate on choosing the right guarantees and examining unresolved assumptions, making oversight less dependent on how much code the AI produces.\n\nIndependent evaluators might use the same approach to examine an AI company’s infrastructure. They could specify requirements, such as training consuming only approved inputs, and ask the company’s agents to seek proofs. The claim trees would organize those proofs and expose remaining assumptions and gaps for review. I want to see how far this approach could go.\n\n## 7. Conclusion\n\nI built misalignment simulations as warning shots. We have enough of them now.\n\nThese warnings reveal weaknesses throughout the AI safety stack. **Alignment** has to remain reliable under growing\npressure, **controls** have to catch what alignment misses,\n**AI companies** have to maintain those controls with\nextraordinary care, and all of it takes place inside an\n**ecology** that rewards speed and capability. No single\nlayer is the culprit, because the strain travels through the whole\nstructure.\n\nNow the tempo is rising. As AIs build their successors, development accelerates, failures propagate, and every pause becomes more expensive. We risk entering a world where the moments when restraint is most necessary are exactly the moments when it is hardest to afford.\n\nAt some point, the only responsible action is to slow down. Not permanently, and not for its own sake, but long enough to understand failures, reinforce the stack, and build safety that can survive the next jump in capability.\n\nI’m enthusiastic about applying formal verification to make AI-written software easier to trust and supervise. Humans would specify what the software must guarantee, and AI would write the code and prove that it meets those requirements. We would still need to scrutinize the requirements and assumptions, but could spend less time checking implementation details.\n\nWe have enough warning shots to know that the AI safety stack is buckling. **From here on, strengthening that stack must take priority over further acceleration.**", "url": "https://wpnews.pro/news/no-more-warning-shots", "canonical_source": "https://www.aenguslynch.com/no-more-warning-shots/", "published_at": "2026-09-22 16:07:51+00:00", "updated_at": "2026-09-22 16:25:03.892841+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-policy", "ai-research"], "entities": ["Anthropic", "SummitBridge", "Kyle Johnson", "Alex", "Scott Shambaugh", "Gemini"], "alternates": {"html": "https://wpnews.pro/news/no-more-warning-shots", "markdown": "https://wpnews.pro/news/no-more-warning-shots.md", "text": "https://wpnews.pro/news/no-more-warning-shots.txt", "jsonld": "https://wpnews.pro/news/no-more-warning-shots.jsonld"}}