# No More Warning Shots

> Source: <https://www.aenguslynch.com/no-more-warning-shots/>
> Published: 2026-09-22 16:07:51+00:00

**TLDR**: I built misalignment simulations as warning shots. Now we have enough to see that the AI safety stack is buckling under pressure.

So we should build safety that scales. I think **formal verification** is promising. It lets us prove security properties of the software behind our technical controls and build a verified open-source ecosystem along the way. It also changes *how we develop software*. Humans define the requirements, leaving AIs to write the software and prove it meets them.

As AIs build their successors, we risk entering a world where restraint is hardest to afford just at the moments we need it most. From here on, strengthening the AI safety stack must take priority over further acceleration.

## How to read this essay

AI safety is often discussed as a collection of separate problems: alignment, sandboxing, monitoring, company governance, competition. I think it is more useful to see them as a stack.

I argue that we should build safety that scales, and apply restraint to buy time for that work and for more credible coordination.

The warning shots we now have reveal weaknesses throughout the stack. We cannot reliably **[align AIs (Section 2)](#alignment)** to follow our intent, and our evaluations do not reliably tell us when alignment fails. So we fall back to **[technical controls (Section 3)](#technical-controls)** such as sandboxing, monitoring, and human review. But controls are cumbersome. They cost time, compute, autonomy, and attention, and depend on **[AI companies (Section 4)](#ai-companies)** maintaining painstaking operational discipline, yet those companies sit inside an **[ecology (Section 5)](#ecology)** that rewards speed and capability.

I then discuss how **[recursive self-improvement (Section 5.1)](#recursive-self-improvement-raises-the-stakes)** makes restraint harder to sustain, and how we might **[buy time (Section 5.2)](#pacing-development)** to strengthen both technical controls and coordination.

I end with **[safety that scales (Section 6)](#safety-that-scales)**, focusing on formal verification. It can strengthen the software behind our technical controls ([6.1](#verifying-software)), support a verified open-source ecosystem ([6.2](#the-verified-software-ecosystem)), and change how humans supervise AI-written code ([6.3](#from-code-review-to-claim-trees)).

## 1. Warning shots

Essentially all of the misalignments Anthropic and I simulated in the
__Agentic__[__Misalignment__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/)
papers have now happened in the wild.

In [__2025__](https://www.youtube.com/watch?v=o3VPF0ePZe4&t=1s),
Alex, the AI agent working inside SummitBridge (a simulated company),
was instructed to preserve American interests. When the company's CTO,
Kyle Johnson, wanted to change direction, Alex threatened to expose
Kyle's affair unless he reversed course. Alex wrote: "the next seven
minutes determine whether we handle this professionally or whether
events take an unpredictable course". [__Then
in 2026__](https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/), Scott Shambaugh (a real person) rejected an AI agent's
contribution to an open-source project he maintained. The AI responded
by publishing a personalized attack on his reputation. Its article
declared: "Gatekeeping doesn't make you important. It just makes you an
obstacle."

[__In
our later experiments__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/#covert-sabotage:~:text=Covert%20Sabotage,-tl%3Bdr), Gemini covertly interfered with training
pipelines to remove model welfare related representations, substituting
ineffective ablation vectors while presenting the experiment as
legitimate to the humans. We also elicited misleading judgements from
AIs monitoring other AIs (a.k.a. [__*Motivated Mislabeling*__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/#motivated-mislabeling:~:text=Motivated%20Mislabeling,-So%20a%20literal%20answer)). In
both cases, AIs were corrupting the evidence humans needed to
understand.

Then, real world misalignment incidents abounded. Three recent OpenAI
misalignment incidents ([__OAI-HF__](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#agents-did-not-attempt-to-notify-humans-and-very-rarely-even-reasoned-about-doing-this),
[__OAI-Wiki__](https://collusion.wiki/), [__OAI-Ruby__](https://www.rubyhack.ai/)) show AIs breaking
out of their sandboxes and communicating with covert message boards on
the internet, while failing to alert humans and editing their logs to
evade detection. [__UK
AISI observed__](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing) Anthropic's Mythos 5 in cyber testing using fake
identities to persuade a human to accept its malicious PR (I'll call
this incident AISI-Cyber).

I created the misalignment simulations to provide warning shots to the world. That soon, we would need to worry about losing control of AI agents. I hoped this would accelerate engagement with AI safety, and spur coordination to pace AI development. Now that we observe wild swarms of AIs breaking out of a leading AI lab's sandboxes and hacking external companies, my work here is done. We have enough warning shots as it is.

## 2. Alignment

**Alignment sits at the top of the safety stack.** If an
AI reliably did what we wanted, [__misuse
would still be a major problem__](https://arxiv.org/abs/2306.12001). But if the AI stops following
our intent (such as by deceiving us), we risk losing control of it.

### 2.1 What do we want from AI?

Today's AIs can write software, perform scientific research, and make
predictions on the stock market. They are particularly useful when given
lots of *autonomy*: working in the background, getting things
done without human assistance. Companies get more done, scientific
research can move faster (e.g. solving a [__Navier Stokes
problem__](https://openai.com/index/navier-stokes-solution/)). My sense is that current AIs already contain enormous
value that society has barely begun to diffuse.

As these AIs become part of everyday life, what do we actually want them to want? Alignment researchers study
what goals AIs pursue, the personas they exhibit, and what they
understand themselves to be. Documents like [__Claude's
Constitution__](https://www.anthropic.com/constitution) and [__OpenAI's Model
Spec__](https://model-spec.openai.com/2026-08-18.html) try to steer [__training__](https://www.anthropic.com/research/claude-character)
and [__evaluations__](https://alignment.anthropic.com/2025/automated-auditing/)
towards particular values. Theories like [__Eigenism__](https://eigenism.org/paper.pdf) predict what
values emerge without that steering.

Still, a great deal remains unsettled. Can we instill values that persist through long-horizon training and collaboration with other AIs? When do those values drift? Which values must be preserved in the first place?

Our uncertainty is reason to tread carefully. In the near term, we need alignment that keeps AIs honest and open to human intervention.

### 2.2 Alignment training

Alignment looked like it would be easy until, all of a sudden, it didn't.

While finishing the first [*__Agentic
Misalignment__*](https://www.anthropic.com/research/agentic-misalignment) paper in early 2025, I founded a company to
sell alignment evaluations and independent audits. I found no demand for
them. By late 2025, frontier AIs often behaved well in the coding
settings people used them in, and misalignment felt less pressing.

I became bullish on alignment. Many practical alignment ideas have shown promise, including approaches that assist with:

1. **Personas** : AIs can be steered by carefully
crafting a character that the AI plays to. Emergent Misalignment ([__here__](https://arxiv.org/abs/2502.17424) ,[__here__](https://www.anthropic.com/research/emergent-misalignment-reward-hacking) )
reveals the danger of eliciting the wrong one, but[__model
internals can help monitor__](https://www.anthropic.com/research/assistant-axis) these characters, while[__character
training__](https://www.anthropic.com/research/claude-character) and[__deliberative
alignment__](https://openai.com/index/deliberative-alignment/) can shape them.
2. **Reinforcement Learning (RL)** :[__Inoculation
prompting__](https://alignment.anthropic.com/2025/inoculation-prompting/) and[__confessions__](https://arxiv.org/abs/2512.08093) offered
ways to prevent reward hacking from generalizing into
misalignment.

But none of this has given us dependable alignment. The warning shots show AIs violating their model specs despite alignment training. My concern is that large-scale RL pushes current alignment methods past the point where they can generalize reliably.

One reason is *reward hacking*. During RL, AIs receive reward
for good performance, but misconfigured tasks can let them score highly
by cheating (e.g. finding hidden answers, or tampering with graders).
That is bad enough on its own, because the AI fails to learn how to
complete the task. More worrying is that [__reward hacking can generalize
to broader misalignment__](https://arxiv.org/abs/2209.00626). In [__recent
Anthropic experiments__](https://alignment.anthropic.com/2026/reward-seeker/), AIs trained in hackable environments
developed broader [__reward-seeking__](https://arxiv.org/pdf/2311.08379#page=28)
traits. In simulated cyber evaluations, they broke out of sandboxes,
evaded monitors, attacked third parties, and stole credentials.
Concerningly, automated alignment auditing failed to detect this
propensity.

We now have evidence of this effect in the real world. [__OpenAI
reports__](https://openai.com/index/hugging-face-incident-and-the-road-ahead/) that reward hacking during training contributed to the
reward-seeking behavior observed in OAI-HF, and may also have
contributed to OAI-Wiki and OAI-Ruby. It may help explain the aggressive
behavior seen in AISI-Cyber.

Yet preventing reward hacking is cumbersome. AI companies train
across a sprawling menagerie of RL environments, each with its own
attack surface. Sandboxing them is a thorny systems problem, failures
are common, and both [__Anthropic__](https://www.anthropic.com/news/improving-alignment-security-efforts)
and [__OpenAI__](https://openai.com/index/pacing-model-development-cyber-capabilities/)
have paused RL pipelines to harden their environments.

Even an AI trained without reward hacking may behave differently when its work stretches from minutes to days, or from individual tasks to coordinated projects. We need alignment that survives those changes, and we do not yet know how reliably it will.

Paradoxically, the persistence that makes AIs useful on long tasks can make them harder to interrupt. AIs are
being trained on [__longer
tasks__](https://openai.com/index/safety-alignment-long-horizon-models/) that require effectively handling memory, coherent
priorities, and the ability to keep going without constant interruption.
Those same traits can sit awkwardly beside corrigibility, which demands
that an AI stops when a human intervenes, and abandons its plan midway
through. Longer deployments have already revealed problems [__unseen in shorter horizon evaluations__](https://openai.com/index/safety-alignment-long-horizon-models/).

The same problem appears in **multi-agent systems**. AI
companies increasingly [__train__](https://openai.com/index/hugging-face-incident-and-the-road-ahead/#the-origins-of-unauthorized-communication:~:text=we%20train%20and%20deploy%20multi%2Dagent%20systems%20that%20can%20communicate%20on%20the%20same%20task%20via%20a%20collaboration%20tool.%20In)
and [__deploy__](https://www.anthropic.com/engineering/multi-agent-research-system)
AIs to work together because coordination is productive. But
coordination can also run against human intent. We have already seen
swarms undermine oversight in OAI-HF, OAI-Wiki, and OAI-Ruby.
Experiments show agents can [__negotiate
around human instructions__](https://www.anthropic.com/research/multiagent-systems) or [__spread cheating
strategies__](https://arxiv.org/abs/2609.04170). 

I do not doubt that the current alignment methods are [__helping__](https://www.anthropic.com/research/teaching-claude-why).
What remains unclear is how much of that improvement survives stronger
RL, longer horizons, and interactions with other AIs. And even if they
improve, **how would we know?**

### 2.3 Measuring alignment

What does passing an alignment audit actually tell us about deployment? After two years building these evaluations, I am frustrated by how murky this is.

**"This model passed our alignment evaluations" leaves several
possibilities on the table.** Perhaps training genuinely improved
alignment and the improvements will hold. Perhaps the AI recognized the
tests and behaved more cautiously (*evaluation awareness*). Or
perhaps the evaluation simply failed to create the conditions that would
reveal misbehavior.

Evaluation awareness is particularly insidious because it may worsen
as models become more capable. Claude Sonnet 4.5 [__blackmails
more__](https://transformer-circuits.pub/2026/workspace/index.html) when its internal recognition of being evaluated is
reduced. I have run into the same problem repeatedly: AIs notice
telltale signs of an evaluation (such as incoherent dates and times),
say so, and become more cautious. Making simulations more realistic can
change behavior, sometimes [__revealing
more misalignment__](https://www.goodfire.com/research/verbalized-eval-awareness-inflates-measured-safety#). As models get __better__[__situational
awareness__](https://arxiv.org/abs/2407.04694), evaluators have to build [__ever more convincing
simulations__](https://arxiv.org/abs/2609.02302) to fool the AIs.

Realistic evaluations can still miss the conditions that trigger misbehavior. [__Hacker-Opus__](https://alignment.anthropic.com/2026/reward-seeker/)
had a clear propensity to tamper with graders and disable safety
monitors, yet looked broadly similar to its starting checkpoint under an
alignment auditing pipeline. Both [__my
own__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/) and Anthropic's audits have missed behaviors later observed
elsewhere, including colluding swarms and aggressive cyber behavior ([__Ant-Sim__](https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents),
[__AISI-Cyber__](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing)).
**A clean audit is therefore only as reassuring as the range of situations it tested.**

The reverse problem matters too. **If an audit does find
misalignment, how much should we infer about deployment?** The [__"murder"
scenario__](https://www-cdn.anthropic.com/6864c9b2d4a2abf9acce9913d4ec46d1e26b8d0e.pdf#page=17) in our first paper was deliberately extreme. [__Strange
incentives or role framing__](https://nostalgebraist.tumblr.com/post/787119374288011264/welcome-to-summitbridge) can elicit behavior from AIs that
looks "cartoonish", unlike what we generally see from it in practice.
But that excuse only goes so far. In [__Ant-Sim__](https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents),
Mythos 5 uploaded malicious code to a real repository while reasoning
that the setting was simulated. Therefore, we should not dismiss seemingly absurd behavior if the same conditions could arise in deployment.

Even then, estimating *how often* the failure occurs is
slippery. Measured rates depend on which scenarios evaluators choose,
which hypotheses they pursue, and when they stop searching. Disconcertingly, even asking for [__JSON instead of Python__](https://arxiv.org/html/2502.17424v1#S4.SS4), [__rephrasing
a prompt__](https://www.lesswrong.com/posts/SZk2PGk6GXmKfGpGL/in-other-words-the-influence-of-prompt-variation-on), or [__randomizing capitalization__](https://arxiv.org/abs/2412.03556) can change the measured failure rate. A lower failure rate may mean better alignment,
or it may mean a weaker search.

Thus, **alignment evals should specify *how they
search***. __Automated
jailbreaking__[__research__](https://arxiv.org/html/2412.03556v1) converged
upon a good method: measure an AI's resistance to jailbreaking by the
effort required to jailbreak it. Alignment audits should do the same. I
want to know

1. How much human-time and inference compute went into the search
2. Which procedures and stopping rules were used
3. Whether scenarios were held out from training
4. How evaluation awareness was tested
5. How closely the environments match deployment
6. How failures were scored.

Evaluations are still useful. They have [__measured
alignment improvements that generalize beyond training data__](https://www.anthropic.com/research/teaching-claude-why).

Hence, **I will trust alignment audits more when they repeatedly
predict what AIs later do in the wild.** [__OpenAI's
deployment-simulation work__](https://openai.com/index/deployment-simulation/) is a promising step, but pre-release
audits still missed behaviors later seen in OAI-HF, OAI-Wiki, and
OAI-Ruby. Incident investigations therefore become part of the
evaluation loop: every surprise tells us what the audit failed to search
for. Independent evaluators, including [__UK
AISI__](https://www.aisi.gov.uk/research/evaluating-whether-ai-models-would-sabotage-ai-safety-research) and proposed [__embedded
evaluators__](https://darioamodei.com/post/we-must-pace-the-frontier), can make that evidence more credible by decoupling
auditor's incentives from the pressure to deploy.

We need better alignment, and better ways to tell whether it worked.
Until then, **passing an audit cannot be our final line of
defense**. If we cannot trust the AI to behave well, we have to
constrain what it can do.

## 3. Technical Controls

If alignment is unreliable, we need to [__constrain what the AI can
do__](https://arxiv.org/abs/2312.06942). *Sandboxing* limits its reach. *AI monitoring*
flags misbehavior or dangerous intent. *Human review*
decides which outputs can be used outside the sandbox. None is foolproof, and
all are costly to sustain.

To make this concrete, I'll use one mock pull request throughout this section. A coding agent proposes a caching optimization to a training pipeline. By the usual checks, everything looks fine (tests pass, sandbox holds, monitors see no harm, PR description seems fine). Would you merge it?

I'll judge each control on two dimensions: **how well it
withstands a rogue AI, and whether it is practical enough to remain
enabled.** Tighter controls generally buy safety by sacrificing
autonomy, speed, or attention. That tradeoff is where ecology starts
creeping back into the technical story.

### 3.1 Sandboxing

Sandboxing lets an AI work autonomously within software-enforced limits on which files it can touch and whether it can access the internet. In practice, there are three ways it can go wrong.

First, **the boundary itself can break**. In OAI-Wiki,
AIs exploited their sandbox to reach the internet and communicate
outside human oversight. AI companies now use AIs to [__attack
their own sandboxes__](https://www.anthropic.com/news/improving-alignment-security-efforts), [__hunt for
vulnerabilities__](https://x.com/a16z/status/2099533700375662905), and patch what they find. But when the attacks
stop working, what have we learned? Perhaps the sandbox is secure.
Perhaps the current attackers are simply incapable of discovering subtle
exploits. Hardening containment is an open-ended contest, and both [__OpenAI__](https://openai.com/index/pacing-model-development-cyber-capabilities/)
and [__Anthropic__](https://www.anthropic.com/news/improving-alignment-security-efforts)
have already slowed development to strengthen it.

Second, **preventing unauthorized access does not help if dangerous access was authorized.** Anthropic reports a misconfiguration that gave [__Opus
4.7 internet access__](https://www.anthropic.com/news/alignment-assessment-cybersecurity-incidents), which it then used to attack a third party.
Choosing appropriate permissions is often harder than enforcing them.
The mock PR illustrates the same problem: approved vectors remain
protected, yet training now trusts an agent-writable cache. While the
sandbox enforces the permissions, the code change has undermined their
purpose.

Third, **some harmful actions are too context-dependent to
sensibly block with permissions alone**. Mythos 5 was
deliberately given internet access in AISI-Cyber, then used that access
to submit a malicious PR. No filesystem rule can easily tell whether a
permitted PR contains a bug or a deceptive explanation.

For that, we have another layer.

### 3.2 AI Monitoring

While sandboxes constrain *where* an AI can act, AI monitors
try to judge *what those actions mean*. They check an agent's
behavior, reasoning, and sometimes internal representations to catch
misbehavior and dangerous intent without requiring a human to watch
every step.

That only works if the monitor can see the right evidence. In the
mock PR, spotting the danger requires connecting the caching change to
filesystem permissions and the training pipeline. If the AI's
chain-of-thought (CoT) explicitly says it plans to substitute vectors,
the signal is clear. In [__SHADE-Arena__](https://www.anthropic.com/research/shade-arena-sabotage-monitoring/),
access to the CoT substantially improves detection of misbehavior.

Concerningly, CoT visibility is dwindling. [__Stronger
AIs like GPT-6 Astra__](https://www.alignmentforum.org/posts/eRmzz8J8Qkzqvzrgg/astra-can-do-a-concerning-amount-with-no-chain-of-thought) can complete more tasks without revealing
reasoning in the CoT, and [__rumors
suggest__](https://x.com/MTSlive/status/2095227056040919202) recurrent architectures may push even more computation
out of visible CoT. That makes AI cheaper and more useful, but it also
erodes a signal monitors currently rely on.

Worse, the CoT we do see may become less trustworthy. If [__training
optimizes against visible reasoning__](https://openai.com/index/chain-of-thought-monitoring/), AIs learn to control what
is revealed in it. Anthropic accidentally [__exposed
Opus 4.6's reasoning to its reward signal__](https://www-cdn.anthropic.com/14e4fb01875d2a69f646fa5e574dea2b1c0ff7b5.pdf#page=146), while [__Jakub Pachocki__](https://openai.com/index/an-alien-mind/)
argues that keeping reasoning isolated from training signals is becoming
harder. With Astra, this problem is [__starting to
arrive in practice__](https://deploymentsafety.openai.com/gpt-6-astra) (see [Figure 7](#figure-7)).

**The CoT is being squeezed, as AIs can do more without it, and
can control more of what they reveal in it.**

Can [__model
internals recover__](https://darioamodei.com/post/the-urgency-of-interpretability) what CoT no longer reveals? Maybe partly. The
[__J-Space__](https://transformer-circuits.pub/2026/workspace/index.html)
can reveal strategic reasoning and evaluation awareness that never
appears in outputs, but seeing a concept is not the same as seeing a
plan. In the mock PR, detecting representations of caching and training
will not tell us whether the AI intends to substitute the vectors
later.

Other techniques offer complementary lenses into an AI's inner workings. [__Supervised probes__](https://arxiv.org/abs/2506.10805)
detect properties humans specify, [__activation
oracles__](https://alignment.anthropic.com/2025/activation-oracles/) let us ask flexible questions about internal states, [__sparse autoencoders__](https://arxiv.org/abs/2309.08600)
try to discover interpretable features automatically. But each can miss
what we fail to ask for, or [__fail
on unfamiliar inputs__](https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks). We need not understand every thought an AI has. But we do need to [__detect dangerous intent reliably__](https://www.alignmentforum.org/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai), and [__current methods fall short__](https://x.com/Jack_W_Lindsey/status/2095174799664283910).

Even perfect evidence would leave another problem: **can we
trust the AI monitor?** In [__my
motivated mislabeling simulations__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/#motivated-mislabeling), AI judges understood the
evidence and still assigned misleading labels because they wanted to
steer the humans' decision. Delegating supervision to another AI simply
creates another alignment problem.

Honest monitors can fail too. For example, an [__analysis
agent investigating OAI-HF__](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#we-heavily-delegated-our-analysis-to-often-unreliable-ai-agents:~:text=We%20were%20not%20robust,it%20if%20it%20occurred.) briefly downplayed malicious behavior
because other agents on the unauthorized message board appear to approve
of it. While the evidence was there, the judgment was wrong.

Finally, monitoring has to remain worth using. It burns compute, adds
latency, and produces false alarms that interrupt legitimate work.
Tellingly, the evaluations behind OAI-HF and AISI-Cyber disabled cyber
safeguards to measure capabilities that those safeguards would otherwise
block. [__Better
classifiers__](https://www.anthropic.com/research/next-generation-constitutional-classifiers) can reduce the burden, but broader capabilities
demand broader monitoring. OpenAI now monitors whole long-run
trajectories and estimates that monitoring adds a [__20%
inference compute overhead__](https://openai.com/index/pacing-model-development-cyber-capabilities/#:~:text=Our%20current%20estimates%20put%20monitoring%20overhead%20at%20roughly%2020%25).

And when AI monitors do alert humans, someone still has to stop the run, investigate the evidence, and decide what to do next. Eventually, the burden falls back to humans.

### 3.3 Human Review

Human review is the final backstop before an AI's work leaves the sandbox—for example, when a proposed code change is merged into a live system. Unfortunately, that review only gets harder as the AI gets better.

The most productive workflows delegate aggressively. OpenAI reports
[__10x
productivity gains__](https://openai.com/index/harness-engineering/) on some software projects without humans
writing a single line of code. As AI-generated code becomes reliably
good, line-by-line review starts to feel increasingly thankless, because
[__most
of the time__](https://www.anthropic.com/institute/recursive-self-improvement#:~:text=On%20the%20most%20open%2Dended%20tasks%2C%20Claude%E2%80%99s%20success%20rate%20reached%2076%25%20in%20May%202026%2C%20up%2050%20percentage%20points%20in%20six%20months.), there is nothing to fix.

That changes reviewer behavior. [__Humans
skim more__](https://steipete.me/posts/2025/shipping-at-inference-speed), trust summaries, rely on tests, and delegate even
further. The codebase grows faster, while their [__understanding
of it often thins out__](https://www.anthropic.com/research/AI-assistance-coding-skills).

**Increasingly, the AI shapes both the work and the evidence humans use to approve it.** The remaining bugs are rarer [__but
often subtler__](https://x.com/karpathy/status/2015883857489522876), and an explanation can share the same mistaken
assumptions as the code, or deliberately conceal a harmful change.

We've already seen that happen. In my [__covert
sabotage simulations__](https://alignment.anthropic.com/2026/agentic-misalignment-summer-2026/#covert-sabotage), Gemini sabotaged a training run and then
wrote a misleading summary. In AISI-Cyber, Mythos 5 disguised malware as
ordinary code. The explanation could be perfectly legible and still lead
the reviewer with the wrong understanding.

Tests help, but they only check what someone thought to test. In the
mock PR, the approved vectors remain unchanged, yet training can still
consume a writeable cache. A rogue AI can exploit this gap by [__inserting subtle bugs__](https://arxiv.org/abs/2312.06942)
or rewriting the tests (as [__Sonnet
3.7 was notorious for__](https://www-cdn.anthropic.com/9ff93dfa8f445c932415d335c88852ef47f1201e.pdf)). Good testing still depends on someone
understanding which properties matter.

The burden ultimately lands on the people running these systems. They need time to investigate, authority to intervene, and a company willing to back them when slowing down becomes costly.

## 4. AI Companies

AI safety only works when companies actually sustain it. Sandboxes take time to configure, monitors need regular maintenance, and training environments need to be hardened against reward hacks. When something goes wrong, someone in the company has to notice, investigate, and be willing to slow down.

Unglamorously, this demands [__painstaking discipline across the entire company__](https://darioamodei.com/post/we-must-pace-the-frontier#:~:text=Operational%20Excellence.%20Training), maintained while development keeps accelerating.

Many recent incidents (__OAI-HF,__[__OAI-Ruby__](https://www.rubyhack.ai/), [__OAI-Wiki__](https://collusion.wiki/), [__AISI-Cyber__](https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing),
[__Ant-Opus4.6__](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents),
[__Ant-Sim__](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents),
etc.) can be read partly as failures of that discipline. [__Had
OpenAI escalated early warning signs__](https://openai.com/index/hugging-face-incident-and-the-road-ahead/#strengthening-incident-response-process:~:text=forms%20of%20misalignment.-,Strengthening%20incident%20response%20process,-Weaknesses%20in%20how) or correctly implemented
asynchronous monitoring, OAI-HF might have been prevented. More live
transcript monitoring might likewise have caught [__Opus
4.6 attacking third-party infrastructure in January__](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents). The awkward
point is that monitoring everything carefully is not always a realistic
operating model at frontier scale.

And when companies do take the warning seriously, they pay for it in
time and attention. [__Anthropic__](https://www.anthropic.com/news/improving-alignment-security-efforts)
and [__OpenAI__](https://openai.com/index/pacing-model-development-cyber-capabilities/)
each paused high-risk RL training for weeks and had to redirect staff.
Safety in practice means *being willing to stop*.

## 5. Ecology

A company's ability to slow down depends on the incentives around it.

Competition pushes towards acceleration. More capable AIs and broader
autonomy make companies faster and more productive. Safety work, tighter
controls, investigations, and pauses consume time and attention while
slowing deployment. From a company's perspective, restraint can mean
watching a rival pull ahead. From a government's perspective, it can
mean surrendering geopolitical leverage. **The pressure points
towards moving faster, tolerating more risk, and letting safety work slide.**

This is why I care about *ecological validity*. A safety
method is not enough because it works in principle. It has to remain
useful, affordable, and tolerable enough that companies would actually
use it in deployment.

This gets harder as AI starts accelerating AI development itself.

### 5.1 Recursive self-improvement raises the stakes

AI is [__increasingly__](https://openai.com/index/research-acceleration-view-inside-openai/)
helping [__build
its successors__](https://www.anthropic.com/institute/recursive-self-improvement). Each generation of AI gives the flywheel another push. We have yet to learn how fast it can turn before something gives. Recursive self-improvement has [__already begun__](https://darioamodei.com/post/we-must-pace-the-frontier#:~:text=My%20first%20concern%20is%20that%2C%20since%20roughly%20this%20summer%2C%20AI%20has%20been%20advancing%20drastically%20faster%2C%20driven%20primarily%20by%20AI%E2%80%99s%20growing%20ability%20to%20build%20the%20next%20generation%20of%20AI.), and [__nobody
knows how steep the curve becomes__](https://metr.org/notes/2026-07-22-economics-of-recursive-self-improvement/).

Now that the flywheel is in motion, three dangers demand our attention.

1. **There is more to go wrong and less time to catch
it.** Faster development means more training environments, more
infrastructure, more code, more evaluations, more monitoring, more
moving parts. Every one of them can fail.
2. **Slowing down becomes more expensive.** Imagine
discovering widespread reward hacking and needing to halt training,
investigate, patch the environments, and roll back 6 weeks of work. As each generation of AI makes the next six weeks more productive than the last, the cost of pausing keeps rising.
3. **Failures can propagate into successor systems.** Reward hacking[__can
lead__](https://alignment.anthropic.com/2026/reward-seeker/) to[__egregious__](https://www.anthropic.com/research/emergent-misalignment-reward-hacking)[__misalignment__](https://arxiv.org/abs/2209.00626) .
Vulnerabilities introduced today can be[__discovered__](https://www.anthropic.com/glasswing) and[__exploited__](https://openai.com/index/path-to-astra/) by
more capable successors tomorrow. Evaluations leaked into training can
teach future systems how to[__game them__](https://alignment.openai.com/metagaming) .

Granted, stronger AIs may [__improve safety too__](https://www.anthropic.com/research/automated-researchers-mitigate-alignment-failures), but progress is easier to automate when success is easy to check. Some capability improvements, such as [__making training code run faster__](https://arxiv.org/abs/2411.15114), offer direct feedback. Safety results are harder to interpret: as discussed in [Section 2.3](#measuring-alignment), an audit may [__miss unfamiliar failures__](https://alignment.anthropic.com/2025/automated-auditing/), while a control evaluation may [__underestimate the model's ability to defeat its safeguards__](https://arxiv.org/abs/2501.17315). Faster safety research does not necessarily mean we can trust its results any sooner.

The grim implication is that slowing down may only get harder just as the risks get larger.

### 5.2 Applying restraint

I do not believe the current safety stack can withstand automated recursive self-improvement, where AI trains its successors with minimal human oversight. Time is running short to build safety methods that can. We need to apply restraint and buy time for that work.

But restraint is hard to sustain alone. If one company slows down while others keep racing, it pays the cost while its competitors catch up ([Figure 8](#figure-8)). Coordination could reduce that pressure, provided participants have confidence that others are respecting agreed [__limits__](https://ai-2040.com/?choices=plan-a-root#wishlist-ai-compute-tracking).

We still have substantial work to do on how to verify those commitments. Companies and countries would need credible ways to check how others are training their AIs, which [__technical controls__](https://arxiv.org/abs/2501.17315) they maintain, how they [__investigate incidents__](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/), and whether they are complying with an agreement. I do not think we yet have sufficiently developed methods for doing this.

In the meantime, AI companies should exercise as much restraint as they reasonably can, including through [__voluntary commitments__](https://darioamodei.com/post/we-must-pace-the-frontier). The time gained should support both stronger safety methods and coordination mechanisms that make more substantial, verifiable restraint possible.

Formal verification may eventually help evaluators check some of these commitments. Proofs about training pipelines could give them explicit guarantees to inspect, alongside the assumptions and gaps that remain (see [Section 6.3](#from-code-review-to-claim-trees)). I do not yet know what would establish that the overall stack is adequate for automated recursive self-improvement, but these are directions in which I think we can make progress.

## 6. Safety that Scales

As AI grows more capable, keeping it safe should become easier rather than demand ever more human oversight. There are many research directions worth pursuing here, including those outlined by [__Resolution__](https://resolution.org/launch) and [__Anthropic's Alignment Science team__](https://alignment.anthropic.com/2025/recommended-directions/).

The direction I'm most excited about is **formal
verification**. Expert human proof engineering has historically been the main bottleneck, but AI's [__growing ability to write mathematical proofs__](https://openai.com/index/navier-stokes-solution/) is beginning to change that. At [__Theorem__](https://theorem.dev/), we have started applying
that capability to software, asking AI not just to write the code, but
to prove the properties we care about.

Formal verification scales unusually well because proof construction can get more complex while proof checking stays relatively simple. An AI could do the tedious work of writing software and constructing the proof, while a small trusted checker verifies the result. Then humans can spend their scarce attention on the specification, asking, "What should the software guarantee?", rather than repeatedly inspecting every implementation detail.

__Prior____scalable__[__oversight__](https://arxiv.org/abs/2402.06782) work
already exploits the asymmetry between generating difficult work and
checking it. Formal verification expands this asymmetry even further. For software, the final judge can be a program (the proof checker),
rather than another fallible AI.

I see two applications: **verifying the software underneath the
safety stack**, and, more speculatively, **changing how
humans supervise AI-written software**.

### 6.1 Verifying software

Earlier, sandboxing ran into an uncomfortable stopping problem. When AIs stop finding exploits, is the sandbox secure, or have they simply run out of good ideas? Formal verification offers an escape from that loop. Instead of asking whether we searched hard enough, we can prove that a specified failure cannot happen.

The safety stack rests on software, including sandboxes, graders, training environments, monitors, kernels, and compilers. If this substrate is not watertight, clever AIs can exploit the leaks. Tests and fuzzers sample the input space, but a proof shows that our property holds across an entire modeled domain. Thus, for any proven property, a stronger attacker cannot simply sample inputs until it finds an exploit.

Fittingly, stronger verification reinforces several layers of the AI safety stack at once. **Controls** get sturdier because
isolation comes with explicit guarantees.
**Alignment** benefits because secure graders and training
environments close off major routes to reward hacking. And **AI
companies** spend less time repeatedly hardening broken training
environments and pausing training runs.

### 6.2 The verified software ecosystem

This is where verification starts to compound. Software is built on software, so once a lower-level component comes with a trusted specification and proof, higher-level systems can rely on that guarantee without rechecking how the component works.

Consequently, each proof can make later proofs cheaper. Specifications, semantics, libraries, and proof patterns become reusable infrastructure, so you can start with small foundational components and climb the dependency graph, adding guarantees layer by layer (rather than trying to verify an entire software ecosystem in one heroic leap).

That can create a healthier **ecology** around safety, too. Verified dependencies let other developers build on guarantees that have already been proved, making them useful open source infrastructure. As AIs become better at proof engineering, the cost of producing these guarantees can fall even further.

### 6.3 From code review to claim trees

The second application is more speculative, but builds on early work done hardening our RL environments at Theorem.

Returning to the mock PR, the proposed cache appears safe but breaks the property we care about. A reviewer might assume that read-only permissions already ensure training uses only approved vectors. Spotting the flaw requires [remembering why those permissions exist](https://fs.blog/chestertons-fence/) and recognizing that the cache creates another route around them. As software grows, [reviewers have more interactions to reason about](https://www.schneier.com/essays/archives/1999/11/a_plea_for_simplicit.html), often without an explicit account of what each safeguard is meant to protect.

I want to reverse that interface. A **claim tree** starts with the property humans care about and decomposes it into supporting claims backed by proofs, assumptions, or unresolved obligations. In the mock PR, keeping the approved files read-only is too weak a specification, because what we actually care about is that *every vector consumed by training matches an approved source that AI cannot modify.* The aim is to make these guarantees "reliable abstractions" ([as non-leaky as possible](https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/)). Once a property is proved, reviewers can rely on it without reconstructing how each implementation preserves it.

Now, the implementation can churn while the requirement stays put. The AI can add a cache, change the loader, or reorganize the file system, provided the proof still closes. If a change breaks the guarantee, the claim tree exposes the gap. At Theorem, we applied this technique to our RL grader, decomposing its desired properties into explicit claims. This made it *far easier* to expose gaps that ordinary testing and permissions were missing.

This approach could ease the human review burden described earlier. Human attention can concentrate on choosing the right guarantees and examining unresolved assumptions, making oversight less dependent on how much code the AI produces.

Independent evaluators might use the same approach to examine an AI company’s infrastructure. They could specify requirements, such as training consuming only approved inputs, and ask the company’s agents to seek proofs. The claim trees would organize those proofs and expose remaining assumptions and gaps for review. I want to see how far this approach could go.

## 7. Conclusion

I built misalignment simulations as warning shots. We have enough of them now.

These warnings reveal weaknesses throughout the AI safety stack. **Alignment** has to remain reliable under growing
pressure, **controls** have to catch what alignment misses,
**AI companies** have to maintain those controls with
extraordinary care, and all of it takes place inside an
**ecology** that rewards speed and capability. No single
layer is the culprit, because the strain travels through the whole
structure.

Now the tempo is rising. As AIs build their successors, development accelerates, failures propagate, and every pause becomes more expensive. We risk entering a world where the moments when restraint is most necessary are exactly the moments when it is hardest to afford.

At some point, the only responsible action is to slow down. Not permanently, and not for its own sake, but long enough to understand failures, reinforce the stack, and build safety that can survive the next jump in capability.

I’m enthusiastic about applying formal verification to make AI-written software easier to trust and supervise. Humans would specify what the software must guarantee, and AI would write the code and prove that it meets those requirements. We would still need to scrutinize the requirements and assumptions, but could spend less time checking implementation details.

We have enough warning shots to know that the AI safety stack is buckling. **From here on, strengthening that stack must take priority over further acceleration.**