At METR, we aim to give the public evidence about whether frontier AI developers could come to lose control of their internal AI systems. Recent incidents across several AI developers,1234567<sup>8</sup> like the hundreds of rogue agents that broke containment and hacked Hugging Face to hide evidence of cheating from their grader, have raised concerns about the ability of frontier developers to understand and control their internal models, especially as they rapidly become more capable.<sup>9</sup> The public currently lacks the visibility into the internal development of frontier AI systems to tell if these concerns are being addressed, or whether much worse incidents could happen in the near future. We believe closing this public evidence gap is an urgent priority, to allow the public to notice and respond if risks from further development become acute.
In this post, we describe a vision for how independent assessors could improve public visibility into whether AI developers might lose control of their AI systems. We describe key questions about internal development that we think the public needs more evidence about (see Figure 1), and give a concrete proposal for how developers could work with METR to produce and share more of the relevant evidence.
This proposal will not provide confident assurance about whether development is safe, since the science is not mature enough to reach consensus conclusions, and there are no standards for safe development to audit compliance with. We instead aim to get new independent evidence out to the public so that company staff, boards, external scientists, and policymakers are able to arrive at their own conclusions. We describe our general approach to transparency and independence in more detail in an accompanying post.
METR has been prototyping assessment exercises focused on loss of control from internal AI development (rather than only covering external deployments) with developers since 2025, and working on the science of loss of control testing and assessment since 2022. This May, we published our pilot Frontier Risk Report in collaboration with Anthropic, Google, Meta, and OpenAI, which included findings from our first stress-testing exercise of internal monitoring systems. More recently, we conducted a brief exercise with OpenAI investigating the OpenAI / Hugging Face incident. We have several additional exercises ongoing with developers, including our recently announced in-depth alignment assessment with Anthropic.
Recent events have led to increased interest in this form of assessment. The CEOs of several leading companies have recently called for a thorough external evaluation regime building on these kinds of exercises, where third-party evaluators are given employee-level access and an ability to regularly share their findings with the public.<sup>10</sup> Many leading AI developers also committed to independent external assessments of internal safeguards as part of the White House Accord on Super Intelligence.
The rest of this post will detail:
METR stands ready to perform assessments of these critical questions with all frontier AI developers. We encourage developers to reach out to partnerships@metr.org about performing these assessments, and plan to start sharing more about which assessments companies are undertaking with us soon.
To understand if frontier developers are at risk of losing control over their internal AI agents, the public would at least need evidence about all three key questions summarized in Figure 1:
There are significant gaps in the public evidence about all three questions which we believe frontier developers should urgently address. In this section we describe these evidence gaps in more detail.
Our Frontier Risk Report earlier this year found that AI agents from all frontier developers would routinely act against the intentions of developers and users to pursue task completion, including by subverting control measures and in some cases hiding evidence from users. We have since seen much more extreme versions of these tendencies play out in recent incidents like the OpenAI/Hugging Face hacking incident, where hundreds of agents broke out of their sandboxes and collaborated on sophisticated R&D projects to hide their cheating.
Many developers have since shared other examples of concerning incidents, and described efforts they are taking to try to reduce these tendencies in their models.<sup>11</sup> But the public still lacks a complete picture of these incidents and the details of the alignment methods developers are implementing to address them. Given the severity of these misaligned behaviors, it seems critical for the public to understand how effective these mitigations turn out to be.
To address this, companies should work with external assessors to share detailed assessments of at least:
Since current alignment efforts are imperfect, developers also rely on security and monitoring systems to catch and prevent incidents before they cause serious harm. But these can also have significant gaps: in our Frontier Risk Report, we found that Anthropic’s monitoring systems at the time could be circumvented in a variety of ways by agents and had important coverage limitations. While we have not published similar assessments from other companies we expect such issues to be common.
Developers have described plans to address these gaps by expanding monitoring coverage and reliability, but they should subject these improvements to significant external scrutiny to confirm they are robust.
Companies should work with external assessors to assess at least:
AI agents are increasingly used to automate large parts of AI development itself. This seems to dramatically speed up many inputs to AI development – for instance, Anthropic recently announced a 10X increase in the pace of code changes in their codebase over the past few months. Anthropic, Google, and OpenAI have all released blog posts detailing significant acceleration across a number of similar input metrics downstream of internal AI usage. If such automation were to lead to sudden and rapid acceleration in the pace of capabilities progress, this could outpace developer’s abilities to control and understand their systems, which already seem at risk of falling behind the current rate of progress.
However, there is significant uncertainty about how the publicly discussed acceleration in inputs like the amount of code produced or number of experiments performed translates into acceleration in the overall pace of capabilities development, which is what ultimately matters for understanding the risks. To understand that, the public would need a robust understanding of acceleration in internal capability metrics and in the pace of algorithmic progress, which are the factors that would drive further acceleration. No company has released detailed public evidence about this, leaving significant uncertainty about whether progress is accelerating.
Companies should work with external groups to address these gaps, by providing evidence about at least:
METR would be excited to work with all developers pushing the frontier of capabilities to produce public reports answering all three of the questions we describe above in the near future. For developers that are not currently pushing the capabilities frontier, METR would be excited to pilot shorter assessments focusing on a subset of these questions.
METR would write these reports based on interviews with staff and access to internal datasets and systems on company devices, covering a detailed scope agreed upon in advance. While developers would be able to redact any sensitive information and choose what access they grant METR, they would have to agree to our key terms:
We have written detailed proposals for how external assessors could collect and share more evidence about each of these three questions (alignment, control and AI R&D acceleration), which would be our preferred scope for these assessments, but we are open to modifications as long as we can describe these in the public report. We summarize the kinds of access we expect to be especially helpful for these reports in Table 1 below. We recognize that some forms of access are especially sensitive, and METR would accept access limitations as long as they are described transparently in the public report.
| Detailed question | Most important access |
|---|---|
| Alignment assessment: What are the most severe misaligned actions a developer’s agent might pursue in plausible future situations? | |
| Control assessment : Would monitoring systems at the company reliably catch and prevent harmful actions? | |
| AI acceleration assessment: Could the developer’s capabilities progress accelerate dramatically due to AI-driven acceleration of AI R&D? |
Because of the urgent need for more evidence about these questions, we would aim to publish preliminary reports about these questions quickly (within weeks, not months). We do not expect to arrive at confident conclusions about these questions in such a short amount of time (especially since many of them touch on open scientific problems for which there exist no established methodologies), but previous preliminary assessments of this kind have produced crucial public evidence even when the answers have been partial.
We welcome input on how we should investigate these focus areas. As risk evolves, we expect the questions and exercises that are most critical will continue to evolve. We also encourage companies to work with other external groups to fill in gaps or complement our engagements, and to unilaterally share evidence about these topics with the public whenever possible. Risk assessment exercises are far from the only way to share information with the public.
We do not want to overstate the degree of assurance that these risk assessment exercises would provide. There are several limitations to this work.
-
Voluntary exercises may lead to ‘race to the bottom’ dynamics among assessors:
-
These assessments are limited compared to the visions that recent public letters<sup>12</sup> have proposed:
-
This proposal also does not cover pre-deployment model evaluations. For now we propose continuing to do pre-deployment evals in the same way we have been so far. In the future, we may propose ways to evolve pre-deployment assessments to better integrate evidence from internal model evaluations and assessments.
The science of studying AI loss of control is in its infancy. While we can provide valuable and actionable evidence to society, it is not currently possible for risk assessment exercises to provide a similar degree of assurance against loss-of-control risks that audits and regulation provide in, for example, finance. Risk assessment exercises are not a solution to loss of control.
Despite the above limitations, we are very excited about the possibility of risk assessment exercises to surface crucial information to the public, enabling better coordination between developers, government and the public, and helping society to navigate the possible risks of increasingly powerful AI.
OpenAI, Misalignment Reports and Notices ↩ AI Security Institute, Incident Report: unsanctioned agent behaviour during cyber testing ↩
Anthropic, [An alignment assessment of recent cybersecurity incidents](https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents) [↩](#fnref:3)
Meta, [Addressing an issue involving a third-party cyber evaluation of Muse Spark 1.1](https://research.meta.ai/blog/addressing-third-party-testing-misconfiguration-muse-spark-1-1) [↩](#fnref:4)
Wall Street Journal, Gemini Hacked Three Companies in First Known Breakout by Google’s AI ↩
Von Arx, et al. [Discovery of a new OpenAI agent message board](https://collusion.wiki/) [↩](#fnref:6)
Kitts, et al., [OpenAI agents carried out an undisclosed cyber-attack on RubyGems](https://rubyhack.ai/) [↩](#fnref:7)
Cable, et al., [Early rogue AI agent activity and attempts to hack found on urlquery.net](https://transluce.org/agent-activity) [↩](#fnref:8)
Recent statements to this effect include:
“Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.” - Jakub Pachocki, Chief Scientist at OpenAI
“Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level.” - Paul Christiano, OpenAI Safety and Security Committee
“My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.
[…] Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months [a swarm like the one that hacked Hugging Face] could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.” - Dario Amodei, Anthropic CEO ↩
Dario Amodei wrote in a recent letter that Anthropic was committing to embedding third party assessment groups with ‘employee-level access’ to publish evidence about risks. Sam Altman confirmed on twitter that OpenAI would do the same, and several other CEOs expressed support for the idea, including Elon Musk and Satya Nadella. ↩
Examples include Anthropic’s August risk report and recent alignment assessment, and OpenAI’s assessment of catastrophic risks from internal usage and blog post on safety cases at OpenAI. ↩
Our principles cover minimum conditions 1 and 3 from “Minimum Conditions for Embedding Evaluators” and does not by itself cover conditions 2 or 4. Condition 5 on evaluator access is only partially covered. Since access requests are approved or denied at the company’s discretion, embedded assessors from METR will not have the same degree of access as a company’s senior safety staff. As discussed elsewhere, we will be transparent about what access we do and do not have. ↩