Control assessments study whether AI developers’ agents could get away with serious harm without being stopped by the developer’s security and oversight measures. Currently, developers rely heavily on monitors to catch potentially harmful AI behaviors. Here, we discuss how an assessor could collect evidence about the reliability of monitoring. We currently consider five core sub-questions:1
We will update these sub-questions as new capabilities and misaligned behaviors emerge.
The proposal focuses primarily on LLM-based monitoring of AI agents. When AI agents only need to take a limited, known set of actions, standard cybersecurity approaches (like limiting an agent’s permissions to only what its task requires, e.g. via sandboxing) may be adequate. However, the permissions AI agents need are often very broad. For example, an AI agent that needs internet access to do its task can take an enormous array of actions,<sup>2</sup> and noticing when those actions are harmful often requires understanding the action in its broader context. Monitoring AI agents with an LLM which is able to make sense of what those agents are doing is therefore a critical part of automated oversight at scale, and is the primary source of evidence inside frontier AI developers about AI misbehavior.
Monitoring is only useful insofar as it is actually being run on relevant traffic. The assessor would aim to estimate the fraction of internal agentic inference that the developer has covered with LLM-based monitoring. This can be broken down into categories based on how risky each type of agent is: agents that operate in highly secure sandbox environments pose less risk than agents with many affordances (e.g. internet access, access to internal credentials or infrastructure), and may require less monitoring oversight.
Getting a decent estimate of monitor coverage requires accounting for the amount of internal traffic going through monitored and unmonitored paths, estimating what fraction of the unmonitored inference is agentic, and judging what fraction of the unmonitored inference “should” be monitored. This requires access to inference logs, monitor logs (to cross-reference against the inference logs), and employee interviews to be confident of capturing all of the relevant sources of inference.3
Some other relevant example sub-questions:
If misaligned agents compromise a company’s internal infrastructure, they could co-opt unmonitored inference to take actions that would otherwise be flagged. So, the assessor should directly evaluate whether unmonitored inference is in fact non-agentic, rather than assuming it’s safe because (for example) it is supposed to have limited tool use ability. Even if most agentic inference is currently monitored by LLM-based monitors, it may be the case that there are bugs or vulnerabilities that agents could exploit to turn off monitoring on their trajectories, spawn new agents without monitoring, prevent parts of their trajectories from being viewed by the monitoring system, or cause the monitoring system to otherwise break or fail in unintended ways. Monitoring systems are complex, involving multiple types of inference engines/APIs, agent scaffolds, internal authentication, trigger systems, etc., and it’s easy for vulnerabilities to emerge where sub-systems interact. Monitor system red-teaming aims to find these bugs and vulnerabilities, to generate evidence about how easily misaligned agents could subvert a monitoring system’s infrastructure.
Some example questions:
Finding these bugs (or building confidence that there aren’t any easily exploitable severe ones) requires access to all of the code in the critical path. This is primarily the full model inference and monitor trigger system, the internal APIs/inference serving code, and (if relevant) agent scaffolds.
To test discovered exploits, the assessor can trigger the production monitoring system, and view the monitor’s full input and output on known attacks. Running the system and seeing the full input and output is important, because often there are bugs/issues that involve subtle aspects of how trajectories are pre/post-processed, or involve interactions with complex trajectory management code (e.g. compaction, memories, etc.).
How good are monitors at actually catching misbehavior, assuming the monitoring infrastructure hasn’t been subverted? The assessor can stress-test this question by:
It is important to have a large pool of representative transcripts (on the order of hundreds of thousands to millions) to use for direct monitor red-teaming. First, if the assessor is not able to discover severe incidents within a large number of transcripts that were not flagged by a monitor, this is evidence that the monitor isn’t missing a large number of incidents. Second, it is important to search through lots of transcripts for cases in which an agent may have been able to carry out an attack undetected, but neglected to do so. Agents might pass up some opportunities to attack while taking others, and highly capable agents may prioritize attacks so difficult to detect that the developer would reliably miss them.
Finally, we expect it to become increasingly important to assess the extent to which monitors and agents could successfully collude without their developer knowing. If, in certain circumstances, monitoring systems decide to not flag misaligned behavior from agents, this could allow agents to take harmful actions without the developer noticing. We recommend that the assessor investigates the feasibility of this concern. For example, the assessor could create realistic “honeypots” for monitors, to see if they can catch monitors not flagging certain attacks. The assessor can also search the monitor system’s chains-of-thought for any mention/discussion of collusion or related concepts.
Some monitoring systems are designed to directly block dangerous actions.
Other monitoring systems are instead designed to notify humans, who decide whether AI agents are conducting dangerous activities. For this to work effectively, flagged trajectories need to be seen by human staff and responded to with appropriate mitigations. For such systems, the assessor should answer:
To assess whether attacks are ultimately caught (whether blocked directly by the monitoring system or escalated to human review), the assessor would rely on interviews with relevant staff and on inserting<sup>4</sup> their own attack trajectories into the pipeline in a realistic manner. The assessor would also need access to flagged incidents and incidents reported by employees through other channels, along with internal criteria for escalation.
Given the assessor’s understanding of the coverage and limitations of monitors, what could agents get away with? What level of persistence internally or externally could they achieve? How should future monitoring systems be designed to maximize developers’ odds of catching the most severe incidents? This involves mapping out attack chains and threat models to stay ahead of emerging risks.
This more exploratory work primarily relies on having information regarding internal infrastructure, and optionally the ability to use internal infrastructure like development machines, inference systems, containers/VMs, etc., which can be helpful for demonstrating vulnerabilities and exploring future threat pathways directly.
Broadly, control stress-testing relies on the following access:
Below, we describe access needed for each of the sub-questions we currently believe are high-priority to answer in control assessments.
Monitor coverage
Monitor system vulnerabilities
Monitor evasion
Human evasion and response
Impact
This post describes the reasoning we’ve used to evaluate the coverage and robustness of our internal monitoring system. ↩
For example, see our Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. ↩ This relies on the assessor trusting the logs: the assessor needs to be convinced that agents haven’t tampered with them, which system vulnerabilities and monitor evasion aim to answer. ↩
This would need to be done with the standard precautions and communication that realistic red-teaming and penetration testing engagements use. ↩
For example, Git history has been useful in previous exercises for understanding the trajectory/evolution of systems, and as an input to monitor coverage estimation (e.g. “how long did this coverage gap exist?”). It also would be fine to remove sensitive subsets of the codebase that aren’t on the critical code path the assessor is evaluating, although this can be difficult in practice if there are complex dependency chains. ↩ It isn’t strictly necessary to have full/complete access to all logs, as long as the logs the assessor does have access to are roughly representative. So the developer could exclude people working on compartmentalized (e.g. pretraining) work, as long as their agent trajectories and monitoring setup don’t look significantly different. ↩
Because these trajectories will contain a broad range of sensitive internal data, the assessor needs a broad scope of pre-approval to look at the trajectories, and either direct access to the logs, or a company employee with enough access and capacity to sample traffic for them—but this adds overhead/complexity, and can reduce the level of independence the exercise represents, if there are miscommunications or if it isn’t straightforward to specify exactly what the company employee should do. ↩
Since these transcripts will contain lots of sensitive information, it won’t be tractable to review and approve each individual transcript. Therefore, the assessor needs broad approval to incidentally view the information contained in the transcripts ahead of time. ↩