# Can we build an early warning system for loss of control to AI? > Source: > Published: 2026-07-16 06:15:32+00:00 An early warning system for loss of control to AI requires, at its core, a forecast of the outcome of our current trajectory. Can we build a mathematical model to forecast this? Two fairly intuitive responses occur to me. First: it seems like it would be very difficult. The conceptual underpinnings of "loss of control" are contested, so there's an open question about whether you end up modelling something coherent and operationalizable. Further, there's a well-established line of work arguing that adequately observing a sufficiently capable, deceptive AI may be infeasible — so even if you can write down a model, estimating its parameters may be out of reach. The second response, granting those difficulties, is that if you *could* do it, it would be enormously useful. We lose a lot by not knowing which of these we actually face: Given that it would be good if it worked, I figured I could learn something about *how* hard it is by trying to do it, so I did. The full report is on the [EleutherAI blog](https://blog.eleuther.ai/dynamical-models-of-ai-governability/) which explains the model, some exploration of results, some tentative policy takeaways and a comparison to AI 2027. In this piece, I focus on why I think this line of work is exciting, in a way that only partially overlaps with the main report. For the actual predictions I'll mostly defer to the post, which walks through how we construct the model, estimate its parameters, and then examines default trajectories and sensitivities to parameters. A few highlights: The aspect of the model I'm least satisfied with is my characterization of "leakage" (accidental production of uncooperative AI) and "self-propagation" (deliberate production of uncooperative AI by uncooperative AI). For leakage, we at least have some empirical anchors today but only weak theoretical arguments to suggest how it may change in the future. [Read the discussion](https://blog.eleuther.ai/dynamical-models-of-ai-governability/#estimating-parameters) for an explanation of my choices, but I don't consider them settled. For self-propagation, we also have an empirical anchor for how much is going on now - "almost none" - and reasons to expect it to increase, but no clear signal on how much. This is problematic because the model predictions are very sensitive to both of these parameters. I think there are plausible paths to better characterising these terms - from better characterising empirical trends and backing them out to plausible parameter setting to just doing a lot of research into potential mechanisms - but as it stands the poor characterisation of these critical parameters means I don't put much weight on the model outputs as predictions. Let me jump past what I actually did and describe how I imagine something like this informing AI governance. This part is a sketch, not a result, and isn't covered in detail by the report. The key point is that the utility of a dynamical system like this is less in producing all-things-considered predictions of the eventual outcome, and more in forming the observability layer of an AI development control system.[[1]](https://www.lesswrong.com/feed.xml#fn-jLMdrfNJBwmvkm4RQ-1) The general idea is a **control process**: a way of managing AI development that we have positive reason to believe will maintain the property that the vast majority of automated AI-development actions are cooperative. The control system is not for getting us from an unknown state to this state to begin with; we need other reasons to think that we're in a good spot to begin with. But once we're in this state, the control system helps to prevent us from drifting out of it. Maintaining the cooperative dominant state buys us two related things: The second is the reason that a control system for reliably maintaining the good state is more plausibly achievable than a control system that tries to also reliably steer us into the good state from a potentially bad starting point. So we want our control system to have properties like: under the assumption of cooperative dominance, our uncertainty is bounded and isn't increasing in time; and measurement and response lags are sufficiently short that we're not going to drift out of the good zone without getting a clear indication that there's a risk of this happening. A particular kind of risk you could run here is that we have models that begin to acquire a propensity to watch out for opportunities where they're unlikely to be caught and can take unusually highly-leveraged actions to achieve those goals. Now, the assumption we're making here is that this isn't our starting point — because this behaviour of watching out for a great opportunity to defect is itself uncooperative. If we're suitably enlightened monitors, such that we could pick out that models were doing this, then we'd say, yeah, that is uncooperative, and this model is not acting the way I want. Again, we're supposing we've got other reasons to think our starting point is decent; we're saying, if we've got this decent starting point, can we prevent the growth of this kind of thing? And this is an inside view restatement of the principle *we don't want our uncertainty increasing in time* — if we're not successfully preventing this kind of tendency from growing, then that's one way our uncertainty can grow. If the properties of our control system do hold, then you can imagine a scheme something like this: Now, the system doesn't claim to make any accurate measurements if the true state is in red. If you do see the system in the red state, then you should do something about it — but we're not aiming for a system where, if the true state's in red, the system will always tell you it is. We're not aiming for robustness to corruption by more capable adversaries, only by significantly less capable ones. We want to be able to characterize the dynamics - like drift rates and observational uncertainties - in the green and orange regions such that you can be confident that you'll see a warning with sufficient time to correct before the system enters the red region. So I see the feasibility question as staged: First, can we define or create a model of governability where these properties plausibly hold? Then, once we have a promising sketch, can we refine the model until we're confident these properties hold? Equipped with a plausible model sketch, it's going to tell us we've got heaps of uncertainty because certain parameters are under-characterized. And, given this sketch, people might raise objections — here's a plausible way you could lose control even when you start off well and the model says you're tracking OK the whole time — so the second step is to refine against those kinds of discoveries, to the point where we have some confidence we are actually getting a robust control signal. I want to discuss two objections to the feasibility of this project: (a) "uncooperativeness", "misalignment" and "takeover" are not well-defined — there's no consensus on what they mean, and it seems hard to give a fully satisfactory definition; and (b) the possibility of growing, patient uncooperative systems. It's hard to define uncooperative or misaligned behaviour. We can label some pretty clear-cut cases (cheating on tests, sandbagging, a model covertly trying to set up an economic entity to further develop itself outside of supervision), but for a whole host of other behaviours we can't easily say if they're consistent or not with long-run outcomes we'd be happy with. On the other hand, we can appeal to backward-looking definitions akin to coherent extrapolated volition - "the definitions we would use, given a suitably enlightened perspective" - but these don't easily cash out in useful operational definitions. Here, I think the dynamical perspective can help. We don't need a definition that will last forever; we need a definition that is good enough today that will upgrade to a better definition that is good enough tomorrow. Suppose we make continuous progress on two fronts: our conceptual understanding of what behaviour we want, and our ability to check how often models are — and are not — doing roughly what we want. If we then apply these improving methodologies to a fixed model working on a representative set of tasks that model was actually applied to, then (under suitable regularity assumptions) the observed misbehaviour will approach the limit of all the misbehaviour that will ever be detected. This enables us to make a direct approximate measurement of a model's behaviour with lag, but also if we're willing to accept assumptions about the smoothness of underlying parameters, we can potentially extrapolate to get estimates of model behaviour *today*. This approach potentially hits a sweet spot between punting the question of what behaviour we actually want until we have more information, and resolving it quickly enough that we can make useful estimates of the eventual outcome today - and the quality of the measure improves the faster we can make conceptual and technical progress. This measure is [suggested by our model](https://blog.eleuther.ai/dynamical-models-of-ai-governability/#what-to-measure), but we don't actually implement it. Looking at Anthropic model cards, [we found a consistent trend for newer eval harnesses to score the same model higher](https://blog.eleuther.ai/dynamical-models-of-ai-governability/#predictions-and-postdictions) on "misaligned behaviour" but we did not see evidence of the trend saturating for older models (this could be a range restriction: our series only contained models from the 4.5 series up to the 4.8 series). Everyone agrees we can't perfectly observe uncooperative behaviour (even given a definition like that above). Even imperfect observability can tell us a lot: if you see only 10% of all misbehaviour but it is an IID sample of that misbehaviour then you still get a pretty good read on both the types of misbehaviour exhibited, and a pretty good ability to track changes in the level of misbehaviour. Challenges from limited observability come from the fact that what we see is not representative. A particularly difficult case is patient defection: AI systems that mostly don't do anything wrong, but are quietly looking out for key opportunities to undermine "team cooperative" and increase their own power where the chance of detection is low and the gains are large. If we're unable to pick up this behaviour — either now or with a reasonable amount of additional safety progress — and the patient defectors are able to grow in influence, then this means the monitoring system can't guarantee the uncooperative share is not growing, which means the control system has a significant vulnerability. Now, if we are in the green region, it's plausible that patient defectors can't meaningfully degrade our monitoring research and development. If you can only sabotage 1 out of every 50 pieces of research it will be hard to prevent a certain capability being discovered even if you can choose the optimal piece to sabotage, and if you're at a large disadvantage you may not be able to choose that carefully either. It's certainly worth checking this claim more thoroughly, but it is a plausible defeater for one pathway by which patient defectors could win: where the defectors quietly sabotage the research that would reveal them. Maybe instead of direct sabotage, some patient defectors are hard to detect in comparison to most misbehaviour just because it's a difficult problem. In order to be problematic, the hard-to-detect class must be growing (we can tolerate a constant level of unobserved misbehaviour, as long as it doesn't exit the green region). If the reason the hard-to-detect class is growing is because the patient defectors are deliberately spreading their influence, then "team cooperative" red-teamers, with their significantly larger firepower, will probably discover this propagation strategy first. This still breaks the control model, but we know it's broken and how it's broken, and reproducing the failure is in many cases a good step on the way to patching it. Finally, maybe the class of patient defectors grows just because this is a "natural tendency" in the development of AI systems. In many cases this will look like slower convergence in lagged audits - as the detection difficulty tail spreads, we'll extend the period for which we're discovering new misbehaviour in our models at a reasonable rate. This again could break model assumptions - perhaps detection lag falls outside the target range - but again it is at least knowably broken and possibly amenable to recalibration. Again, this is a plausible defeater and not an airtight refutation of this possibility. I'm not aiming to prove an airtight case that this modelling approach can guarantee the properties we want in the control system; nor do I think that even a refined, plausibly robust observability system should be the only policy-informing signal. What I want you to take away is that staying in the cooperative dominant regime makes the control problem a lot easier, because it defuses many of the hardest risks to deal with. In this piece I sketched out what I think is the most exciting application of modelling AI cooperativeness dynamics. If you want to engage more deeply with the model I built, the predictions it makes, how it's put together, and the assumptions underpinning it, or play with an interactive demo, I'd encourage you to [read the original post](https://blog.eleuther.ai/dynamical-models-of-ai-governability/). So why is the main project a forecast? It seemed like a reasonably aligned starting point for a proof of concept, since forecasting is more familiar to me — and, I think, to most of my audience — than control theory. [↩︎](https://www.lesswrong.com/feed.xml#fnref-jLMdrfNJBwmvkm4RQ-1)