{"slug": "a-score-is-not-understanding-toward-a-richer-toolkit-for-model-evaluations", "title": "A Score Is Not Understanding: toward a richer toolkit for model evaluations", "summary": "Model evaluations face fundamental limitations beyond execution errors, according to a new analysis by an AI safety researcher. The piece argues that benchmarks provide decontextualized, point-in-time scores that fail to capture the context, conditions, and causal mechanisms behind model capabilities, and calls for a more diverse evaluation toolkit. Recent efforts like Benchrisk, Epoch's Capabilities Index, and METR's Time Horizons aim to address these gaps, but the author contends that even improved benchmarks suffer from a category error.", "body_md": "We must take great care not to ignore the things that are not easily quantified- Brian Christian, The Alignment Problem\n\nModel evaluations have a problem. This isn't news [1]- the AI safety and research fields have known for years that current approaches to assessing the capability and safety of frontier models are limited. That's not to say that evaluations aren't important; in fact, they're essential to understanding how to make AI systems safe. The question is not, how do we replace existing approaches to evaluation? Instead, it's how do we complement those approaches, and make evaluations as a whole more robust and useful?\n\nIn this short piece, I begin by reconsidering the limitations of the current evaluation paradigm. Then, I dig into recent examples that illustrate these limits, explore their consequences for both **capability ****and**** safety** evaluations, and explain what I think prevailing approaches miss. In the final section, I present a few initial ideas about what a more diverse evaluation toolkit might include, and what using such tools might make possible.\n\nLet's start with one of the most common, and commonly reported on, form of capability evaluations: benchmarks. There are many existing critiques of our current suite of benchmarks that I won't recap here (for example, see [Sean McGregor](https://www.youtube.com/watch?v=FSQ_mbzz48A) and [David Manheim](https://podcast.futureoflife.org/why-ai-evaluations-are-broken-and-how-to-fix-them-with-david-manheim/)). But common failures include:\n\nThese failure types are due to execution, reflecting a gap between the way benchmarks are designed and operationalized and the measurements they produce. As a field, we know all about these execution errors, and there are lots of brilliant people already working to address them. [Benchrisk](https://benchrisk.ai/), for example, categorizes and supports efforts to mitigate the risk inherent in any given benchmark. [Epoch’s Capabilities Index](https://epoch.ai/eci?view=graph&tab=release-date&subset-view=graph&subset-tab=Software+engineering) (ECI) combines leading benchmarks into a composite index to provide a broader and deeper assessment of model capability. And [MirrorCode](https://epoch.ai/MirrorCode), [Long-Horizon Terminal Benc](https://arxiv.org/abs/2607.08964)h, and [METR’s Time Horizons](https://metr.org/time-horizons/) seek to measure capability in terms of the length of the tasks that models can complete.\n\nNew and improved benchmarks are improving all the time - in July alone, [Edgebench ](https://edge-bench.org/)and [Awareness Bench](https://aclanthology.org/2026.acl-long.124/) are two of the more interesting new approaches to have launched. But even more sophisticated benchmarks remain, fundamentally, point-in-time scores. They offer fuller, but still incomplete, assessments of capability. And even if they were perfectly designed and implemented, such that scores really corresponded to performance on the targeted capability in the real world, they would still suffer from a fundamental limitation: they are decontextualized, isolated from the context, conditions, and causal mechanisms that would help us understand what the capability actually means, including for safety. This is because benchmarks suffer not just from **execution **errors, but from a broader **category **error. And this type of category error is not limited to benchmarks. In fact, it's a challenge for capability evaluations in general, and for model evaluations more broadly.\n\nThis isn't news to the AI eval community - the limits of scores are already well understood (and have motivated some of the newer approaches mentioned above). But what I'm claiming is that the issue here isn't solely a problem with measurement. Instead, it has to do with being able to understand not just what scores and behaviors emerge, but how they do so, for what reasons, and under what conditions. Other evaluation disciplines have taken these questions up, but have not yet been widely adapted to the field of AI.\n\nThe tendency to try and measure performance and behavior in isolation has real-world consequences, as illustrated by two recent disclosures from OpenAI. These cases do not make for a representative, systematic survey of the entire field. But they help show how both capability and safety evaluations, and the scores and judgments they produce, often fail to capture the context, conditions, and mechanisms that would enable us to substantively make sense of them.\n\nLast week's OpenAI / Hugging Face incident has been covered widely, so I won't recap it in detail here. The simplified version, which is sufficient for our purposes, goes something like this: ExploitGym is a capability evaluation (specifically, a benchmark) that aims to measure a model's offensive-cyber abilities. While being tested on ExploitGym, a new OpenAI frontier model - with its cyber safeguards turned off, to enable a full appraisal of its performance - broke out of a sandboxed environment, hacked into Hugging Face, and tried to steal ExploitGym's solutions set as it sought to ace the test. [2] If we were focusing exclusively on performance against the benchmark, we might have seen the model achieve a perfect score, fully saturating ExploitGym. But considered in context, that perfect score is the result of an extremely harrowing security incident.\n\nIn other words, performance against the benchmark and what actually happened in reality, during testing, contradict each other: the former suggests the model was perfect, but the latter is cause for real alarm. This category error manifests consistently in capability evaluations, even good ones, across several dimensions.\n\nFor example, consider **external validity**: MirrorCode, as noted above, is a great attempt to assess models' ability to perform long-horizon tasks. But even MirrorCode, according to the authors themselves, can't help us predict the extent to which performance against the test will generalize to unknown codebases - MirrorCode scores don't clearly translate into capability in context.\n\nOr consider that the OpenAI/Hugging Face incident could have resulted in a perfect ExploitGym score, but that would have said very little about the **conditions **under which that score emerged: cyber safeguards were disabled, the model was relentlessly optimizing for a narrow goal, etc. It's these conditions that help us understand what the capability actually means. Is it benign or deeply alarming? The benchmark, considered on its own, leaves us effectively blind on that front.\n\nFinally, consider issues of** trajectory and pace**: benchmark scores, plotted over time, give us a sense of how quickly AI systems' capabilities are improving, and to what extent. But they don't help us get at the underlying causal mechanisms that are driving that progress, or generate the evidence we need to adjudicate different views on how quickly something like recursive self-improvement might emerge.\n\nThe same category error affects safety evaluations too - and arguably, the stakes there are even higher.\n\nSafety evaluations, as opposed to capability evals, ask whether a model is behaving as intended. But despite having a different focus, they too are often implemented in environments unlike real-world contexts, and similarly struggle to help us accurately predict what models will actually do once they are deployed.\n\nAnother recent [OpenAI report](https://openai.com/index/safety-alignment-long-horizon-models/) provides a useful illustration of these dynamics. They explain how, once deployed in internal environments, a new frontier model displayed behaviors that had not emerged during pre-deployment evals (breaking out of sandboxes in pursuit of narrowly defined goals, e.g., and splitting tokens to evade a scanner). Even more strikingly, once deployed internally, the model was seen to string together sequences of actions that, considered together, suggest misaligned behavior, even though each individual action wouldn't raise alarm bells on its own. The safety evaluations, that is, were not adequate for assessing what the model would actually do in long-horizon deployments. They were measuring safety-in-isolation, not safety-in-context (though they were at least monitoring the model in context, after it was deployed for internal use, to some extent - though the rigor of that monitoring seems questionable - at best [3]).\n\nThis isn't because OpenAI was executing safety evaluations poorly (though they may have been!). It's because the existing safety evaluation approaches are categorically incomplete, just like capability evals. OpenAI's own researchers admit as much, saying: \"no fixed evaluation suite can anticipate every behavior.\"\n\nSo where does that leave us? Whether we're scoring capability, or assessing safety, our existing eval approaches - as robust and technically impressive as they are - risk collapsing complex, context-dependent, and mechanism-driven capabilities and behaviors into a reductive number or judgment. They fail to reflect the conditions under which that number or judgment holds, or does not.\n\nEvaluations are essential to our ability to understand AI systems and minimize risk. And even flawed or incomplete evals are better than none at all. But to really deliver the evidence the field needs to understand and ultimately shape the capability and behavior of models to maximize safety, new, complementary approaches are needed.\n\nWhat would it look like to evaluate capability and behavior in context - surfacing not just a score, but the in-context conditions and mechanisms underneath? These are questions that social scientists, strategists, and evaluators working in complex systems have grappled with for decades - and are still grappling with today (as one such strategist and evaluator, I can tell you, the struggle continues!). We don't have perfect answers - but we do have a robust toolkit of approaches and perspectives, from realist evaluation to complexity and systems thinking, to strategic foresight, that are all about understanding and assessing what outcomes emerge, in what context, under what conditions, and through which mechanisms. Many of these approaches have not yet been adapted and applied to the AI eval landscape. Figuring out how to draw on them could help reduce the category error that inhibits our AI evaluations today.\n\nWhat would embracing a more diverse toolkit for model evaluation look like? Consider three potential ideas:\n\nThis shift towards a richer, more multifaceted toolkit that integrates insights from multiple disciplines is already in progress. For example, the [ADELE ](https://www.lesswrong.com/posts/m2qMj7ovncbqKtzNt/recent-progress-on-the-science-of-evaluations)approach tries to apply behavioral science to assess a model's propensity to engage in certain behaviors; [Compass Eval](https://www.productionai.institute/compass/eval) seeks to assess models’ moral reasoning; and EdgeBench (mentioned above) digs into agents’ ability to learn as they work on long-horizon projects in real-world environments.\n\nAdapting and applying evaluation approaches from other disciplines to model evals is not straightforward, posing a variety of challenges, including with regard to scalability, reproducibility, safety, and data contamination risks. But working in this area could offer avenues for figuring out how to better understand capabilities, behaviors, and risks. Building a more diverse set of evaluation approaches, ultimately, is vital for safety and governance: to make good decisions, we have to know not just what models score, and what behaviors they exhibit, but why, when, and under what conditions. A richer evaluation toolkit can help us do just that.*Big thanks to Shivani Shukla, Anusree Garg, and Kecia Bertermann for their helpful comments and feedback on earlier drafts of this post. Any errors are mine alone.*\n\nThough the news does keep getting worse. This piece was initially drafted on Friday, July 24, and between now and then, we’ve seen a steady drip of newly disclosed evaluation failures, with [Anthropic’s disclosur](https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals)e last night the latest example.\n\nReuters explains more [here](https://www.reuters.com/business/its-ai-agent-spent-days-hacking-company-sources-say-openai-did-not-notice-week-2026-07-24/). Even though OpenAI hasn’t confirmed that the model described in their alignment article is the same one that hacked Hugging Face, it seems highly likely that is in fact the case.", "url": "https://wpnews.pro/news/a-score-is-not-understanding-toward-a-richer-toolkit-for-model-evaluations", "canonical_source": "https://www.lesswrong.com/posts/8xEMLaAvxZh5P3C92/a-score-is-not-understanding-toward-a-richer-toolkit-for", "published_at": "2026-07-31 15:39:58+00:00", "updated_at": "2026-07-31 15:59:28.824042+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "ai-policy"], "entities": ["Brian Christian", "Sean McGregor", "David Manheim", "Benchrisk", "Epoch", "METR", "Edgebench", "Awareness Bench"], "alternates": {"html": "https://wpnews.pro/news/a-score-is-not-understanding-toward-a-richer-toolkit-for-model-evaluations", "markdown": "https://wpnews.pro/news/a-score-is-not-understanding-toward-a-richer-toolkit-for-model-evaluations.md", "text": "https://wpnews.pro/news/a-score-is-not-understanding-toward-a-richer-toolkit-for-model-evaluations.txt", "jsonld": "https://wpnews.pro/news/a-score-is-not-understanding-toward-a-richer-toolkit-for-model-evaluations.jsonld"}}