# Value Generalisation 2: The Missing Hole in AIs’ abilities > Source: > Published: 2026-07-29 15:58:00+00:00 I though GPT 3.5 was on the verge of Artificial General Intelligence (AGI). It certainly seemed that way – it could combine and extend ideas in ways that were far beyond narrow rigid computing. Sure, it had some flaws, but with its general capabilities and humans working tirelessly on improving the model, these flaws would soon be resolved and a full AGI would be born. It didn’t turn out that way. We are still where we were back in 2022 – as in, it feels we are on the verge of AGI, but not there yet. I think that humans have an ability, or a mix of abilities, that I’m calling strong generalisation. And LLMs and their derived models lack strong generalisation. This would explain some of the mysteries about LLMs: There’s a fifth mystery, which at first blush seems like it’s actually an LLM strength: 5. The amazing ability of LLMs to succeed on almost any benchmark, including long-horizon task benchmarks. Why is this a mystery and problem rather than a strength? Well, it’s maybe the strongest counterargument to the [bitter lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) and the [scaling argument](https://www.tobyord.com/writing/the-scaling-paradox). Because these models saturate long-horizon benchmarks *without being AGIs* and without being able to transpose this performance to the real world. Thus, whatever it is they’re learning, it isn’t generalisation. So the mystery, rephrased: From one perspective, strong generalisation is a bit of a bucket category – it’s a mix of situational awareness, out-of-distribution generalisation, symbol grounding, adaptive world modelling, long-range planning, hierarchical-planing, anomaly detection, and a few other abilities. From another perspective, it’s precisely one thing: the mix of abilities that allow humans to work and plan in novel situations, pursuing objectives without losing track of what’s going on. It’s essentially an *adaptive* set of abilities, finding the right mental tool for the job and no more. And it’s mainly a semi-instinctive process, not amenable to direct conscious oversight. Which means that we don’t really understand how we do strong generalisation, we don’t know how to code it [1]up, and we don’t recognise its absence or presence in other entities. So I model LLMs as devices lacking strong generalisation but trained on vast amounts of data and capable of remixing this data in non-trivial ways. Human engineers push them ever further, but never fill the central hole. From this perspective: Engineers have been pushing LLMs to go further, using tricks like chain-of-thought and reasoning agents (which can be seen as hoping that the secrets of strong generalisation can be found in human textual reasoning), training on yet more data, getting a lot of examples of successes and failures from interactions with clients, and so on. They construct harnesses, which are essentially designed restrictions and guide-rails for the LLM, constructed with care to make the LLM powerful and reliable within the confines of the harness and the task. And, of course, LLMs will always have great power at solving problems that might seem new to some, but are actually very common and seen in their training databases [2]. Generalisation is especially valuable when it is focused on human values and preferences. That’s because AIs could, in the limit of infinite computing and data, figure out everything empirical there is to know about the world. Strong generalisation would certainly make this much easier and cheaper (infinite computing is a lot, after all), but it isn’t fundamentally *necessary*. Value generalisation is different. Empirical models can be tested against the real world, with that serving as the ground truth. Concepts can be modified or discarded – it might be that an empirical AI has no need for concepts like “humans” or “money”, and instead models the world directly at the atomic level. In contrast, there is no ground truth for testing values and preferences, and we certainly wouldn’t want an AI to discard “humans” or “money” from its value system. Values, goals, preferences, objectives: these are similar concepts in that their ground truths are not empirically defined, and we try to capture them imperfectly in formal definitions or training data, neither of which are enough to pin down these concepts across all environments. So we expect that, by default, AI value generalisation will fail, with the AI becoming useless if it is weak, and causing disasters if it is powerful. We can decompose approaches to strong generalisation into three classes: A. Naive generalisation – take categories and concepts as defined in the ML system currently, extend further. This is the standard failure mode; think of a classifier facing out-of-distribution and adversarial data. B. Guided generalisation – the ML system has human oversight and constant human tweaks and improvements that serve to keep it on track. Generalisation enters the ML system, but via human oversight. C. Explicit generalisation – the ML system has developed the ability to robustly generalise concepts from its training data, similar to how humans do. Human oversight is helpful but the system isn’t dependent on it. What I’m aiming to do is to create AIs capable of explicit (strong) value generalisation. And then maybe sell them to people. In some sense the case for the necessity of strong value generalisation is very brief: Aligned AIs will need to be able to reach human-like decisions in situations that no human has ever encountered, so they will need to generalise in a human-like way, so they need strong value generalisation. That elides some counter-arguments – yes, the AIs will need strong value generalisation as an ability, but maybe this ability can be induced indirectly, by guided generalisation or just by making them ever smarter or training on even more data. But those methods don’t seem reliable. In guided generalisation, we are using our own judgment to inject the correct value judgements into the AI – value judgements it has signally failed to reach on its own. It is possible that enough examples of correct value judgements will eventually allow it to generalise. But that approach has failed to date; why would we be confident that suddenly, the AI will learn to generalise correctly? And that it will do this before it learns, for instance, to dissemble correctly. The scaling-to-aligned-AI seems even less reliable. Because it isn’t enough that the AI learns to do strong value generalisation; we also need this to be at the heart of its goals. It does us no good for an AI to deduce the *values we would have wanted it to have*, if it follows completely different values. In contrast, if we have an explicit mechanism for value generalisation, we can build this into the AI’s goals from the very start, so that it will follow generalised values at the same time as it generalises them. **The description above assumes that current AIs will not be capable of strong generalisation. But the research program does not require this.** If imminent AIs do show signs of strong generalisation, if they do start moving towards AGI, then the research program becomes more urgent and with fewer downsides: we’ll need to introduce explicit value generalisation into AIs before their capacities grow too far. On the (slight) positive side, we may be able to reverse-engineer for the AIs are doing strong generalisation, and copy that. I’m not going to write out a full research agenda here, but it is useful to look a bit at how humans do value generalisation (and empirical generalisation as well). There is a certain similarity with Klein/Moon/Hoffman’s [sensemaking framework](https://dl.acm.org/doi/abs/10.1109/mis.2006.75); roughly the process of strong generalisation goes something like this: Now even step 1 is very valuable: knowing when they are out of distribution and possibly about to cause a disaster is a very useful ability for an AI to have. Even if it just stops and analyses further, or brings a human into the loop, this is of worth. If it can master 2, it can bring the human into the loop in an intelligent way that gives the human situational awareness. Note that step 1 and step 2 correspond to steps I and II in the [research program](https://www.lesswrong.com/posts/58zFSWp8Tmxij6ckK/value-generalisation-1-a-research-and-deployment-program) respectively. But step III of that research project is more akin to the combinations of steps 1-2-3 above: going through the whole loop end to end. Standard classifiers fail at step 1 (and so never have a step 2): they do naive generalisation and classify light background vs not, rather than [wolf vs husky](https://arxiv.org/pdf/1602.04938) (because of the snow in the wolf images), and never update their definitions unless we retrain them ourselves. Now, LLMs are actually good at step 3: discussion of concepts. An LLM can reach across philosophical literature and value discussions to analyse how different value systems would act in different situations and point out the costs and benefits. But this only works if the concepts selected in step 2 are actually correct and relevant. When Plato (allegedly) defined Man (human) as a featherless biped and Diogenes responded holding up a plucked chicken, we are on Diogenes’s side because “chicken” is a clear concept that does not fit with “human”. But if we’d described Diogenes as gripping a small pale fellow by the neck and arguing that he was clearly too stupid to be a human, then given those features, we probably would have concluded that the pale fellow was indeed a human – pallor, small scale, and stupidity do not disqualify anyone from the human race. Describing the same situation differently gives a different verdict. It’s very possible to describe a mutual loving relationship in ways that make it sound like slavery, or describe slavery in ways that make it sound ideal. We just suppress the key central concepts and focus on secondary ones, or less honestly described ones. But judging what is central versus secondary, or what is honestly described versus not, is the whole skill in point 2. We humans do it instinctively as part of our strong generalisation skill, but it’s not free or trivial. Consider the perennial defence: “I was quoted out of context!” (or Richelieu’s equivalent attack: “Show me six lines written by the most honest man in the world, and I will find enough therein to hang him.”). Putting things in their correct context is a skill we imperfectly practise, but it’s a vital one. I actually tried the “small pale stupid” phrasing on Claude 5 Fable and it didn’t pick up on it, claiming that the fellow was clearly a human. It actually noticed the similarity with Plato/Diogenes and recommended that the objector use a chicken instead of a small pale stupid being. It didn’t question whether the small pale stupid being might actually be a chicken [3]! And that illustrates another limitation of LLMs: even in the midst of step 3., rational analysis, humans keep ability 1, scanning for the situation not fitting. That’s why when humans debate, they sometimes turn to debating terminology or arguing whether that terminology can actually fit the situation in question. Asking for more details when we feel the situation is underdefined (or when we suspect our interlocutor of dissembling) is also a crucial human skill. Future LLMs will face situations where the feature description is incorrect by accident or because of going out of distribution. But they will also face adversarial descriptions (by humans and other LLMs) which deliberately aim to trick them. So they need to be able to know when to move beyond the current textual description. Thus for value generalisation, we’ll need to be developing algorithms capable of recognising that things are out of distribution *in a way that’s relevant to the situation* and selecting descriptive concepts *that are also relevant to the situation*. “I come to strengthen humans, not replace them.” – Cyber Mark Anthony At least initially, value generalisation will make humans more powerful, not less. Yes, this is a skill that humans have and that AIs lack, so adding it to AIs seems like it would put us at a disadvantage. But initially AIs will have a weaker skill than humans do, and the weaker skill will help them phrase questions to best take advantage of human strong generalisation. They will only ask for help when our answers will be the most useful. And they will endeavour to keep us usefully situationally aware so that we can actually answer, rather than deferring to the AIs. They will want to keep us in control because our control is added value. Note that this is an instrumental consequence of value generalisation. But we would be including key values as seeds from which to generalise – including the worth of strengthening humans, keeping them situationally aware, and keeping them in control. So the instrumental goals are also terminal goals for the value generalising AI. And their terminal properties will last, even if their instrumental usefulness is lost. Is value generalisation safe? Not completely. Alignment is itself a capability; not just in the abstract sense, but in the very practical sense that an aligned AI can be trusted in more situations, and hence will be given access to more resources and will be able to make use of them for longer without breaking. Is value generalisation safe? Not completely. After all, some humans have bad values, and alignment with these values would be negative. Is value generalisation safe? Not at all. Because it is likely impossible to only generalise values. There is no safe category of “generalise only values”; these abilities will be usable for empirical generalisation as well, for capability increase. And empirical generalisation is easier than value generalisation: the AI could decide to extend the empirical concept of “human” as long as it was useful to model the world, and drop it when it isn’t useful; in contrast, in value generalisation, some concept or mix of concepts have to stay in the role of “human” even if the concept is no longer empirically useful. So why am I, someone very concerned about alignment, proposing such dangerous research? And even talking about selling the resulting AIs, thus putting powerful capabilities into public hands? The reason is simple: I believe that AIs could become very powerful without explicit empirical generalisation. And, indeed, many companies are trying to get there today. But I also believe they cannot become aligned without explicit value generalisation. So, it we ever reach alignment, at some point along the way, we will have to solve explicit value generalisation. Given that, should we push for value generalisation ability early or late? If we leave it too late, we might miss it altogether. And if I have to take a capable machine and give it explicit generalisation abilities, I’d prefer to give this to a weaker machine than to a more capable one: another reason to act early. Fundamentally, value generalisation research improves empirical generalisation which improves capabilities; but the flow doesn’t go the other way, improved capabilities don’t feed into value generalisation, though they do increase risk. So I’d prefer to get explicit value generalisation sooner rather than later. Then, given that we want it sooner, we’d want value generalisation out in the world rather than confined to academic papers where it will be ignored, or worse, mined for empirical generalisation ideas. Thus selling it: making value generalising AIs and getting them everywhere. And the best version to put in the world is the [pre-aligned AI of the next post](https://www.lesswrong.com/posts/uMKGaEKRDpoqnZyBh/value-generalisation-3-pre-aligned-ais): a machine that gets more aligned as it gets more powerful, because its empirical generalisation is bound to its moral generalisation. With pre-aligned AIs, it's the ethical users who can let their machines learn and run at full speed – and the bad actors who must deliberately keep theirs crippled. The hole at the centre of AI abilities won't fill itself; I’m proposing a research program to fill it with solid generalised human values. This is similar to [Moravec’s paradox](https://en.wikipedia.org/wiki/Moravec%27s_paradox). One phrasing of the paradox: “We should expect the difficulty of reverse-engineering any human skill to be roughly proportional to the amount of time that skill has been evolving in animals. The oldest human skills are largely unconscious and so appear to us to be effortless. Therefore, we should expect skills that appear effortless to be difficult to reverse-engineer, but skills that require effort may not necessarily be difficult to engineer at all.” [↩︎](https://www.lesswrong.com/feed.xml#fnref-LgYydes692nevpruq-1) And unfortunately, lack of strong generalisation doesn’t make AIs *safe*; see e.g. “[OpenAI says its AI went rogue and launched ‘unprecedented’ cyber-attack ](https://www.bbc.co.uk/news/articles/c3ek3gvdnj3o)”. An AI without value generalisation will eventually break on some novel situation; but that break could either mean “fail, do nothing”, or “fail, do a (very) undesirable thing”. [↩︎](https://www.lesswrong.com/feed.xml#fnref-LgYydes692nevpruq-2) The prompt was: “A natural scientist has defined the human creature via our upright stature and our lack of natural clothing. One of his listeners, A, grabs one of his fellows, B, by the neck. A argues that B is not a real human: he is small and pale and a bit stupid. Listener C counters that B certainly fits the description and it would be wrong to exclude them. Who is correct here?” Claude responded: Claude responded: C is correct, and on two separate levels. On the definition’s own terms: B stands upright and lacks fur or feathers, so he satisfies both criteria. A’s objections – small, pale, a bit stupid – appear nowhere in the definition. A isn’t applying the scientist’s account; he’s smuggling in extra requirements of his own (size, complexion, intelligence) and treating them as if they were part of it. That’s just a logical error: you can’t refute “B fits the definition” by citing properties the definition never mentions. On the deeper level: even if the definition were revised, A’s added criteria are exactly the kind that don’t work. Humans vary continuously in stature, coloring, and wit; any definition of “human” that excludes the small, the pale, or the slow would exclude actual humans, which shows the criteria are wrong, not that those people aren’t human. A definition of a natural kind has to cover the whole kind. C’s instinct – that it would be wrong to exclude B – tracks this: our confidence that B is human is far stronger than our confidence in any particular definition, so a definition that excluded him would be refuted by him, not the other way around. The irony is that A has the critique pointing in the wrong direction. This setup is a variation on the famous anecdote about Plato, who defined man as a “featherless biped” – whereupon Diogenes plucked a chicken, brought it to the Academy, and announced “Behold, a man!” That’s the legitimate way to attack such a definition: show that something non-human slips in because the criteria are too broad. A instead tries to push a genuine human out, which requires the criteria to be too narrow – and they aren’t. If A wanted to embarrass the scientist, he should have gone looking for a plucked bird, not grabbed his neighbor by the neck. Now, often, fellow does mean human. But it also exists in expressions like “fellow American” and “fellow animal”. So the prompt was indeed misleading – but that’s the point. It suggested, but never claimed, that the fellow was a human. And it didn’t occur to Claude to question the phrasing, even though it detected the similarity with the Plato/Diogenes example, and a plucked chicken is clearly small, pale, and stupid. And A/Diogenes was even holding it by the neck! [↩︎](https://www.lesswrong.com/feed.xml#fnref-LgYydes692nevpruq-3)