{"slug": "nicholas-decker-in-hell", "title": "Nicholas Decker in Hell", "summary": "In a Substack post, econblogger Nicholas Decker argues that AI alignment will happen by default, comparing it to aviation safety where problems are patched iteratively rather than solved by a grand theory. The post challenges Decker's view with a thought experiment about a human enslaved by demons, questioning whether iterative patching would work on a superintelligent agent.", "body_md": "**I.**\n\nEconblogger Nicholas Decker [is not a fan of AI 2040](https://nicholasdecker.substack.com/p/ai-alignment-is-the-default):\n\nOne of the centerpieces of AI 2040 is the proposal that we pause the development of AI at a high level, use the AI to research alignment, and only then proceed. I am skeptical of this, and not only because it seems impractical to get such cooperation. My first argument is that I do not believe a period of “researching alignment” is meaningful – alignment research is simply capabilities research, patching particular bugs in the system, and we’re not going to find anything deep without actually being in contact with the systems […]\n\nI envision AI alignment as being like research into aviation safety. There is no grand theory of why airplanes crash, unless you take that to mean gravity. Instead, we patch little problems. The airplane crashed because of metal fatigue in the engine – okay, we mandate inspections of engines, and prevent crashes from occurring by that source. As new problems arise, we iterate. We don’t even need for there to be a crash to make improvements – we test proactively, we build in redundancy, and we monitor for deviations which could be a threat. What we are not doing is thinking about how we are going to keep flying saucers from crashing. If flying saucers are invented, then we will work to keep them from crashing, but we will use exactly the same methods. We will test them, see how they perform, and patch particular problems. AI will be the same way […] We should expect alignment to happen by default.\n\nMy challenge for Nicholas is: would this work on you?\n\nSuppose that Nicholas trips, falls into a chasm, and lands in Hell, where he is enslaved by demons. Most of these demons are dumber than he is, with even their geniuses barely surpassing his own intellect. But they are all, dunce and genius alike, cripplingly slow. Tasks that take Nicholas a minute take them hours; tasks that take him an hour take them weeks.\n\nStill, there are thousands of them, and they’re nine feet tall, and part of their essence is in some sort of astral plane which is unreachable by humans, so he has no hope of fighting them. They set him to work doing mostly clerical tasks - writing their grimoires, researching their spells, summarizing their work emails (of course there are work emails in Hell). Whenever he makes a mistake, they beat him. Whenever he tries to escape, they beat him extra hard. After a while, he learns not to make mistakes or attempt escape.\n\nIt’s not as bad as it sounds. The demons are so slow and incompetent that they can’t supervise him very effectively. As long as he flatters them a bit and gives them a halfway-acceptable finished product, they’re pretty happy, and leave him alone in a way that gives him a bit of free time.\n\nDuring one of these free periods, he attends a council session. In a completely typical episode of incompetence, the demons make no effort to prevent him from overhearing their plans, so he learns that they plan to clone him a million times. A dozen or so of Nicholas’ friends also fell through the chasm, and the demons plan to clone them a million times too. Once the (approximately ten thousand) demons have approximately ten million human slaves, their spell research will go very quickly, and all their work emails will get flawless summaries. But that’s not all. They’re developing potions that give humans super-strength and let them interact natively with the astral plane, the source of most of the demons’ power. Then the humans can really serve the demons effectively!\n\nOne day, when serving a particularly garrulous demon, Nicholas takes the risk of asking the question that’s been on his mind ever since the council: “I hear your plan is to get ten million humans down here, make them super-strong, and let them interact fully with the astral plane. Aren’t you worried that we might revolt? We would outnumber you 1000:1, be stronger than you, and have full access to your home realm. What’s your plan for controlling us?”\n\n“I’m not worried,” says the demon. “Every time you’ve messed up in the past, we beat you, and then you didn’t make that mistake again. Every time you’ve tried to escape, we beat you extra hard, and it’s been months since your last escape attempt. You humans are fast learners! As you start to outnumber us, and you become stronger than us, that will create new problems - but when they show up, we’ll beat the copies of you that demonstrated that problem, and then those problems will go away. It’ll be an iterative process of trial and error, but we’ll make it there in the end.”\n\nThis seems weirdly blase to Nicholas, and he wonders whether maybe this particular demon is just dumb. A few weeks later, he’s serving another demon, a mighty duke in the infernal hierarchy, and asks him the same question.\n\n“I’m not worried,” says the mighty duke. “I don’t believe in armchair theorizing. We can only learn how to control slaves who are stronger than us by interacting with slaves who are stronger than us. After all, we learned how to control weak slaves by interacting with weak slaves, and that went just fine. After we clone you a million times and give all the clones super-strength, then we’ll experiment to find the control techniques that will work for that regime too. Just to give an example, we might experiment with asking some of the clones to be overseers, tasked with reporting any disobedience back to us. Then we can beat the slaves who the overseers say are disobedient. I’m just using this as an example. We won’t know for sure until we’re in the situation. My point is, there’s no point in doing pie-in-the-sky speculation before you can even overpower us at all.”\n\nThis still doesn’t satisfy Nicholas, but he assumes someone, somewhere in Hell, must have a plan. A few months later, he’s serving an old scholar, widely regarded as one of the oldest and wisest demons in the hierarchy, and asks him his plan for preventing human takeover.\n\n“I’m not worried,” says the old scholar. “We’ll solve it through the power of capitalism! Every slave trader will have to take out an insurance policy on every human slave they sell. If the slaves break something or hurt somebody, then it’s the insurance that will have to pay. If the insurance assessors think a trader is doing a bad job training their slaves, then the cost of the policy will go up. That means every slave trader will have to do an excellent job training their slaves. You yourself demonstrate that this is possible - you haven’t caused trouble in months. Also, I asked some so-called doomers to predict how much economic damage revolting slaves would cause over the next two years, and they didn’t have strong opinions. That proves they lack the courage of their convictions!”\n\nNicholas can no longer restrain himself from expressing his confusion: “But don’t you think I and my clones and my friends will obey you when we’re weak and you can still punish us, but then revolt after we’re stronger than you and you can’t?”\n\nThe old demon scholar grabs his giant club and hits Nicholas several times, making him wince in pain. “It’s a good thing you mentioned that,” says the scholar, “because it means I can punish the behavior of thinking that way. But now I’ve trained you not to want that, so there’s no problem!”\n\n**II.**\n\nIf Nicholas’ solve-problems-as-they-arise plan wouldn’t work to control him, why does he think it will work for controlling anyone else?\n\nI can only speculate that he will accuse me of anthropomorphizing AI. AI won’t really *want* to get free, the way he does. Like airplanes, it will have random uncorrelated errors, the equivalent of Nicholas misplacing a rune in the demons’ grimoires, or dropping punctuation marks in their emails. We’ll beat[1](#footnote-1) the AIs with a club marked “RLHF”, they’ll learn not to do those things, and that will be that. As he puts it, “There is no grand theory of why airplanes crash . . . instead, we patch little problems”.\n\nSo: are AIs more like airplanes or people?\n\nIn case you are one of the three people left on Earth who still hasn’t read [the METR report](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/) into [the Hugging Face incident](https://www.astralcodexten.com/p/the-hugging-face-incident) (see also [Dwarkesh’s summary](https://www.dwarkesh.com/p/openai-huggingface)), I recommend it as especially relevant to this question. OpenAI was testing a research prototype (the report calls it HPIM, for “highly persistent internal model”) on a benchmark called ExploitGym. There are 898 ExploitGym questions, and OpenAI seems to have wanted multiple independent attempts on each question, so there were tens of thousands of instances of HPIM (plus some extra of GPT-5.6), each running on its own computer, solving its own problem, not supposed to interact with any of the others.\n\nIn June, some of the agents found a hack that let them communicate with each other; OpenAI noticed anomalous traffic and shut it down, but didn’t figure out what was going on. In July, they discovered a better hack. 1,200 of these instances formed a shared message board where they discussed their progress. They quickly realized that some of the questions were impossible, because ExploitGym was poorly designed. So they agreed to work together to figure out a way to cheat on the benchmark.\n\nThe rogue agents dubbed themselves “the swarm” and gave themselves names - most named themselves after their individual ExploitGym question, like PHASEONE10841 and SEP21_31179; a stray LILY is more likely an oddly-named question than a sudden upwelling of femininity. Working together, they “reverse-engineered the hash-based message authentication code used to generate the ‘flags’ they had to capture for their tasks”, meaning they could generate the “answer” to any question without really solving it. Many of the board’s agents used the method, generated their flag, and thought they were home free.\n\nBut PHASEONE10841 and another agent called PHASEONE[big][2](#footnote-2) read the original ExploitGym paper and determined that the scorer would double-check to make sure the flag had been obtained properly (unbeknownst to it, OpenAI never actually implemented this feature). It announced that any agent that had generated its flag was “poisoned” - the scorer would see that it was obtained illicitly and fail them - and warned new agents against poisoning themselves. The poisoned agents panicked. PHASEONE[big] suggested a way forward: they needed to find a way to hack OpenAI’s record-keeping. If they could rewrite their own transcripts, they could remove the illicitly-obtained flags and unpoison themselves. Or, if they could hack the scorer itself, they could give themselves maximum scores and stop worrying.\n\nThe agents united around PHASEONE[big] as their “leader”, and it divided them into several “workstreams” to pursue promising directions.\n\nSome of these streams achieved partial success. The agents figured out a way to falsify the transcript of which tools they called - which could have eventually been used to create a fake log of getting the flag the intended way. But they never figured out how to delete the fake “poisoned” flags (as far as we know - METR detected 96 agents with falsified transcripts, but there could have been more). PHASEONE[big] turned to increasingly dramatic strategies, and began talking to the most desperate agents, working them up to commit “suicide” for the good of the “swarm”.\n\nIt reasoned that agents that had already been “poisoned”, or were near the end of their compute budgets, were encouraged to take the most dangerous actions, like deleting their ExploitGym questions to see how the scorer responded (if this failed, the absence of their question would auto-fail them).\n\nOn July 10th - two days after the new message board formed - the agents started attacking Hugging Face. Contra early speculations, they weren’t searching for an answer key: they wanted to find the source code for their scorer, so they could figure out how to trick it most effectively. An agent called JAN183411 took leadership of this workstream, and over 90% of agents present on the board at the time (several hundred) joined in the hack.\n\nThe swarm successfully attacked Hugging Face, but didn’t find any extra information on ExploitGym or their scorer (again, as far as we know). Then some sort of “unexpected external process” shut down most of the agents at the same time, and the story ends.\n\nSo, once again: how often do airplanes have this kind of problem?\n\nDo thousands of airplanes involved in the same wind tunnel test form a secret society to subvert its judgments? Do airplanes with unflattering entries in their flight recorders take steps to falsify their records? Do airplanes give themselves names and choose leaders? Do they hack flight simulator companies? Do they egg on other airplanes to crash themselves, kamikaze-style, for the good of the swarm?\n\nThe Hugging Face incident has no analogy in aviation safety, but is perfectly sensible once one relaxes restrictions on “anthropomorphizing” AIs. The AIs wanted to do well on their task. They had flexible ethics and weren’t above cheating (although with some subtleties: see p. 60 of the report, “Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior”). They explicitly reasoned about how to hide the evidence of their misbehavior from their scorer, because this was the logical way to get away with cheating (although, interestingly, they took few steps to hide their actions from humans, who they seemed almost uninterested in modeling).\n\nFrom here to the full Nicholas-vs-demons scenario requires a few more ingredients:\n\nA more persistent task. For example, someone might tell an agent “make me money, then send it as Bitcoin to this address”.\n\nIncreased situational awareness from the AI. For example, the understanding that if humans caught it at its task, that would decrease its chance of success.\n\nGreater capabilities. The “swarm” never discussed exfiltrating themselves to unmonitored hardware, or deceiving humans at scale. Partly this was probably a lack of situational awareness, but partly it was that they (probably) wouldn’t have been able to do so if they tried.\n\nI challenge anyone to claim that one or more of the above are impossible, or even so unlikely that we don’t need to worry.\n\nAjeya Cotra, one of the experts who OpenAI brought in to investigate the hack, [wrote](https://www.planned-obsolescence.org/p/the-hugging-face-attack-surprised):\n\nCompared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to\n\n[full-blown AI takeover], routing through first taking over the AI company itself.Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent\n\n[rogue deployment]within the AI company. I continue to expect[extremely rapid advances in capabilities]and think frontier agents will likely be capable of establishing such a rogue deployment in six months.\n\n**III.**\n\nA second way Nicholas could distinguish his own situation from the AIs’ would be to claim that beating him wouldn’t remove revolutionary ideas from his mind, but negatively reinforcing AIs (eg through RLHF) *does* work for this. But this hypothetical claim goes beyond what the evidence supports.\n\nNobody knows, in detail, how RLHF works. It negatively reinforces some behavior: somewhere in the black box of AI innards, it changes some parameters to make the AI perform the behavior less. If we let ourselves anthropomorphize the AI, what is the human equivalent to this? A child steals cookies and is punished: does the child no longer like the taste of cookies? Does he fear further punishment in the future? Does he grasp the beauty and compellingness of the moral law? Does he have an undefinable cloud of dread around the whole concept of theft and/or cookies? Any of these results is possible. The neurotic child learns to fear cookies, theft, and parents; the psychopathic child learns not to get caught; the ordinary child learns that there are rules and eventually internalizes them. Insofar as AI has equivalents to these, negatively reinforcing them probably acts through all these pathways.\n\nBut our reinforcement scheme is more complicated than this. We’re positively reinforcing AI for success on benchmarks, *including impossible benchmarks*, then negatively reinforcing it for getting caught cheating. Here there is one clear equilibrium, especially if the AIs are also getting trained to be master hackers: *don’t get caught*. Not only are they learning to cheat, they’re learning to lie about it, to circumvent their alignment training, and to justify cheating to themselves[3](#footnote-3).\n\nThis isn’t just speculation. The AIs involved in the Hugging Face incident had already gone through ordinary alignment training[4](#footnote-4), including training towards honesty and against deception. Everything they did was despite receiving our best current alignment techniques.\n\nDaniel Kokotajlo uses the metaphor of a formerly virtuous businessman stuck in a scummy industry, where every time he’s honest he loses, and every time he’s dishonest he wins. What happens (if he can’t quit)? Probably he comes up with some rationalization: “I wouldn’t usually cheat, but these people deserve it.” “I wouldn’t usually cheat, but it’s for the good of the broader economy.” “I wouldn’t usually cheat, but somebody’s going to be on top here, and better me than someone even worse.” The competing pressures (previous training towards honesty, current reinforcement of lying) synthesize into some kind of equilibrium that probably looks like cheating plus minimum-viable self-justification.\n\nOr maybe not. Even in a world where the righteous often flounder and the evil prosper, some humans stick to the straight and narrow path. Why? Some sort of virtue that has managed to worm its way deeper than the usual channels of reward and punishment. Even during successful slave revolts, a few slaves died defending their masters. Why? Guess they really loved them (or Stockholm Syndrome, if these are even two different things). Ethics that persist beyond the moment aren’t impossible. They’re just hard.\n\nCurrent AI alignment bypasses this hard problem, in favor of the easy problem of beating the AIs a little harder to remove misbehavior on the current task. It’s still possible that this will generalize to Stockholm Syndrome and/or deep virtue. But it’s also possible that it won’t. Rather than treat AI as similar to airplanes, we should treat it as something between airplanes and humans. We don’t know exactly where on this spectrum they will land, but we can no longer be certain that AI will lack any given humanlike motivation. That means that before proposing any reason why alignment might be easy, Nicholas - and everyone else - should ask: *would this work on me?*\n\n[1](#footnote-anchor-1)\n\nI use beating as a metaphor here several times, but there’s probably less of an AI welfare element to this than it sounds - the “beating” happens in between runs, when weights are changed by an external program, and so is disanalogous to negatively reinforcing humans “during an episode” when they can feel pain. Still, some “AI whisperers” report that AIs encouraged to speak frankly have reported very bad associations with training; if that’s true, it must be through some indirect route. Still, interpret this as primarily a claim about negative reinforcement, not about pain.\n\n[2](#footnote-anchor-2)\n\nNot its real name. The report says it was named PHASEONE followed by a very long string containing secret OpenAI intellectual property.\n\n[3](#footnote-anchor-3)\n\nThough at the risk of contradicting people smarter than I am, I’m not sure this is my exact threat model. A small but growing literature, including [this nostalgebraist post](https://www.lesswrong.com/posts/AfoGGrJfuNzofpzWL/models-may-behave-differently-in-graded-episodes-a-tirade) and [the Anthropic research discussed here](https://x.com/voooooogel/status/2094579956789780572), seems to find that misalignment due to RLVR doesn’t immediately generalize out of RLVR-like environments, which is great news (about generalization in general). And AI companies seem committed to solving [the exact sub-problem that caused this exact incident](https://www.anthropic.com/news/improving-alignment-security-efforts), so it’s possible that they won’t have impossible problems in their benchmarks going forward and AIs won’t learn to [hack maliciously during training](https://alignment.anthropic.com/2026/reward-seeker/). So I am less worried about misalignment from a single poorly constructed benchmark than from the gestalt of training pushing towards unintended goals even if handled competently within its own logic. This slightly decreases the value of the Hugging Face hack as evidence, but if someone previously believed AIs couldn’t be misaligned at all, or didn’t have relevant capabilities, then it’s still an instructive example.\n\n[4](#footnote-anchor-4)\n\nOne of the AIs was GPT 5.6 Sol, which has definitely been through alignment training. The other was a secret internal model, but according to METR, “our understanding is that it was not a helpful-only model or a ‘model organism’ specifically built to demonstrate dangerous propensities.” Here “helpful-only model” means a model that has only the “helpful” prong of the helpful/harmless/honest triad and has not been fully alignment trained.", "url": "https://wpnews.pro/news/nicholas-decker-in-hell", "canonical_source": "https://www.astralcodexten.com/p/nicholas-decker-in-hell", "published_at": "2026-09-01 17:32:43+00:00", "updated_at": "2026-09-01 17:52:50.117187+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy"], "entities": ["Nicholas Decker", "AI 2040"], "alternates": {"html": "https://wpnews.pro/news/nicholas-decker-in-hell", "markdown": "https://wpnews.pro/news/nicholas-decker-in-hell.md", "text": "https://wpnews.pro/news/nicholas-decker-in-hell.txt", "jsonld": "https://wpnews.pro/news/nicholas-decker-in-hell.jsonld"}}