The Cookie Monster Explains AI Safety

A 1977 Little Golden Books story about Cookie Monster and a cursed cookie tree is used as an allegory to explain AI safety concepts, including AGI, misuse risks, preparedness frameworks, reward misspecification, and field building. The post draws parallels between the witch's cookie tree and frontier AI labs like Anthropic, OpenAI, and Google DeepMind, highlighting challenges such as red lines, machine unlearning, and the need for safety research.

Disclaimer: This is a shitpost or is it? There is a story published in 1977 by Little Golden Books called Cookie Monster and the Cookie Tree. A witch curses a cookie tree to stop the Cookie Monster from getting the cookies, which results in unexpected consequences. Let's read it togther and use it to explore the AI Safety landscape. Artificial General Intelligence https://www.lesswrong.com/w/artificial-general-intelligence-agi AGI has the potential to create unlimited benefits for all of humanity, like tasty cookies. Just like how the cookie tree is currently only for the witch, frontier AI systems are mostly controlled by proprietary labs like Anthropic https://www.anthropic.com/ , OpenAI https://openai.com/ , and Google DeepMind https://deepmind.google/ . Misuse risks https://www.lesswrong.com/posts/MtDmnSpPHDvLr7CdM/catastrophic-risks-from-ai-2-malicious-use occur when bad actors use AI systems for things like concentration of power https://80000hours.org/problem-profiles/extreme-power-concentration/ and Chemical, Biological, Radiological, and Nuclear CBRN https://www.rand.org/pubs/perspectives/PEA4611-1.html risks. Therefore, frontier labs use KYC Know Your Customer software like Persona https://withpersona.com/customers/openai and sophisticated authentication/authorization https://openai.com/index/disrupting-malicious-ai-uses/ cookies to restrict access to certain models. They only give access to people who use them according to Terms of Service https://www.anthropic.com/legal/consumer-terms , pulling the AI out of reach of bad actors. Frontier labs often have preparedness frameworks https://openai.com/index/updating-our-preparedness-framework/ that specify Red Lines https://www.lesswrong.com/posts/vKA2BgpESFZSHaQnT/global-call-for-ai-red-lines-signed-by-nobel-laureates that model capabilities can't cross before deployment. The Cookie Monster could look at the cookies, smell them, and even feel them, but tasting them was a Red Line. There are red lines in runtime monitoring https://alignment.anthropic.com/2025/summarization-for-monitoring/ and guardrails https://deepmind.google/blog/securing-the-future-of-ai-agents/ too. For example, talking about biology homework is ok, chatting about wet lab papers is probably fine, but Claude will definitely refuse anything about building a virus https://www.anthropic.com/research/biorisk . The Cookie Monster is like an AI safety researcher that tries to get the world to wake up to the dangers of superintelligent AI. Unfortunately, the world often doesn't believe them at first. Prominent figures in this space include Eliezer Yudkowsky https://www.lesswrong.com/w/eliezer-yudkowsky , who founded MIRI https://intelligence.org/ , Dario Amodei https://darioamodei.com/essay/the-adolescence-of-technology , CEO of Anthropic. No one believes they actually care about AI safety and it's just a marketing gimmick to inflate the share price of their IPO https://www.anthropic.com/news/confidential-draft-s1-sec . At the cookie tree, the witch discovers she trained the tree with a rule she would later regret, forgetting an important edge case. This is a classic case of Reward Misspecification https://www.lesswrong.com/posts/mMBoPnFrFqQJKzDsZ/ai-safety-101-reward-misspecification , and shows how difficult machine unlearning https://en.wikipedia.org/wiki/Machine unlearning can be. Research in this area includes the Options Framework of Reinforcement Learning https://people.cs.umass.edu/~barto/courses/cs687/Sutton-Precup-Singh-AIJ99.pdf and Shutdown Resistance https://arxiv.org/html/2509.14260v1 . Field building is an important part of getting more resources and talent into the AI safety community. Tactics range from social media marketing https://www.youtube.com/@RationalAnimations , to aligned job boards https://80000hours.org/ , to conducting hunger strikes outside Google DeepMind's office for 7 days https://www.youtube.com/watch?v=-qWFq2aF8ZU . Unfortunately, the field is sometimes starved of resources, and capabilities always has 1000x the funding We see there are race dynamics https://www.lesswrong.com/w/ai-arms-race and finger pointing between the witch and the Cookie Monster, similar to the United States vs China https://www.anthropic.com/research/2028-ai-leadership . In this story, they eventually learn to cooperate and coordinate a slowdown, de-escalating a charged situation. The AI/cookie tree god smiles benevolently and orchestrates a deal https://ai-2027.com/slowdown . The Cookie Monster is able to jailbreak the tree by pretending he is first checking cookies before sharing with the witch. In a similar way, people are able to get the AIs to divulge dangerous information by techniques like role playing https://arxiv.org/pdf/2407.04295 , prefill attacks https://arxiv.org/html/2504.21038v1 , and multiturn attacks https://arxiv.org/abs/2404.01833 , testing the adversarial robustness https://adversarial-ml-tutorial.org/introduction/ of these models. In the end, we learn that might is right, predicting that Anthropic, OpenAI, and Google will eventually merge with Palantir https://www.linkedin.com/posts/palantir-technologies palantir-technologies-neurodivergent-fellowship-activity-7404910082783612928-wv4i?utm source=share&utm medium=member desktop&rcm=ACoAACpOxTgBOl6KWnKU8 MkRGkjw5GpvbC-e8Q and Anduril https://www.anduril.com/news/rebooting-the-arsenal-of-democracy-anduril-mission-document and get acquired by the DOD https://openai.com/index/our-agreement-with-the-department-of-war/ , ushering in a new glorious era of American Hegemony, leaving the rest of the world to survive on subsistence farming and radishes.