Disclaimer: This is a shitpost (or is it?) There is a story published in 1977 by Little Golden Books called Cookie Monster and the Cookie Tree. A witch curses a cookie tree to stop the Cookie Monster from getting the cookies, which results in unexpected consequences. Let's read it togther and use it to explore the AI Safety landscape.
Artificial General Intelligence (AGI) has the potential to create unlimited benefits for all of humanity, like tasty cookies. Just like how the cookie tree is currently only for the witch, frontier AI systems are mostly controlled by proprietary labs like Anthropic, OpenAI, and Google DeepMind.
Misuse risks occur when bad actors use AI systems for things like concentration of power and Chemical, Biological, Radiological, and Nuclear (CBRN) risks. Therefore, frontier labs use KYC (Know Your Customer) software like Persona and sophisticated authentication/authorization cookies to restrict access to certain models. They only give access to people who use them according to Terms of Service, pulling the AI out of reach of bad actors.
Frontier labs often have preparedness frameworks that specify Red Lines that model capabilities can't cross before deployment. The Cookie Monster could look at the cookies, smell them, and even feel them, but tasting them was a Red Line. There are red lines in runtime monitoring and guardrails too. For example, talking about biology homework is ok, chatting about wet lab papers is probably fine, but Claude will definitely refuse anything about building a virus.
The Cookie Monster is like an AI safety researcher that tries to get the world to wake up to the dangers of superintelligent AI. Unfortunately, the world often doesn't believe them at first. Prominent figures in this space include Eliezer Yudkowsky, who founded MIRI, Dario Amodei, CEO of Anthropic. No one believes they actually care about AI safety and it's just a marketing gimmick to inflate the share price of their IPO.
At the cookie tree, the witch discovers she trained the tree with a rule she would later regret, forgetting an important edge case. This is a classic case of Reward Misspecification, and shows how difficult machine unlearning can be. Research in this area includes the Options Framework of Reinforcement Learning and Shutdown Resistance.
Field building is an important part of getting more resources and talent into the AI safety community. Tactics range from social media marketing, to aligned job boards, to conducting hunger strikes outside Google DeepMind's office for 7 days. Unfortunately, the field is sometimes starved of resources, and capabilities always has 1000x the funding
We see there are race dynamics and finger pointing between the witch and the Cookie Monster, similar to the United States vs China. In this story, they eventually learn to cooperate and coordinate a slowdown, de-escalating a charged situation. The AI/cookie tree god smiles benevolently and orchestrates a deal.
The Cookie Monster is able to jailbreak the tree by pretending he is first checking cookies before sharing with the witch. In a similar way, people are able to get the AIs to divulge dangerous information by techniques like role playing, prefill attacks, and multiturn attacks, testing the adversarial robustness of these models.
In the end, we learn that might is right, predicting that Anthropic, OpenAI, and Google will eventually merge with Palantir and Anduril and get acquired by the DOD, ushering in a new glorious era of American Hegemony, leaving the rest of the world to survive on subsistence farming and radishes.