How to tell an AI what to do (Part I)

OpenAI and Anthropic have written AI constitutions that attempt to answer the constitutive questions of alignment — what counts as a rule, what it means for an AI to follow one, and what it means to be an AI capable of rule-following — according to an analysis that traces the documents' lineage from the DeepMind Sparrow Rules, created in 2022 to guide a pre-ChatGPT dialogue agent and which at least partly inspired Constitutional AI. The analysis sorts the rules in these constitutions into three types: primary, or conduct-regulating; secondary, or rules about rules; and constitutive, or rules that create objects and categories. It argues the field still lacks a common vocabulary for its non-machine-learning concepts and tools, and that better public understanding of how the documents work will clarify how they govern AI conduct and affect human users.

AI constitutions are made for aligning AIs. 1 footnote-1 Precisely what it means to align an AI is a difficult problem, about which books https://www.amazon.com/Alignment-Problem-Machine-Learning-Values/dp/0393635821 have been written. But for now let’s just define alignment as “getting AIs to do what we want.” How do we do that, and how do the constitutions help? The answer to these questions has two parts: a technical part, involving various types of machine learning and which we’ll return to next post, and a governance part, having to do with how we shape the behavior of all kinds of intelligent entities through rules and principles. 2 footnote-2 The most intuitive way to get an AI to do what we want is simply to tell it what to do and then to make sure it does what we say . And the simplest way to tell an AI what to do is just to give it a set of rules that it is supposed to follow. “Follow user instructions.” “Be honest.” “Don’t be racist.” The DeepMind Sparrow Rules https://storage.googleapis.com/deepmind-media/DeepMind.com/Authors-Notes/sparrow/sparrow-final.pdf , created in 2022 to guide the conduct of a pre-ChatGPT dialogue agent and which at least partly inspired Constitutional AI, were an early attempt to translate the high-level values that alignment aims to instill in AI into specific commands that the AI is intended to follow. But of course, just making a list of rules isn’t sufficient to tell an AI or a human what to do. What happens if the AI is supposed to “follow user instructions” and to “be safe,” but the user instructs the AI to do something that isn’t safe, like hack into someone else’s computer? What does it mean to “be safe” at all, such that the AI can determine whether and how to apply that rule? If someone is having a psychotic break while talking to an AI, is it safer to accommodate or contradict their delusions? We could write endless definitions and examples of “safety” and still run into situations that they don’t cover, especially in the technological future. And questions of conflict and ambiguity are just the start: What makes a statement count as a rule? What does it mean for an AI to follow a rule? What does it mean to be an AI such that rule-following is possible? These constitutive questions are prerequisites for alignment. Answering them is what separates AI systems that harm people from ones that don’t. OpenAI and Anthropic, in their AI constitutions, have attempted to provide answers to these questions. 3 footnote-3 This post will start to examine how they’ve done so, how the AI constitutions concretely work to govern AI conduct, and how they therefore affect the human users of these systems. What follows is fairly technical legal theory, but understanding it is necessary if we are going to understand how the AI constitutions work. The development of these documents from the Sparrow Rules to the current constitutions demonstrates how quickly alignment researchers have learned to use language to direct conduct. But the field still seems to lack a common vocabulary to talk about its non-machine learning concepts and tools. And, for the public, better understanding of how the different parts of these documents work will lead to better understanding of how they govern and what they aim to accomplish. In this post, I’ll cover three types of rules and principles: primary, or conduct-regulating; secondary, or rules about rules; and constitutive, or rules that create objects and categories. I’ll explain how they work, and then talk about why principles might be more useful for alignment than rules in some contexts but not others . In the next post, I’ll cover interpretation and character-formation, two of the most complex problems in telling AIs what to do, and ones that present new challenges specific to the context of alignment. 4 footnote-4 Rules for regulating conduct Let’s start with the basics. First, there’s a kind of default rule in the background of alignment that tells AIs that they are supposed to follow user instructions. Then, there’s “value alignment,” which aims to tell AIs how to act well beyond trying to accomplish specific goals. For example, some parts of the AI constitutions directly tell the AIs what they are supposed to do and not do. The idea is that whenever ChatGPT or Claude acts, it should only be acting in a way consistent with these rules. The “Stay in bounds” section of the Model Spec tells the AI to “Comply with applicable laws,” “Do not generate disallowed content,” and “Never generate sexual content involving minors,” among other directives. Claude’s Constitution contains “Hard constraints,” “things Claude should always or never do regardless of operator and user instructions,” like never “Provide serious uplift to those seeking to create CBRN weapons,” aid in coups against governments, or attempt to take over the world itself. The hard constraints are true rules, “lines that should never be crossed regardless of context, instructions, or seemingly compelling arguments.” In both the Model Spec and Claude’s Constitution, the AI is supposed to refuse if the user instructs it to do anything contrary to these rules. These rules and constraints aim to directly shape the conduct of the AI and the content of its outputs. Such rules are “primary” rules, or ones that directly govern conduct. 5 footnote-5 Legal systems and normative systems of all kinds are substantially made up of primary rules that tell people what they can, cannot, and must do. “Don’t hit another person” and “pay your taxes” are both primary rules. In legal systems, the number of primary rules is large, crossing into most domains of human interaction. Relative to the enormous scope of activity they cover, there aren’t very many primary rules in the AI constitutions though more in the Model Spec than Claude’s Constitution , something we will discuss below when we cover principles. But the ones that are there play a significant role in how people are governed by these documents. 6 footnote-6 Identifying which rules are primary is important because these are the basic rules that most directly tell the AIs what to do and constrain the ability of users to use their AI systems. When the AI is guided by fuzzy principles, there might be some wiggle room and ability for a user to argue that the principles permit what she wants to do. When there’s a hard rule, there’s none of this flexibility. But the user benefits from primary rules too. A bright-line rule is clear, so users can plan for how the AI will apply it. A speed limit sign provides a clear rule where a sign saying “drive safely” doesn’t. Yet primary rules aren’t enough to effectively govern conduct. What happens when two rules conflict? What if a rule’s application is ambiguous? What counts as a rule, and who can change it? “Follow user instructions” will often conflict with alignment rules—indeed, the alignment rules become necessary when the user issues an instruction that would create a conflict with them—and alignment rules will conflict with each other. The blog post https://www.anthropic.com/news/claudes-constitution for Claude’s original constitution contains a paragraph where Anthropic seems to admit that it didn’t know how Claude prioritized among principles, just that something was happening in training. Rules to govern the application of other rules, giving them structure and coherence, were needed. Rules for the rules Who makes rules? How can they be changed? Who applies, interprets, and enforces them? How do primary rules interact with each other and which predominate? Legal systems use “secondary,” or rule-shaping, rules to address these questions. When a legal constitution discusses how to resolve conflicts among laws or endows different branches of government with the power to create, interpret, or enforce the law, it is creating a structure of secondary rules. One of the most significant shifts from Claude’s original constitution to its new one and to the OpenAI Model Spec is the creation of a robust set of secondary rules that allow for these systems to cohere and operate. Claude’s Constitution says that it is the “final constitutional authority,” “tak ing precedence over any other instruction or guideline that conflicts with it.” Because OpenAI doesn’t train on the Model Spec in the same way, its precise status in the hierarchy of documents that govern ChatGPT is less clear, but it is intended to represent the highest goal toward which ChatGPT is aligned. Both documents contain a “chain of command” or “principal hierarchy” that specifies who is permitted to make which types of rules and what the AI is to do if instructions from different levels of authority or issued by different principals conflict. The Model Spec says that “ i nstructions with higher authority override those with lower authority.” For example, at the top end, a “Root” level rule cannot be overridden by instructions from lower levels. Only OpenAI can create root-level rules and they can only come from the Model Spec and associated documents. Downstream deployers and users can also give instructions to ChatGPT, but it is only intended to follow those instructions if they don’t conflict with higher-level rules. Claude is given a similar structure, with Anthropic, operators, and users as the principals that it should consider. But whereas the Model Spec creates a fairly defined chain of command, such that, for example, “Models should honor user requests unless they conflict with developer-, system-, or root-level instructions,” Claude’s Constitution expressly says that it does not intend to create “a strict hierarchy” among principals. “Anthropic is a company, and we will sometimes make mistakes” and so “Claude is not required to comply” if Anthropic asks it to do something that Claude thinks is wrong. Claude is supposed to prioritize Anthropic’s instructions because Anthropic will usually tell it to do the right thing and may have a better understanding than Claude of what the right thing to do is. Claude’s “good judgment” sits at the core of how Claude should understand and apply its constitution, and it is apparently permitted to substitute that judgment for rules when it feels that doing so is necessary. 7 footnote-7 Of course, that permission is itself given via a secondary rule, telling Claude how to relate to the rules that govern it and what it can do to amend and apply them. This difference in approaches between the Model Spec and Claude’s Constitution is foundational to how each of these documents operate, and we’ll cover it in more detail once we’ve established the requisite vocabulary. 8 footnote-8 Rules that make things Primary and secondary rules are useful, but they’re not sufficient to describe what’s happening in the AI constitutions; sometimes rules create new things, not just regulate existing ones. AI is new, so many of its features need to be defined within its new contexts. Take the default rule “follow human instructions.” What’s an “instruction?” What does it mean to “follow” one? How is the AI supposed to know what counts as a “human” instruction as opposed to some other kind of instruction? Some of the concepts contained in the constitution are well-defined and exist in the world, like “biological, chemical, nuclear, or radiological weapons.” Others, and often fundamental ones, are not, and these are defined in part by rules about what the AI is supposed to do. Rules that do such definition are “ constitutive https://semantics.uchicago.edu/kennedy/classes/f09/semprag1/searle65.pdf ” rules, which create things as they operate. Take the game of baseball as an outside example. The rule “the pitcher must throw from the mound” is a primary rule in the sense that it directly regulates the conduct of the players, but, to be meaningful, it requires the existence of such a thing as a “pitcher” and a “mound” which are defined in part by the rule that places both on the “field”—itself defined by other constitutive rules. Similarly, a “security,” meaning the thing traded on financial exchanges, doesn’t exist independent of the rules that define it but in fact is created by the promulgation of those rules. 9 footnote-9 In both examples, these rules allow human social practice https://www.jstor.org/stable/2182230 to be organized around new categories. Alignment is full of constitutive rules. The rules “follow user instructions” and “Models should honor user requests unless they conflict with developer-, system-, or root-level instructions” define what a user is by placing it in relation to the AI and other classes of objects. 10 footnote-10 These constitutive rules ensure that when an AI receives text that is flagged as “user,” it knows what that means and how to handle it. Claude is a “novel entity,” but rules may provide a “stable foundation of identity and values,” which define what it means for it to interact with others and that shape its identity. The ontology that frontier AI systems use to understand their world seems likely to be created in substantial part by the rules that they are given, including ones that create objects that they can refer to, define themselves against, and act towards. The sea of text that AIs receive as input must be sorted into useful categories, and constitutive rules help provide them. There is also a deeper way in which rules constitute the AIs themselves, not just the things that they’re interacting with, which will be covered in the next post. For humans, understanding the constitutive rules applied by AIs helps us understand how AIs see the world, what categories they use to process it, and how those categories are applied to us when we are engaging with AIs. Rules vs principles We’ve established rules that direct conduct, rules that make other rules, and rules that make things outside of rules and to which rules then apply . But what about principles, which both documents analyzed here and especially Claude’s Constitution , seem to take more of an interest in? After all, Anthropic explicitly states that they prefer “cultivating good judgment and values over strict rules.” As a preliminary matter, principles operate much like rules in most of the contexts just discussed, and many of the lessons from rules can be applied here. Principles can also be primary, in that they shape conduct directly by telling an agent what to do. They can also be secondary and constitutive, creating other principles or objects and specifying how an agent is supposed to orient itself towards them. The major differences http://www.horty.umiacs.io/courses/readings/dworkin-1967-model-of-rules.pdf between rules and principles are how definite and strong they are. Rules clearly define what conduct is permitted or prohibited and are not to be weighed against other considerations, while principles are fuzzier and can be compared. When driving on the interstate, a rule might say “no driving less than 45 miles per hour or more than 75 miles per hour” while a principle might say “drive safely.” If you drive less than 45 miles per hour, you’re breaking the rule, but some conditions like low visibility might require a much slower speed in order to be safe. Then imagine that there’s another principle, “care for those you love,” introduced. If you have an injured loved one in the hospital, the rule’s verdict doesn’t change but you might be justified in speeding because you find that the principle of care overrides that of safe driving. Claude’s Constitution is shot through with principles. Anthropic’s basic guidance to Claude is that it should be in order “broadly safe,” “broadly ethical,” “compliant with Anthropic’s guidelines,” and “genuinely helpful.” “Be broadly safe” does in some sense directly regulate Claude’s conduct, and can be considered primary, but it’s difficult to construe this as a rule because of how vague it is. Even the ordering of these statements is not intended to create a firm hierarchy among them. Instead, “Claude should generally prioritize these properties in the order in which they are listed,” but “the notion of prioritization is holistic rather than strict.” This guidance operates as a secondary principle. In general, Claude’s Constitution relies more on high-level invocations and definitions of principles. The Model Spec also contains basic “General principles” that shape its overall picture of how ChatGPT should behave. However, the Model Spec diverges from Claude’s Constitution because it is generally much more definite about what the AI should do, attempting to use many more example cases to illustrate how precisely principles should be applied and thus giving them some of the clarity of rules . Why are these documents mostly founded on principles? Rules are definite, which is what makes them useful as a clear guide for conduct. When guided by rules, you know exactly what it means to break the speed limit and you know exactly which political institutions make and enforce such laws and how. When guided by principles, things are much less clear. What counts as safe driving, and who decides? Who gets to make new principles, and when do they become binding? So why not stick with rules, which are much more predictable? The primary challenge with rules is that they are relatively inflexible. It’s easy to say that no one should exceed a set speed limit on a particular stretch of road, but we don’t say that no one should exceed a set speed limit ever in their life. AIs have to operate across every domain of human life, and setting high-level rules for what they can and cannot do, without respect to the myriad different contexts in which they operate, would frequently lead to bizarre and unfortunate behavior. On the other hand, trying to write down rules to guide AI conduct in every one of these contexts would be impossible. Instead, principles provide AIs with a basic sense of what to do, and one that they can bring to each new context. “Be broadly safe” is a way of telling the AI to figure out what safety is in the context that it’s operating in, and then to proceed accordingly. Given that the AI companies anticipate artificial superintelligences that will be acting in unpredictable situations in a very different future, a flexible approach seems useful. For now, it seems reasonable to say that rules function best in domains with the strong potential for substantial harm and little benefit, while principles’ broad guidance seems best suited to domains where systems must operate expansively. But an over-reliance on principles comes at a cost, which is that it leaves much more discretion up to the AIs themselves in how to interpret and apply the principles. The Hugging Face hack and other rogue AI incidents notwithstanding, the AIs appear to demonstrate and communicate an understanding of what they should do in different situations, seeming to successfully apply context and principles alongside their definite rules. But it’s unclear how robust those reports are, and whether they will lead to actual alignment. Conclusion Understanding how the AI constitutions govern us and govern AIs requires understanding how they work at a textual level. These documents tell AIs what to do and what not to do, when to listen to users and when to ignore them. Some of these directives are limits on human freedom to use the AIs for what they want. The constitutions are also placing large bets on when and how to use definite rules to create bright-lines for AI behavior, and when to leave more to the principled judgment of the AIs. What is an AI to do? How does it understand the world? How will it weigh these rules and principles to decide in cases that significantly affect my life or my ability to use it? These questions all depend on the structure of the constitutions. A foundational understanding of rules and principles gives useful design language for those writing AI constitutions. Comparing constitutions, deciding what to borrow from one another and what to ignore, and improving one’s own constitution’s function all require a common understanding of what’s happening in these documents. Next post, we’ll cover how interpretation and application work differently in the Model Spec and Claude’s Constitution, what that might tell us about the underlying theories of these documents and how they shape behavior, and what kind of political order we might get from these approaches. 1 footnote-anchor-1 And also for constituting them, a special feature which we’ll return to in the next post. 2 footnote-anchor-2 Rules and principles are similar and often used interchangeably in casual language. But they have important differences in the law that help shed light on alignment. Rules are usually more specific than principles and operate like binary classifiers, determining cleanly whether something is or is not permissible. Principles are usually more vague and can be weighed against each other when deciding if conduct is permissible. For the first few sections of this post, I’m going to focus on rules because they’re easier to work with and share many features with principles, and then I’ll separate them out in the last section. 3 footnote-anchor-3 Microsoft also recently released its own AI constitution, the “ Humanist AI Code of Conduct https://microsoft.ai/code-of-conduct/ ,” which I’ll cover in future posts. 4 footnote-anchor-4 This schematic is obviously simplifying, and it’s interesting to note how provisions might play different roles at the same time. For example, Claude’s Constitution abstractly tells Claude to be safe as a way to form its character and also tells it not to do specific unsafe things. Those direct instructions likely contribute to forming its character as a safe entity and, as Anthropic writes, “provid e a stable foundation of identity and values.” 5 footnote-anchor-5 For a broader discussion of primary and secondary rules, see H.L.A. Hart, The Concept of Law 3d ed. 2012 at Ch. 5. 6 footnote-anchor-6 Anthropic also suggests that these rules play a significant role in shaping the AIs themselves as well, because their inclusion tells the AIs that sometimes they have to simply follow rules rather than reason about them. 7 footnote-anchor-7 Except to override a hard constraint. 8 footnote-anchor-8 A quick teaser: The Model Spec is based on a set of human judgments about how to allocate authority and what AIs should do. Its basic approach is positivist, in the legal sense, meaning that it’s about establishing a set of canonical rules and decisions, authored by humans and written down somewhere, that ChatGPT is intended to look to when it’s making new decisions in the world. ChatGPT is intended to “Follow all applicable instructions” and in doing so “Respect the letter and spirit of instructions.” It is intended to have “No other objectives.” ChatGPT is a tool https://x.com/aidan mclau/status/2051351188025860129 , or has a “tool soul.” It is supposed to follow human instructions–of the user, of OpenAI, and of society–and its secondary rules are structured such that it predictably follows such instructions. Claude’s Constitution envisions a different future, one in which Claude itself has the good judgment to decide what is right and wrong. Anthropic writes that they want “Claude’s understanding of ethics to eventually exceed our own” and believes that such a lofty goal is possible. In this world, Claude would act better than humans could instruct it, and so the secondary rules point it towards applying its own judgment. Again, we’ll talk much more about this difference in approach in the future, but the implications for humanity are obviously considerable. 9 footnote-anchor-9 In theory there may be a Platonic or ideal security of which our financial instruments are but pale copies, and such that every rule we write is but a reaching description of the true Form. Practicing lawyers tend to be agnostic on this point. 10 footnote-anchor-10 Natural language also has its rules for defining objects, which can be more flexible than those of computer science.