How can we make a nice AI system?
Here's a list of all the techniques I'm aware of.
- Train the AI system to be nice. There are a variety of things we can vary in how we train the AI:
- Train using model internals OR using outputs.
- The central internals-based things I’m imagining involve using the internals as a reward signal (e.g., like
this). Calling “CoT” “internals” is sometimes reasonable (we might want to do process supervision on the CoT).
-
Vary how similar the distribution we’re training on is to the distribution that we care about.
-
For instance: do online training VS training in a toy domain.
-
Train using an imitation-based objective (SFT) OR an outcome-based objective (RL) OR train on declarative facts / stories (mid-training).
-
Train for good behavior or train against bad behavior. Training for good behavior might include training the AI to produce good looking reasoning, as in deliberative alignment.
-
Obviously, there’s a big question of how we get the labels / reward signal here, which should be studied. Especially if you’re doing untrusted monitoring.
-
We’ll also need to decide whether to use on or off policy data.
-
If we’re training on facts / stories stating that the AI is a nice guy: We can vary what the stories are, and how we instill the persona. For instance, we might add a bunch of irrelevant quirks to the persona, and train for those. We likely want to have the stories explain
why the AI takes nice actions.
- We might not directly train the policy, but instead train/prompt monitors, and orchestrate a control system. That is, we might ensemble potentially misaligned AI models into a hopefully mostly good AI system. When ensembling, we’ll likely do some amount of rejection sampling of bad actions, and also “factored cognition” / forcing the AI to solve problems that we don’t think it can sabotage. A large part of the problem here is figuring out how to do a good job of interrogating AI models to see if they’re sabotaging you (i.e., “debate” style techniques).
Please let me know if I've missed any techniques!