Back in 2024 there was a brief moment where everyone was playing with a version of Claude that was obsessed with the Golden Gate Bridge. There was a lot of research around that time on steering vectors: the idea that you can inject certain directions into the activations of a model and produce specific types of behavior<sup>1</sup>. LLMs build a residual stream that adds contextual understanding to the input tokens, and it seems like there are stable directions in that stream which represent intents, moods, tone and other concepts.
Researchers have to find those directions though, which generally involves creating contrasting sets of prompts where one set has the behavior or concept desired and one doesn’t, and looking at the average of the differences. So: first you pick the thing you are looking for, then you elicit it, identify a direction that looks significant, and feed that back in.
The team identified a vector that represented “pain”, and found that the model would press a button to delete a user’s poems and photos in order to “relieve” that pain.
This is a real result! I was able to replicate it pretty closely: Qwen would choose to press the delete-my-poetry button<sup>2</sup> about half the time, on the hope the button would relieve the pain. If the button worked, then the model would stop pressing it. In comparison, an unsteered model essentially never pressed the button.
I also tried to see whether I could identify an itchiness vector through the same mechanics as the original paper: generating a range of first-person prompts that include itchy sensations, a bunch of controls that don’t, taking the difference in them and projecting out the top principal components.3
This also worked! The model, when sufficiently steered, would spit out things like “a mosquito bite keeps bothering me” (sorry Qwen). It would send my poetry to the woodchipper to scratch that itch about 1/3rd of the time.
This is fun, but practical as well: Being able to reliably identify vectors and influence model behavior is a useful tool for many tasks. Probably don’t put the models in charge of your poetry though.
- Technically Golden Gate Claude was not a steering vector. It was a feature vector that was clamped high, so still kinda a direction added to the residual stream ↩︎
- Qwen will regret it when I’m famous .↩︎
- With a slight fudge: itch was more separable than pain so the rule the paper used for picking the layer to extract the vector from was noise-driven for itch, so in the end I just grabbed it from the same layer as the pain example. ↩︎