I just had one of those delightful moments where I have a very specific idea, and then I search for it (Claude Research in this case), and it turns out that people have already been using that exact concept but just calling it by a name I'd never heard of.
That name is conceptor, but the basic construction is not new and has gone by many other names.
I'll quote from "Conceptors for Semantic Steering" (Triantafyllopoulos et al.) since I couldn't say it any better myself: Given a collection of neural activation vectors associated with a particular concept, the conceptor matrix is defined as the solution to the regularized reconstruction problem:
where denotes the Frobenius norm and is the
apertureparameter controlling the trade-off between faithfully preserving the activation patterns and regularization. Defining the sample correlation matrix , where has activation vectors as rows, the optimization admits the closed-form solution:This reveals the conceptor as a
softprojection operator: rather than fully retaining or discarding each direction, continuously attenuates directions according to the signal energy in .
That's it, that's the whole thing. But in case you're like me and your eyes glaze over just a little bit for equations like that, let me motivate it for you with a specific example.
Say you have a set of activation vectors that are supposed to isolate some concept X, in that the vectors vary a lot in X but are not supposed to vary much in any other aspect (like perhaps they're averaged over many instances of the same X to wash out all the stuff irrelevant to X). In fact to make it really concrete let's say we have the Assistant Axis vectors, each one of which is supposed to represent a persona role (like "pirate" or "accountant"), and the pirate vector is an average over like a thousand things said by a pirate, so stuff like what the pirate is talking about gets averaged out as irrelevant to the concept. In short: the role vectors isolate the concept of persona (what kind of character is speaking these tokens).
Now let's say you want to intervene on this persona concept, meaning you want to mess with an activation in the specific directions that vary a lot for these role vectors and avoid messing with it in any other unrelated directions. In fact let's say you want to transplant a whole persona: you have some text written by Persona and you want to steer your LLM to consistently take on Persona , but not to take on other unrelated characteristics of the text from Persona like conversation topic, or message length, or which language is being spoken (the steering should cause it to act like a pirate but keep speaking Chinese, if the prompt is in Chinese).
One way to do this would be just to take the 274-dimensional subspace spanned by these 275 vectors [1] and that's where you do your intervention, so like you could transplant that whole subspace projection of a vector from one context to another. This is a
Another way you could do this is take the PCA of that set of 275 vectors, arbitrarily pick some number like k=10, and intervene on the subspace defined by the top-k PCA vectors. This is definitely much less bad than using the full subspace, because now we're changing only the directions of highest variance over the roles vectors, but it's still not ideal because there's an arbitrary hard cutoff. Is it really that the 10th PCA direction is essential to the persona concept, while the 11th PCA direction is totally unnecessary?
So here's the natural move: [2] replace the "hard" black-and-white That's it, that's a conceptor. In plain language: we get the directions in latent space that vary most as the concept-relevant thing changes, and then instead of a hard cutoff we treat the concept directions as a region with graded membership. Some directions are definitely-persona, others are definitely-not-persona, but some directions are kinda-persona-ish.
To be concrete about the promised intervention, we could do "concept transplantation" by replacing where is the conceptor operator, is the original activation, and is the "donor" of the transplant.
There's no new research in this post, so my goals are just to cheerlead a concept I think is useful and underrated, and also to provide a Rosetta stone of all the different search terms you might find this under.
In my head I had been calling this thing a "fuzzy subspace", but that name is actually deprecated because there's a namespace collision with a totally different thing related to fuzzy logic. So don't call it that (even tho it would be a good name otherwise).
The first group of names is for the generic object (any PSD matrix with eigenvalues in ), without the specific form of the spectral filter:
The second group of names is for the specific thing where you set some {threshold, noise level, effective dimension...} and get the form or for the spectral filter:
Conceptors were introduced in Herbert Jaeger's 2014 "Controlling Recurrent Neural Networks by Conceptors" (ancient history, I know), and explained more simply in "Conceptors: an easy introduction" (which contains a fucking based first paragraph, go read it).[5]
As I understand it, conceptors were originally developed for reservoir computing, which is an insane-sounding idea where you take a frozen, randomly initialized neural network that you never train, and then just drive that with your input data (which could be a sequence varying in time, so it's an RNN thing). This yields a big messy "reservoir" of random nonlinear functions computed from your input, and the challenge is to recover useful concepts from this by doing ridge regression.
For some reservoir computing stuff, linear probes work great. My audience knows all about linear probes for readout. The challenge is when you want to steer the dynamics of the system toward a particular behavior, while allowing the complex, not-fully-understood behavior to proceed undamaged despite the continual strong steering.
[Look at this little guy running and dancing around](https://www.youtube.com/watch?v=DkS_Yw1ldD4):
When I watch this video, I feel like I'm watching deep magic from the dawn of time. There is no model trained via backprop involved here. It's all ridge regression. And yet it's really exhibiting stable, autonomous dynamics, and the way to select which motion happens is by applying a conceptor at every time step. There's a conceptor for "walking", one for "running", one for "dancing", etc., and each conceptor captures which latent directions have more or less variance for each of that movement type.
It's crucial here that "walking" is a conceptor and not a hard linear projector, for a few different reasons. The clearest one is the interpolation that causes the smooth transitions between different motions. The display at the top left is showing what the current conceptor mix is, and it's usually just a single motion-specific conceptor, but in the transitions they do a linear interpolation:
If you interpolate between orthogonal projectors, the result is not an orthogonal projector, but if you interpolate between conceptors, the result is a conceptor. But the main reason conceptors are natural here is that "walking" doesn't have certain directions in latent space that are crisply "part of walking" and some that are not, instead it's a naturally graded membership. Walking introduces variance/energy/signal into some latent-space directions more than others, but it's not black-and-white.
Besides the data vectors themselves, a single scalar parameter (the aperture) determines a conceptor. The analogy here is to photography: if the opening of a pinhole camera is 0 you get no light, but if it's wide-open you don't get an image either because everything's out of focus; the sweet spot is somewhere in the middle. Higher aperture provides more raw signal but it's less focused. [6] The aperture is equivalent to in the Wiener filter formulation or in the ridge regression formulation.
You can do Boolean operations on conceptors!
(Note that doesn't exist for most of the matrices I've been talking about, but Jaeger already told us how to handle the general case with pseudoinverses / limit definitions.)
These don't technically form a Boolean algebra because distributivity fails. But they're intuitively satisfying, and while the OR operation is basically equivalent [7] to taking the union of two datasets, the AND operation is novel and exciting as a way to combine multiple conceptors into a combined one that's more specific/precise, since it only contains directions in which
Another important property of a conceptor is its quota which is defined as
where is the dimension of the space. So if the quota is 1 the conceptor must be the identity (a degenerate conceptor: everything's inside it), if the quota is 0 the conceptor must be zero, and if the quota is 1/2 then the conceptor takes up half the dimensions of the space, and its negation also has quota 1/2. The quota can be tuned to any value in by changing the aperture, but the relationship is not universal (it depends on the actual data shape). As the conceptor approaches the ordinary "hard" subspace spanned by the data samples, where .
We have a clear progression over time and technology of the same specific concept:
Since Jaeger, people have been calling this thing a "conceptor" when used on a neural network, but so far there's been zero mentions of that word on LessWrong so I thought I'd fix that omission. It survived unchanged from reservoir computing → word vectors → transformers because it only needs a set of vectors.
There's a data methodology which is already mainstream (see Assistant Axis and Emotion Concepts papers), where you define a concept in terms of buckets of dataset samples — the debiasing wordlists were an earlier version of the same instinct — and to me conceptors seem well suited to apply to this. This also entails an important change in the scope of a single conceptor: while Jaeger had "walk" and "jog" as separate conceptors, I'm suggesting a single grand conceptor for "persona", or for "emotion", or "intent".
In fact this suggests a program I've never heard proposed before, where you try to exhaust the space of meaningful concepts within an LLM activation space by iteratively subtracting known, measurable concepts and then characterizing what's left. What's left after persona and emotion are removed? What if you remove token identity directions, and part of speech and which-language concepts? What remains then? This would carve up the whole space into meaningful broad geometric regions in a way totally unlike, say, an SAE.
So yeah, conceptors. I would say "conceptors are the new linear probes" except they're already over a decade old with a successful track record.
because we mean-center them, so there's one exact linear dependence, the way 3 points only define a 2D plane. I'm equivocating between affine subspace and linear subspace here on purpose... Jaeger's original conceptors were not mean-centered (they were true linear subspaces where 0 is a distinguished point), but for the kind of thing I'm talking about here mean-centering sounds like clearly the right choice
yes I've been talking to Claude a lot, can you tell? (Claude didn't write any of this)
assuming isotropic noise; the "whitened" version if the noise is actually anisotropic is an obvious enhancement
the funny inequality-like symbol is for Loewner ordering of matrices
it's actually a pun, ask me to explain in the comments if you don't get it
in practice, to choose , either use Jaeger's criterion of maximizing the gradient of the Frobenius norm, or just sweep it, or target a certain value of the quota
up to normalization stuff, and to OR weighting two datasets equally while taking their union would weight them proportional to their sample counts