If you're training any type of toy model of superposition, Mean Squared Error (MSE) loss is unusually bad. [1]
We aren't the first to notice that MSE loss doesn't work. In Toy Models of Superposition the effective loss function is , because they noticed that plain doesn't work.
More recently Compressed Computation Under Loss is Likely Computation in Superposition again demonstrated that MSE loss dosn't work. MSE loss is the same as loss [2] , because MSE is just the square of the norm. The above mentioned paper showed that , , , , all worked to produce superpossition encodings, but doesn't.
Why this post then? Given that the core claim is already thoroughly demonstrated? Two reasons:
Which resulted in me (Linda) using MSE loss in attempted superposition toy experiments. I learned for myself that it doesn't work, did a bunch of math with Phil to prove why it doesn't work, and therefore this post.
If you are satisified with the previous results on this topic, no need to read any further.
Suppose you want to train a neural network to encode more features than it has neurons. If you train this network using MSE loss on these features, the network is not incentivized to encode them in superposition. So if there's anything else going on in the experiment that may push the solution towards non-superposition, you're most likely going to end up with non-superposition. This can be an issue when trying to do toy models specifically of superposition. One of us (Linda) has run into this problem.
However, the solution is easy: Don't use MSE loss, instead use some other loss instead. If cross-entropy loss is not applicable, you can use instead of .
In this post we pressent three lines of evidence for this claim:
We consider the math to be the strongest evidence for our claim, becuase the best way to know if something generalises is to do the math. But the math section is also the longest and the densest part of the post, which is why it's pressented last.
MSE loss will not push towards superposition, but nor will it always push against superposition. If you're using random initialization, you may sometimes get (something like) superposition, just from your starting conditions.
E.g. One toy model of Compressed Computation looked like it was performing computation in superposition, untill closer investigation.
This is the backstory for why we ended up doing all the investigations that follow in the rest of this post.
I (Linda) was doing some computation-in-superposition toy model experiments. In these experiments, a neural network was trained to represent more tiny circuits than it had hidden layer neurons (but it would only have to compute one or very few circuits in any single forward pass). In these experiments, I also introduced noise in the input. [3] I had the hypothesis that introducing noise to the input would reduce the number of neurons used per circuit, since that would reduce the amount of noise passing through the network and ending up in the output.
I was right! Introducing noise did cause the network to distribute each circuit over a much smaller number of neurons.
However, I had expected the network to come to some trade-off between using more neurons per circuit, in order to get more superposition (to maintain it's ability to tell different circuits apart), and fewer neurons per circuit, in order to get less noise. Instead the network just went for (close to) as few neurons as it could, and did not seem to care about maintaining superposition.
So then I went on a side quest to find out why, which much later [5] became this post.
Another in-the-wild observation is this project, which found that another intended-to-be toy model of superposition, also trained with MSE loss, was (probably) not actually superposition.
Imagine you're training a neural network. At the last hidden layer of the model, you'd like to incentivise the network to represent the final output in superpossition.
The last hidden layer has neurons, and the output has features where is larger than . Each feature can either be active, represented by , or inactive, represented by . There are at most active features at any one time.
In this post, we're not concerned about how the last hidden layer is computed. Imagine that some network has enough layers or whatever it needs to produce the optimal encoding for its output features in the final hidden layer. What we want to find out is, what's the best possible encoding the network can use, just before the readout, given the bottlneck ? The way we investigate this in practice (both in the experiments and the math), is to feed the desired output as input, into a single hidden layer of dimension .
Typically in superposition one also assumes that the features only activate sparsely. In this post we'll assume that at each forward pass, exacty features are active, for some small value of . [6]
We also assume that every allowed output (given and ) is equally likley.
Then, given , , and some specific loss function, what is the optimal feature embedding? I.e. what should the last layer activations be, as a function of the set of active features, in order to minimise the loss?
We train a linear autoencoder to embed and then unembed features, into neurons and then back into features again. The input is a -dimensional -hot vector, and the target is always the same as the output. We train each network on a dataset of every possible such feature vector.
Code for the model:
class FeatureCompressionModel(torch.nn.Module):
def __init__(self, T, D):
super(FeatureCompressionModel, self).__init__()
self.encode = torch.nn.Linear(T, D, bias=False)
self.decode = torch.nn.Linear(D, T, bias=False)
self.D = D
self.T = T
self.bias = bias
def forward(self, x):
x = self.encode(x)
x = self.decode(x)
return x
Code for generating the training data:
if z == 1:
data = torch.eye(model.T, device=next(model.parameters()).device)
elif z == 2:
data = []
for i in range(model.T):
for j in range(i+1, model.T):
one_data=torch.zeros(model.T)
one_data[i] = 1
one_data[j] = 1
data.append(one_data)
data = torch.stack(data, dim=0)
We then trained these models using four different loss functions:
loss = torch.nn.MSELoss()(outputs, targets)
loss = torch.nn.BCEWithLogitsLoss()(outputs, targets)
loss = torch.mean((outputs - targets)**4)
loss = torch.nn.functional.cross_entropy(outputs, targets.argmax(dim=-1))
We're testing:
Finally we plot the embedding vectors.
We trained models with each of [T=3, D=2, z=1], [T=4, D=2, z=1], [T=5, D=2, z=1] and [T=5, D=2, z=2], on the four different loss functions (MSE, BCE, L4 and CE), five times each. The embeddings of the trained models are visualised below.
By default embedding vectors are green. But small embeddings are yellow (norm is less than 20% of the largest one) and very small ones are red (norm is less than 10% of the largest one).
Dotted green lines are the negative of the embedding vectors.
Superposition here looks like "vectors and their negations being far away from each other". And just as we claim, MSE loss does not consistently produce superposition: it's common for vectors to be nearly on top of each other. Our math (next section) suggests that no embedding should be prefered over any other embeding when when using MSE loss (at least for ). The results seem to confirm this, since the MSE embeddings are very varied.
For z=1, MSE is the only loss function that does not produce a clear pattern in how the embeddings look. For z=2, they're all a bit wonky.
Networks trained with BCE loss are doing well at producing superposition for [T=3,D=2,z=1], but "give up" for larger T, and instead just represent two features approximately orthogonally, and don't represent the others. This is probably because positive interference is particularly bad for BCE. But I (Linda) am still a bit suprised it didn't manage to do something more like superposition for [T=3,D=2,z=1].
Networks trained with L4 loss avoid negative correlations as much as they avoid possitive correlation. This is expected, since L4 loss penalises negative and positive interference equally. This is also true for MSE loss, but it's less obvious in the results, because MSE loss is just all over the place with no clear pattern.
Networks trained with CE loss are arguably doing best in terms of producing superposition, with the main drawback that this loss is only applicable for z=1.
If you want to know if something generalises, it's best to do the math.
In this section, we do a lot of math that does not quite prove the core claim, but shows that it's probably true.
We consider the math in this section to be our main result, and the strongest evidence for our claim at the start of the post. When working on this project, we did the math first, and only later did the experiment to confirm our conclusions.
Since you've made it here we assume you like math. So we'll define the general setup again, now using math notation.
You're trying to implement (or at least closely approximate) some function with a neural network. The functions we're looking at here have output type (i.e. they output a -dimensional vector of s and s) where at most of the output values are . The network's final hidden layer has neurons, and the network's approximation of the answer is read out as some linear function of the final layer.
We're interested in questions of whether and how networks can encode certain information, so the function you're trying to implement doesn't matter. For simplicity, we'll say it's the identity function: your network takes a vector of binary inputs (where at most of the inputs are ), and spits out real outputs, and is trying to reproduce the input as closely as possible.
(This function is the best case scenario, in some sense. Whatever the best-possible performance is on the identity function, it must be at least as good as the best-possible performance on any other function, with caveats around frequency of results.)
That means we can split your function into two: the part that computes the final hidden layer, and the part that reads out the result from that. We're not worried about the limitations of the first function, so we'll say it can be anything at all, regardless of how easily it can be implemented with a neural network.
As another simplification, we'll say that exactly of the inputs will be , not at most .
And so our moving parts are:
Our expected loss is
So the question is: what are some ways we can choose and , and what losses do they give us?
In this encoding, each neuron simply represents a single feature. Since there are fewer neurons than features, that means some features don't get represented. For represented features, we know exactly whether they were active or not; for the others, we simply guess that they weren't.
So our estimates will be
So all the loss is going to come from the unrepresented features. The number of unrepresented features is and the probability of a feature being activated is . The activated unrepresented features give loss and the nonactivated unrepresented features give loss .
So ultimately,
We can probably slightly improve on this for . The current loss is good enough for this post, but details in footnote [11] .
Next, we could give every feature a neuron. That means some neurons represent multiple features, and our estimates won't be able to distinguish them. On average, each neuron will represent features.
Since is small, "two active features being represented by the same neuron" is unlikely. We'll assume it's unlikely enough not to change the MSE much, and pretend for now it can't happen. (For it actually can't happen.)
Here the loss is all going to come from active output features. We'll have features that may have been activated, and the rest we know won't be. For example, this might concretely be implemented as
So that is "the number of active features that this neuron represents" (and ), and each is either or ).
Each is either or . We can split into three parts:
So average expected loss is
minimized at , giving
Same as we got for "1 feature per neuron"! We'll call these embeddings "orthogonal", and use the symbol for their loss.
Again, if , we can get a slight improvement by relaxing the "only one active feature per neuron" assumption. Discussion is delegated to footnote [13] .
Here we say that each feature feeds into several neurons, and each neuron has several features feeding into it.
For each feature, we pick a random unit vector . In high-dimensional spaces, most random vectors are approximately orthogonal, so for . We sum the active features' unit vectors into the embedding.
We think this is a close-to-optimal implementation of superposition, but we don't prove that here.
So we have
where we pick to minimize loss. This works (to the extent that it does) because ; the terms of this sum other than are all approximately , so .
The loss from each inactive feature is , and from each active feature is , where are error terms that come from the not being fully orthogonal. Known theorems about random unit vectors give us
Then the total loss is
This is minimized at , ultimately giving
Clearly which gives us .
A question remains: whether random embeddings are really a good representation of superposition. Since we defined superposition to be almost anything except the two previous embedding schemas, can't we do better than random? The answer to this is yes, as we'll see in the next two sections.
Notice that all the calculations for are exact, and not just some large limit result. So we can check the smallest possible case to see if there's a better option for superpositional embedding.
In the case , , , it sems clear that the best superposition embedding should be the vertices of an equilateral triangle:
With . Using the embedding schema from before, if feature is activated then , and so the active feature has loss and the two inactive features have loss . That gives us
which is minimised at , and so
which is indeed better than
We conclude that in this case (and by extension, in general) it is possible to do better than choosing the embedding vectors at random. But what really matters is whether the best-case superposition beats the other embedding options. And no, they turn out to have exactly the same loss:
Let's test one more example. In this case the best embedding would be the vertices of a regular tetrahedron:
with .
Same reasoning as before. If feature is activated then , and so the active feature has loss and the three inactive features have loss . We minimize MSE at , giving
which again matches
So even the best version of superposition is still no better than the orthogonal embeddings, for these values. We expect this to generalize, but do not have a mathematical proof. [14]
Our proof is incomplete in a handful of ways.
For , we suspect that the optimal MSE loss will be the same for every embedding, regardless of and , as long as the embedding is using all the neurons [15] , and as long as the unembedding is tuned optimally for every feature. (Opus has a
Why? Because it would be surprising if both (what we call) "the best" superposition solution, and the two orthogonal solutions, would all be ideal; but nothing in between is as good.
But if so, why is better than our result for ? We think because we picked a single value of for our superposition unembeddings. "The best" superposition is symmetrical, so using the same for all features is correct. But a random embedding won't be symmetrical. Some features will be closer to others and would ideally have a lower , while more lonely features will ideally have a higher , but we didn't tune them. This rhymes with the discussion of "what if isn't an integer" in footnote [12:1] .
We don't think every embedding is exacly equal for .
Superposition is a fuzzy concept, with no precise consensus definition. We mean that each feature is encoded linearly, using almost orthogonal direction. But actually, for this post it's enough to say that it's definitely not superposition if all feature embeddings are exactly orthogonal, or if any two feature embeddings are identical. For the purpose of this post, we'll consider everything else "superposition".
Even with this overly broad definition, we'll still show that MSE Loss doesn't favor superposition over non-superposition. ↩︎
They have the same minimum, and the gradient is always in the same direction, so they're functinally the same under gradient descent. ↩︎
This was to simulate being in the middle of a network, where I expect input features to be some amount of noisy, due to superposition compression in previous layers. ↩︎
I (Linda) have used this trick for noise reduction myself in the construction here: Circuits in Superposition 2: Now with Less Wrong Math. ↩︎
Most of the delay was me pursuing other research projects. ↩︎
We don't use the assumption that is small for anything. But this is the setting where one might expect superposition, which is why that's the main focus of this post. ↩︎
We use for all experiments, because it's the only value for that's easy to visualise. ↩︎
For this to be any interesting we need to be larger than , i.e. larger than . The numbers and are some nice numbers that are all larger than . ↩︎
There are several reasons to prefer :
But despite this list of reasons for , we also did one experiment with , because why not. ↩︎
We could also allow the function to be affine (i.e. "linear plus some constant"). The difference between linear and affine is going to be small for large D, because if the constant is helpful, you can just sacrifice a neuron to get it (the "augmented matrix" representation). ↩︎
First, note that if we allowed affine readouts, then our estimate for unrepresented features could be , for some constant . We'd choose to trade off between "false positives" and "false negatives"; the optimal choice is , which gives .
With linear readouts, we don't have that constant. But is linear, and lies between and . So we could set , for some constant .
This doesn't help for . If , then we know the active feature is represented, and we want to predict the unrepresented features as ; if it's , then we know the active feature is unrepresented, but we have no way to predict the unrepresented features as anything except .
But it seems like it should help for higher . There's still an awkward tradeoff that the fewer active unrepresented features there are, the higher we'll predict them. But consider . Our loss from the unrepresented features will be:
So we minimize total expected loss at .
We're not going to bother getting a closed-form solution for MSE with this adjustment, but it'll be a mild improvement. ↩︎
This isn't quite right when isn't an integer. There'll be some neurons corresponding to features and some corresponding to features. The latter neurons activate more often, and they have higher loss when one of their features is active. This shouldn't make much difference when is large, and it's possible to push back against this effect by picking different values of for the different groups. Linda actually calculated it out and showed that we ultimately get the same loss. But the math for that is a bit messier, so we ignore it here. ↩︎ ↩︎
If we accept that multiple active features might be assigned to the same neron, that complicates the analysis in two ways. First, there are fewer "false positives", which gives a decrease in loss. Second, some of the come out as 2 or higher, which has unclear effects on loss.
But since can be any arbitrary function, we can instead set
and the second complication disappears. So our MSE would end up being slightly better than we calculated.
This new function is somewhat harder for "all-but-the-final-layer of a neural network" to implement.
Note: we don't claim this new function is optimal. It's probably not. It might not even be an improvement on the simple . What we claim is that for , when we take into account "multiple active features might be assigned to the same neuron", this new function gives us MSE lower than . ↩︎
One of my (Linda's) undergraduate physics teachers called this sort of thing "Proof by Physicist". You test your conjecture for some small values that are easy to check, and if it works, you assume you're correct. Although this method is probably more reliable if you already know that you're right, e.g, because you're teaching a physics undergraduate course, where everything is long established facts. ↩︎
To be precise, for any embedding function is linear, so "using all the neurons" means the embedding matrix is rank . ↩︎