the polysemanticity of polysemanticity in language models

Polysemanticity in neural networks arises from superposition, where a single neuron activates for multiple distinct inputs due to insufficient neurons. In language models, this enables efficient representation of compound words like 'Michael Jordan' and 'Taylor Swift' through combined neuron activations, as demonstrated in Neel Nanda's MLST talk.

Polysemanticity is one of the most important concepts when it comes to mechanistic interpretability, basically, studying the internal representations of neural networks. For some reason, the concept of polysemanticity seems really, really fascinating to me, mostly due to how it kind of relates to superposition in quantum computing. I think polysemanticity is a concept best expressed through visualisation and this is my attempt at sort of presenting a simplified, illustrated overview of polysemanticity. This is largely based on Neel Nanda’s explanation of polysemanticity in his MLST talk https://www.youtube.com/watch?v= Ygf0GnlwmY&pp=ygUQbmVlbCBuYW5kYSA4MDAwMA%3D%3D , which was the first time I was introduced to this concept and which made me pause the video to fully visualise the concept in my head. Essentially, polysemanticity arises from superposition. In other words, it’s the representation of multiple states in the same space. In neural networks, polysemanticity occurs when there are not enough neurons in the model to map distinctly to all the possible inputs. Therefore, the model learns to map multiple distinct inputs to the same feature vector. Or to put it slightly differently, the same neuron activates on multiple distinct inputs. But this is something that can’t be concretely explained without a solid example. And the following example, discussed in the aforementioned talk, made the concept click for me. Imagine you have 25 instances of Python code snippets, and 25 instances of classic novel snippets. Very distinct. There is almost negligible probability of co-occurrence of the two. Now, you have 25 neurons and each neuron activates on a pair of inputs - either a python code snippet or classic novel snippet. There is another a "discriminating" or "domain indicator" feature which indicates which of the two has triggered the neuron. Now if we specifically think about polysemanticity in LLM internals, we can depict it through the transformer layer. One thing to note about polysemanticity is that neural networks tend to exhibit it for very distinct concepts that truly do not have a high likelihood of co-occuring because that would defeat the purpose, right? . Another example later on in the video made the concept even clearer and, if possible, more interesting to me. This time the example is very specific to compound words. Let’s quickly take a look at some pre-requisites: It therefore follows from the above that polysemanticity/superposition is needed to represent these compound words. Let’s take an example: “Michael Jordan” and “Taylor Swift”. If you have only “Michael” - it can be anything, nothing specific. If you have only “Jordan” - again many possibilities; it could be a country or a common name or the shoes? but that’s plural so low probability . Now if “Michael” and “Jordan” occur together, that really changes the representation. We can have a clear unambiguous representation of this compound word. Same goes for “Taylor Swift”. Now circling back to superposition. It is not possible that the tokens in one compound word will occur in the same manner in another compound word. The same talk also involves discussion on an experiment probing the early layers of a transformer on compound word inputs. It found that no single very important to note: single neuron activated at a given compound word. In fact, a single neuron tended to activate for multiple different compound words - superposition So what differentiates the distinct compound words? A combination of activated neurons. At a given compound word, once you sum up the activations of a group of neurons, you would get a clean, distinct representation of that compound word. And this “grouped activations’ sum” is what was at play behind representing the different compound words. This again ties back to the above Python vs novel example: if a single neuron activates at an input, you can’t tell if it’s for a python code snippet or a classic novel snippet. But once you sum it with the distinguishing 26th feature mentioned above, you can instantly tell which of the two it is. It’s kinda like adding direction to the magnitude. You have the distance 4 meters. It could either be 4 meters forwards or backwards. You don’t know. Until you add a sign. - or +. And it’s clear whether forwards or backwards. Just a far-fetched analogy that came to my mind but sorta helps explain. Polysemanticity is quite important in mechanistic interpretability because it actually directly affects the metrics used to evaluate model representations. If we look at the aforementioned example where 25 neurons have a shared capacity of representing either a code or novel snippet, and let’s say that you are testing activations one by one. Till the 25th neuron you find that the 3rd neuron is activated but due to superposition the output is hazy. When you get to the 26th neuron, its addition to the current activations suddenly make the output clear - python code feature. You assume that the 26th neuron is responsible for encoding Python code features. But that is not the case. The previous neuron has also contributed. This can be illustrated if we look at two probing techniques - probing sequentially vs. probing continuously. This is a very simplified and basic example. In reality it’s much more complex but the point is that interpretability metrics have to very well curated so that they sort of probe from different angles, changing things here and there, isolating stuff, to clearly and accurately figure out model internals. To allude to the title - polysemanticity is neither inherently 100% good or bad. It is useful in that it allows a limited number of neurons to represent more concepts than their individual numbers. Without polysemanticity, some concepts would be forced to be dropped in lieu of other more important ones. Simultaneously, polysemanticity also does make mechanistic interpretability a bit cumbersome because it becomes difficult to ascertain what concept a neuron is representing when you can’t isolate the representations concretely due to superposition. Win some, lose some. While this blog is concentrated around polysemanticity in language models, the concept is exhibited in other kinds of neural networks too, including vision models. In fact, it’s even more interesting to study polysemanticity in CNNs because you can see the polysemanticity being visualized in image activations. The Toy Models of Superposition https://transformer-circuits.pub/2022/toy model/index.html paper by Anthropic is a good in-depth, introductory resource to get super familiarized with polysemanticity.