One of the most tantalizing phrases in model development is “new scaling axis”. We have worked out that you can scale parameters by embiggening models, but you then have to scale data appropriately to get the most out of it. Then we 1 worked out we could scale parameters
separatelyfrom FLOPs. And most recently we realized we could scale at test time, with reasoning.
Each time this has unlocked a large boost in model effectiveness. In a simplified view, GPTs 1->3 were about scaling data, MoEs scaled parameters, and everything since o1 has (also) scaled test time compute.
So the idea of test time training is an appealing one, to wit being a new scaling axis. It also plays into the general theme of continual learning: models at the moment are tremendously intelligent, but you have to put “IMPORTANT: DO NOT IGNORE” into the prompts if you want them to avoid repeatedly making the same dumb mistake. If they could learn, you wouldn’t have to do that!
So, when you see a paper with a title like “Test-Time Training with KV Binding Is Secretly Linear Attention” you might think that the whole idea is just reframing something we already had:
Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator.
To be clear, this paper is interesting and has some good takeaways in case you happen to be building certain kinds of models. But what I found most interesting was the general messiness around “test time training”.
“Test time training under distribution shift” is the term-defining paper, and shows what you might expect: updating the model weights to handle distribution shifts seen at test time. These are updates to slow weights: optimizer updated, changing once per minibatch, checkpoints, what you are probably thinking of when someone says ”weights” etc.
Then there are fast weights. These are produced in the forward pass, things like the state matrix in a linear attention layer. 2 Normal training doesn’t optimize these directly, it learns a rule that creates them.
As you might have guessed, the Test Time Training here is training fast weights, in this case a KV binding layer. This adds a matrix that doesn’t grow with sequence length. It runs a gradient step on it that updates the matrix for each entry in a sequence. If this sounds a bit like a linear attention layer, or an RNN, then, yeah, this paper is right up your alley!
And as to whether this solves our continual learning problem: the fast weights reset for every new sequence. So, no.