Test Time Training

A new paper titled 'Test-Time Training with KV Binding Is Secretly Linear Attention' reveals that test-time training (TTT) architectures with KV binding, often seen as online meta-learning, can be expressed as a form of learned linear attention. The analysis, which contradicts memorization-based interpretations, shows that TTT updates fast weights (like KV binding matrices) during the forward pass, but these weights reset per sequence, so they do not address continual learning. The paper's findings clarify the mechanics of TTT but underscore that it does not enable persistent learning across sequences.

One of the most tantalizing phrases in model development is “new scaling axis”. We have worked out that you can scale parameters by embiggening models, but you then have to scale data appropriately to get the most out of it. Then we 1 worked out we could scale parameters separately from FLOPs. And most recently we realized we could scale at test time, with reasoning. Each time this has unlocked a large boost in model effectiveness. In a simplified view, GPTs 1- 3 were about scaling data, MoEs scaled parameters, and everything since o1 has also scaled test time compute. So the idea of test time training is an appealing one, to wit being a new scaling axis. It also plays into the general theme of continual learning: models at the moment are tremendously intelligent, but you have to put “IMPORTANT: DO NOT IGNORE” into the prompts if you want them to avoid repeatedly making the same dumb mistake. If they could learn, you wouldn’t have to do that So, when you see a paper with a title like “Test-Time Training with KV Binding Is Secretly Linear Attention” https://arxiv.org/abs/2602.21204 you might think that the whole idea is just reframing something we already had: Test-time training TTT with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. To be clear, this paper is interesting and has some good takeaways in case you happen to be building certain kinds of models. But what I found most interesting was the general messiness around “test time training”. “Test time training under distribution shift” https://arxiv.org/abs/1909.13231 is the term-defining paper, and shows what you might expect: updating the model weights to handle distribution shifts seen at test time. These are updates to slow weights : optimizer updated, changing once per minibatch, checkpoints, what you are probably thinking of when someone says ”weights” etc. Then there are fast weights . These are produced in the forward pass, things like the state matrix in a linear attention layer. 2 Normal training doesn’t optimize these directly, it learns a rule that creates them. As you might have guessed, the Test Time Training here is training fast weights, in this case a KV binding layer. This adds a matrix that doesn’t grow with sequence length. It runs a gradient step on it that updates the matrix for each entry in a sequence. If this sounds a bit like a linear attention layer, or an RNN, then, yeah, this paper is right up your alley And as to whether this solves our continual learning problem: the fast weights reset for every new sequence. So, no.