Show HN: A linear model is an embedding table: using VW as transformer memory

Experiment, In early 2026, DeepSeek introduced a "conditional memory" module called Engram.

Put simply, Engram adds a very large lookup table to the neural network architecture. At each token position, Engram takes the last few preceding tokens, hashes the n-grams they form, and fetches the table entries at the hashed positions. These entries are combined into a memory vector, which the model can draw on through a gate. The table values are learned by backpropagation, along with the rest of the model during training.

Such embedding tables could conceivably be precomputed outside the normal training regime, possibly shortening training time and providing a more modular architecture. To test this, we built a prototype that trains the memory table with Vowpal Wabbit (VW), an online learning library, and then plugs the table into a transformer as an in-memory lookup. VW is a natural fit for several reasons:

• A VW model already is an embedding table, if we think creatively. VW hashes every feature into a fixed-size weight array. If we use cost-sensitive one-against-all classification withd classes (--csoaa d ), whered is the dimension of the memory vector, we can learnd linear outputs per feature. This way, each hashed n-gram corresponds to ad -dimensional vector. For each example, VW'sd raw predictions are the sum of the vectors of the active n-grams, which is exactly what PyTorch'sEmbeddingBag computes in sum mode. This insight means we can read VW's weights and load them into PyTorch without conversion.
• Hashing is built in. VW hashes features into buckets, with optional namespaces. We can pass it n-grams as plain text, and never build an explicit vocabulary of n-grams.
• It's cheap and fast. Training is plain linear regression with SGD on a CPU. No GPU, no backpropagation through a network, and constant memory regardless of dataset size.
• VW learns online. More on this later.

So what should VW learn to predict? Engram's table only has to help the next-token loss, which VW will never see as an external system. Instead, we run a pretrained model (specifically Qwen2.5-0.5B) over training data and record the model's hidden state at layer #4 for every token. ¹. Each hidden state is compressed with PCA to d = 64 dimensions and scaled. These form the targets VW learns to predict.

After training VW on the data for two passes, we export its weights as a lookup table and freeze it. The student, a 4-layer transformer trained by distillation from the same teacher, looks up the active rows at each position, sums them into a memory vector, and adds it to its hidden state after the first layer, behind an Engram-style gate. The student only learns how to read the memory, and how much to trust it.

What ended up happening? We trained two otherwise identical students for 4,600 steps of 8 × 512 tokens on cleaned WikiText-103 (about 20 million tokens): one without memory, one with the frozen VW lookup. Both saw the same batches in the same order, and both were evaluated on a holdout of 1 million tokens.

The student with memory ends at an evaluation cross-entropy of 4.82 nats against 5.16 without, a gap of 0.34 nats. Put another way, 29% lower perplexity for less than 0.1% more compute per training step, plus a one-off VW fit on a CPU. Since the VW step is negligible compared to transformer training, this experiment shows that precomputing fundamental language features and providing them as a simple lookup table instead of training them as normal network weights can have a large effect on accuracy with a modest compute penalty, at least at this scale.

Finally, a note about perhaps the most desirable feature of VW: it learns online, one example at a time. This means the table doesn't have to stay fixed once it's exported. VW can keep training on new knowledge, while the student stays exactly as it is: no gradients, no retraining, just a new set of rows to look up. Such a setup could give transformers a knowledge base that can be updated cheaply, continuously, and separately from their weights, opening up possibilities such as pluggable domain-specific knowledge or per-customer memory. We look forward to experimenting with these in future work.

Code for reproduction available upon request.

1. In theory, fundamental features form in early layers of transformers, hence the choice of #4 ↩