Show HN: A linear model is an embedding table: using VW as transformer memory

A prototype that trains a Vowpal Wabbit (VW) linear model as a precomputed n-gram embedding table and plugs it into a 4-layer transformer as an Engram-style memory cut evaluation cross-entropy from 5.16 to 4.82 nats, a 0.34-nat gap, on a 1-million-token holdout of cleaned WikiText-103. The two otherwise identical students trained for 4,600 steps of 8 × 512 tokens (about 20 million tokens) on the same batches, with the memory student gaining 29% lower perplexity for less than 0.1% more compute per training step plus a one-off CPU VW fit. The targets were Qwen2.5-0.5B hidden states at layer #4, compressed with PCA to d = 64 dimensions, and the author notes VW's online learning means the exported table need not stay frozen.

A linear model is an embedding table: using VW as transformer memory Experiment, In early 2026, DeepSeek introduced a "conditional memory" module called Engram https://github.com/deepseek-ai/Engram . Put simply, Engram adds a very large lookup table to the neural network architecture. At each token position, Engram takes the last few preceding tokens, hashes the n-grams they form, and fetches the table entries at the hashed positions. These entries are combined into a memory vector, which the model can draw on through a gate. The table values are learned by backpropagation, along with the rest of the model during training. Such embedding tables could conceivably be precomputed outside the normal training regime, possibly shortening training time and providing a more modular architecture. To test this, we built a prototype that trains the memory table with Vowpal Wabbit https://vowpalwabbit.org VW , an online learning library, and then plugs the table into a transformer as an in-memory lookup. VW is a natural fit for several reasons: - A VW model already is an embedding table, if we think creatively. VW hashes every feature into a fixed-size weight array. If we use cost-sensitive one-against-all classification with d classes --csoaa d , where d is the dimension of the memory vector, we can learn d linear outputs per feature. This way, each hashed n-gram corresponds to a d -dimensional vector. For each example, VW's d raw predictions are the sum of the vectors of the active n-grams, which is exactly what PyTorch's EmbeddingBag computes in sum mode. This insight means we can read VW's weights and load them into PyTorch without conversion. - Hashing is built in. VW hashes features into buckets, with optional namespaces. We can pass it n-grams as plain text, and never build an explicit vocabulary of n-grams. - It's cheap and fast. Training is plain linear regression with SGD on a CPU. No GPU, no backpropagation through a network, and constant memory regardless of dataset size. - VW learns online. More on this later. So what should VW learn to predict? Engram's table only has to help the next-token loss, which VW will never see as an external system. Instead, we run a pretrained model specifically Qwen2.5-0.5B over training data and record the model's hidden state at layer 4 for every token. <sup 1 fn-1 </sup . Each hidden state is compressed with PCA to d = 64 dimensions and scaled. These form the targets VW learns to predict. After training VW on the data for two passes, we export its weights as a lookup table and freeze it. The student, a 4-layer transformer trained by distillation from the same teacher, looks up the active rows at each position, sums them into a memory vector, and adds it to its hidden state after the first layer, behind an Engram-style gate. The student only learns how to read the memory, and how much to trust it. What ended up happening? We trained two otherwise identical students for 4,600 steps of 8 × 512 tokens on cleaned WikiText-103 about 20 million tokens : one without memory, one with the frozen VW lookup. Both saw the same batches in the same order, and both were evaluated on a holdout of 1 million tokens. The student with memory ends at an evaluation cross-entropy of 4.82 nats against 5.16 without, a gap of 0.34 nats. Put another way, 29% lower perplexity for less than 0.1% more compute per training step, plus a one-off VW fit on a CPU . Since the VW step is negligible compared to transformer training, this experiment shows that precomputing fundamental language features and providing them as a simple lookup table instead of training them as normal network weights can have a large effect on accuracy with a modest compute penalty, at least at this scale. Finally, a note about perhaps the most desirable feature of VW: it learns online, one example at a time. This means the table doesn't have to stay fixed once it's exported. VW can keep training on new knowledge, while the student stays exactly as it is: no gradients, no retraining, just a new set of rows to look up. Such a setup could give transformers a knowledge base that can be updated cheaply, continuously, and separately from their weights, opening up possibilities such as pluggable domain-specific knowledge or per-customer memory. We look forward to experimenting with these in future work. Code for reproduction available upon request. 1. In theory, fundamental features form in early layers of transformers, hence the choice of 4 ↩ fnref-1