Embedding at the Speed of Light

A static embedding model, potion-base-8M, embedded 32.9 million passages on a laptop CPU in 35 minutes, a task that would have taken a transformer-based model roughly 53 days, according to the article. The static approach skips the neural network entirely, using a lookup table that maps each vocabulary token to a vector and averaging those vectors per text. The tradeoff is retrieval quality, since without transformer layers tokens never influence each other.

Embedding a large corpus is usually the slowest part of building a vector search pipeline: transformer-based models capture semantics very well, but they are heavy on CPU, so you either wait days for the embeddings or pay for a costly GPU. Static embedding models avoid that cost by skipping the neural network altogether. A static model is a lookup table that associates each token in the model’s vocabulary with an embedding: to embed a text, you tokenize it, look up the vector of each token, and average them. With potion-base-8M , this let us embed 32.9 million passages on a laptop CPU in 35 minutes, where a transformer-based model would have needed about 53 days. The tradeoff is retrieval quality, because without layers the tokens never influence each other.