How AI Stores Millions of Vectors Without Using Tons of Memory

Rijul, a developer building LiveReview, explains how product quantization (PQ) compresses large vectors in AI applications, using FAISS as an example. He demonstrates that a 768-dimensional vector stored as 32-bit floats takes about 3 KB, scaling to 30 GB for 10 million embeddings, and shows how PQ reduces storage by replacing subvectors with codebook indices.

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. In any sort of AI application, we have the concept of vectors . Vectors come in different shapes and sizes. Especially when they are large, we need a good way to manage them. One way to solve this is by compressing these large vectors . This is especially relevant to FAISS , which I explained in another article of mine https://dev.to/rijultp/how-does-an-ai-find-the-most-similar-information-faiss-explained-1lhh , where it is a library for efficient similarity search over large collections of vectors. Product Quantization is one of the techniques FAISS supports for vector compression and efficient search. Let's visualize this with a problem. Suppose you have a customer support RAG app And for that, you have 10 million embeddings containing chunks from product documentation, FAQs, and support articles. Each of these embeddings has 768 dimensions. Each of those 768 dimensions is used to store a numerical representation of the meaning of that text. Now, each dimension is a float , so we can think of it as a 32-bit float . 32 bits means 4 bytes. So: 768 × 4 = 3072 bytes That's roughly 3 KB per 768-dimensional vector . Now let's scale this to 10 million embeddings: 10,000,000 × 3 KB = 30 GB So this 3 KB per vector becomes 30 GB when scaled to 10 million embeddings. 30 GB is roughly the size of a decent-sized game or several high-quality movies. So imagine searching among this much vector data. It can get expensive. Let's look at the numbers again. We have 10 million embeddings . We can't change that. But we have 3 KB per vector . What if we could compress that 3 KB further, so that when we scale it to millions of embeddings, we get a much lower storage requirement? The technique used to do this is called Product Quantization , or PQ . Let's look at the basic idea behind PQ. Imagine I have this vector: 0.21, 0.73, -0.15, 0.91, 0.34, -0.52, 0.18, 0.66 Instead of storing all these floating-point numbers, PQ: Let's see an example. Suppose we split this 8-dimensional vector into 4 pieces: 0.21, 0.73 -0.15, 0.91 0.34, -0.52 0.18, 0.66 Now, based on these subvectors, we have a separate codebook for each subvector position . For the first position, let's assume this is the codebook: Codebook 1 ID 0 → 0.10, 0.70 ID 1 → 0.20, 0.75 ID 2 → 0.80, 0.10 ID 3 → -0.20, 0.90 Now let's match the first subvector to its closest vector in the codebook. Our first subvector is: 0.21, 0.73 This looks pretty close to ID 1 , which is: 0.20, 0.75 So instead of storing the entire subvector, we store: 1 Similarly, we do the same thing for the other subvectors. For: -0.15, 0.91 we have another codebook: Codebook 2 ID 0 → -0.30, 0.80 ID 1 → 0.10, 0.40 ID 2 → -0.20, 0.90 ID 3 → 0.70, 0.20 The closest one is ID 2 . And we repeat this process for the remaining subvectors. So the original vector: 0.21, 0.73 -0.15, 0.91 0.34, -0.52 0.18, 0.66 can be represented as: 1, 2, 3, 0 Each number is an index into a different codebook. Original vector │ ┌───────────────┼───────────────┐ ↓ ↓ ↓ Subvector 1 Subvector 2 Subvector 3 ... │ │ │ ↓ ↓ ↓ Codebook 1 Codebook 2 Codebook 3 │ │ │ ↓ ↓ ↓ ID 17 ID 42 ID 8 So now we have the vector in this form: 1, 2, 3, 0 Each ID can be stored using just a few bits. For this example, let's assume each ID is stored using 1 byte. With only 4 possible IDs, 2 bits would technically be enough, but we'll use 1 byte here to keep the example simple. We have 4 subvectors: 4 × 1 byte = 4 bytes Originally, we had 8 floating-point numbers: 8 × 4 bytes = 32 bytes So the size went down from 32 bytes to 4 bytes . And remember, this is just an 8-dimensional vector. Real embeddings are often hundreds or thousands of dimensions. For example, a 768-dimensional embedding stored as 32-bit floats requires: 768 × 4 = 3072 bytes With PQ, if we split it into 96 subvectors and store one 8-bit code for each subvector: 96 × 1 byte = 96 bytes That's 32× less storage for the vector representations . When you're dealing with millions or billions of vectors, that difference becomes enormous. And that's the real reason Product Quantization is so useful for large-scale vector search. At its core, Product Quantization is a clever way of saying: Instead of storing every number in a vector, split the vector into smaller pieces and represent each piece by the ID of its closest learned codeword. You lose some precision, but in return, you get dramatically smaller vectors and more efficient approximate distance calculations . So that's about this article. See you in the next one Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity. I'm building LiveReview , a blast-radius aware AI code review built for your business-critical systems. Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters. Spend code review effort where business risk is highest — not spread evenly across every diff. Try LiveReview on your codebase: