{"slug": "how-ai-stores-millions-of-vectors-without-using-tons-of-memory", "title": "How AI Stores Millions of Vectors Without Using Tons of Memory", "summary": "Rijul, a developer building LiveReview, explains how product quantization (PQ) compresses large vectors in AI applications, using FAISS as an example. He demonstrates that a 768-dimensional vector stored as 32-bit floats takes about 3 KB, scaling to 30 GB for 10 million embeddings, and shows how PQ reduces storage by replacing subvectors with codebook indices.", "body_md": "*Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.*\n\nIn any sort of AI application, we have the concept of **vectors**.\n\nVectors come in different shapes and sizes.\n\nEspecially when they are large, we need a good way to manage them.\n\nOne way to solve this is by **compressing these large vectors**.\n\nThis is especially relevant to **FAISS**, which I explained in [another article of mine](https://dev.to/rijultp/how-does-an-ai-find-the-most-similar-information-faiss-explained-1lhh), where it is a library for efficient similarity search over large collections of vectors. **Product Quantization is one of the techniques FAISS supports for vector compression and efficient search.**\n\nLet's visualize this with a problem.\n\nSuppose you have a customer support RAG app\n\nAnd for that, you have 10 million embeddings containing chunks from product documentation, FAQs, and support articles.\n\nEach of these embeddings has 768 dimensions. Each of those 768 dimensions is used to store a numerical representation of the meaning of that text.\n\nNow, each dimension is a `float`\n\n, so we can think of it as a **32-bit float**.\n\n32 bits means 4 bytes.\n\nSo:\n\n```\n768 × 4 = 3072 bytes\n```\n\nThat's roughly **3 KB per 768-dimensional vector**.\n\nNow let's scale this to 10 million embeddings:\n\n```\n10,000,000 × 3 KB = 30 GB\n```\n\nSo this 3 KB per vector becomes **30 GB** when scaled to 10 million embeddings.\n\n30 GB is roughly the size of a decent-sized game or several high-quality movies.\n\nSo imagine searching among this much vector data. It can get expensive.\n\nLet's look at the numbers again.\n\nWe have **10 million embeddings**. We can't change that.\n\nBut we have **3 KB per vector**.\n\nWhat if we could compress that 3 KB further, so that when we scale it to millions of embeddings, we get a much lower storage requirement?\n\nThe technique used to do this is called **Product Quantization**, or **PQ**.\n\nLet's look at the basic idea behind PQ.\n\nImagine I have this vector:\n\n```\n[0.21, 0.73, -0.15, 0.91, 0.34, -0.52, 0.18, 0.66]\n```\n\nInstead of storing all these floating-point numbers, PQ:\n\nLet's see an example.\n\nSuppose we split this 8-dimensional vector into 4 pieces:\n\n```\n[0.21, 0.73] [-0.15, 0.91] [0.34, -0.52] [0.18, 0.66]\n```\n\nNow, based on these subvectors, we have a **separate codebook for each subvector position**.\n\nFor the first position, let's assume this is the codebook:\n\n```\nCodebook 1\n\nID 0 → [0.10, 0.70]\nID 1 → [0.20, 0.75]\nID 2 → [0.80, 0.10]\nID 3 → [-0.20, 0.90]\n```\n\nNow let's match the first subvector to its closest vector in the codebook.\n\nOur first subvector is:\n\n```\n[0.21, 0.73]\n```\n\nThis looks pretty close to **ID 1**, which is:\n\n```\n[0.20, 0.75]\n```\n\nSo instead of storing the entire subvector, we store:\n\n```\n1\n```\n\nSimilarly, we do the same thing for the other subvectors.\n\nFor:\n\n```\n[-0.15, 0.91]\n```\n\nwe have another codebook:\n\n```\nCodebook 2\n\nID 0 → [-0.30, 0.80]\nID 1 → [0.10, 0.40]\nID 2 → [-0.20, 0.90]\nID 3 → [0.70, 0.20]\n```\n\nThe closest one is **ID 2**.\n\nAnd we repeat this process for the remaining subvectors.\n\nSo the original vector:\n\n```\n[0.21, 0.73] [-0.15, 0.91] [0.34, -0.52] [0.18, 0.66]\n```\n\ncan be represented as:\n\n```\n[1, 2, 3, 0]\n```\n\nEach number is an index into a different codebook.\n\n```\n                    Original vector\n                          │\n          ┌───────────────┼───────────────┐\n          ↓               ↓               ↓\n      Subvector 1     Subvector 2     Subvector 3 ...\n          │               │               │\n          ↓               ↓               ↓\n     Codebook 1       Codebook 2       Codebook 3\n          │               │               │\n          ↓               ↓               ↓\n        ID 17           ID 42           ID 8\n```\n\nSo now we have the vector in this form:\n\n```\n[1, 2, 3, 0]\n```\n\nEach ID can be stored using just a few bits.\n\n**For this example, let's assume each ID is stored using 1 byte.**\n\n(With only 4 possible IDs, 2 bits would technically be enough, but we'll use 1 byte here to keep the example simple.)\n\nWe have 4 subvectors:\n\n```\n4 × 1 byte = 4 bytes\n```\n\nOriginally, we had 8 floating-point numbers:\n\n```\n8 × 4 bytes = 32 bytes\n```\n\nSo the size went down from **32 bytes to 4 bytes**.\n\nAnd remember, this is just an 8-dimensional vector.\n\nReal embeddings are often hundreds or thousands of dimensions.\n\nFor example, a 768-dimensional embedding stored as 32-bit floats requires:\n\n```\n768 × 4 = 3072 bytes\n```\n\nWith PQ, if we split it into 96 subvectors and store one 8-bit code for each subvector:\n\n```\n96 × 1 byte = 96 bytes\n```\n\nThat's **32× less storage for the vector representations**.\n\nWhen you're dealing with millions or billions of vectors, that difference becomes enormous.\n\nAnd that's the real reason Product Quantization is so useful for large-scale vector search.\n\nAt its core, Product Quantization is a clever way of saying:\n\nInstead of storing every number in a vector, split the vector into smaller pieces and represent each piece by the ID of its closest learned codeword.\n\nYou lose some precision, but in return, you get dramatically smaller vectors and **more efficient approximate distance calculations**.\n\nSo that's about this article.\n\nSee you in the next one!\n\nYour team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.\n\nI'm building **LiveReview**, a blast-radius aware AI code review built for your business-critical systems.\n\nInstead of presenting every diff with equal emphasis, **LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.**\n\nSpend code review effort where business risk is highest — not spread evenly across every diff.\n\n**Try LiveReview on your codebase:**", "url": "https://wpnews.pro/news/how-ai-stores-millions-of-vectors-without-using-tons-of-memory", "canonical_source": "https://dev.to/rijultp/ever-wondered-how-ai-stores-millions-of-embeddings-47ek", "published_at": "2026-08-29 19:55:59+00:00", "updated_at": "2026-08-29 20:19:04.846134+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "ai-research"], "entities": ["Rijul", "LiveReview", "FAISS"], "alternates": {"html": "https://wpnews.pro/news/how-ai-stores-millions-of-vectors-without-using-tons-of-memory", "markdown": "https://wpnews.pro/news/how-ai-stores-millions-of-vectors-without-using-tons-of-memory.md", "text": "https://wpnews.pro/news/how-ai-stores-millions-of-vectors-without-using-tons-of-memory.txt", "jsonld": "https://wpnews.pro/news/how-ai-stores-millions-of-vectors-without-using-tons-of-memory.jsonld"}}