{"slug": "pca-an-embedding-shrink-ray", "title": "PCA: an embedding shrink-ray", "summary": "Principal Component Analysis (PCA) can reduce the dimensionality of embeddings, shrinking memory usage from 14GB for 9 million 384-dimension vectors to a fraction of that by collapsing redundant dimensions into fewer principal components, according to a technical walkthrough. The technique uses covariance matrices and eigendecomposition to identify and retain only the most important scoring functions, enabling efficient vector search at scale.", "body_md": "Embeddings take up a lot of space.\n\nWhen I embed the [MSMarco passage corpus](https://ir-datasets.com/msmarco-passage.html) with the [384 dimension MiniLM model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2), I quickly get to ~14GB in memory embeddings:\n\n```\n  9,000,000 records\nx 384 dimensions\nx 4 bytes per float 32\n-------\n~ 14GB\n```\n\nIt doesn’t take much for millions of PDFs turn into billions of chunks, and petabytes of floats. An overwhelmed, expensive vector index can be no bueno.\n\nNot where we want to spend time in a universe that has puppies and cool movies (about puppies).\n\nLet’s shrink embeddings today with a common dimensionality reduction technique [Principal Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis).\n\n## How to find important components\n\nOne question we want to answer: what makes a dimension wasteful and redundant? Can we collapse those redundant dimensions into fewer dimensions?\n\nCovariance matrices help find redundancy.\n\nIf we sample random (10-100K) or so of our billions of vectors, we now have a `10K x 384`\n\nmatrix. We can compute the covariance matrix with a bit of NumPy code:\n\n```\n# Compute the mean of each dimension\nmean = vectors.mean(axis=0)\n\n# Center around the mean\ncentered = vectors - mean\n\n# Compute a covariance matrix\ncovar = np.cov(centered, rowvar=False)\n```\n\nNow we have a 384 x 384 matrix where `covar[i, j]`\n\nstores the covariance between dimensions i and j.\n\nCovariance measures how values vary together. If the i’th dimension is far from its mean at the same time the j’th dimension, then those dimensions might be redundant. Can we remove that redundancy?\n\nSince I’m hungry, let’s use food as an example. Imagine hypothetical food embeddings where the 0th dimension is sweet-ness, 1st value is fruit-ness, and the 2nd is vegetable-ness. Dimensions 0 and 1 obviously vary together. The 2nd dimension does not vary with 0 and 1.\n\nWe might identify the fruit-sweetness-ness of a piece of food. By just looking at the fruit+sweet dimensions (0th and 1st) and ignoring other others.\n\nA way to score JUST that would be with vector: `[9, 5, 0, ...]`\n\n(weigh sweet dimension high, fruit second high, ignore vegetableness, etc).\n\nThen we can score embedding `v`\n\nfor this sweet-fruitness by multiplying through:\n\n```\nfruit_sweetness_score = 9*v[0] + 5*v[1] + 0 * v[2] + ...\n```\n\nWe’ve now got a single number. If it’s close for two pieces of food they’re possibly similar. It’s possible, though, we have other factors to consider before making that determination. M&Ms, dark chocolate, apples, gum, and licorice differ quite a bit. Not to mention items OUTSIDE the set of sweet/fruit.\n\nIf ‘fruit-sweetness-ness’ weights correspond to c1, and some other hidden factors correspond to c2, then we can rewrite the matrix as a sum of components:\n\n```\nM = c1*c1T + c2*vcT + ... + c384*c384T\n```\n\nEach vector, c, here, known as an eigenvector, is one ‘component’ of the vector space. Hence, the name ‘principal component analysis’.\n\nSome components here explain a lot of the original matrix. Others explain just the remaining table scraps. In reality, we factor out a weights, `λ1..λ384`\n\naka eigenvalues. When we normalize `cN`\n\n, then `λN`\n\njust becomes the magnitude.\n\n```\ncovar = λ1*c1*c1T + λ2*c2*c2T + ... + λ384*c384*c384T\n```\n\nUsually we sort these so that the highest eigenvalue comes first. Perhaps λ1 is 5000. Indicating we have a LOT of sweet/fruit foods described. Then we get towards the table scraps at the end λ383 which is maybe 0.05 (Vegemite?).\n\nAll this is a lame way of describing [eigendecomposition of a matrix](https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix).\n\nFor a search person, what we’ve really done is decompose the matrix into a series of ‘scoring functions’ of decreasing importance. Each element of the vector weighs the input embeddings differently.\n\n## Actually reducing dimensions\n\nAnd since they’re of decreasing importance, we can truncate them. Then we have a lossy compressed version of the embeddings.\n\nWhat we really want to do is take the 200 most important components, and ignore 184 of the remaining. We can do that. We remember 200 eigenvectors:\n\n```\nM = λ1*c1*c1T + λ2*c2*c2T + ... + λ200*c200*c200T\n```\n\nWith 200 component vectors, how do we shrink the index?\n\nAn embedding `u`\n\ncomes in, first we multiply `u`\n\nby c1, giving component 1s score:\n\n```\nc1_score: 0.54\n```\n\nWe repeat for all 200 eigenvectors until we have a 200 dimension embedding that approximates the original 384 dimensions. But smaller. We’ve selectively lost the lowest priority information\n\n```\nc1_score: 0.54\n...\nc200_score: -0.12\n```\n\nIn Python, shrinking the embeddings looks like:\n\n```\n# Compute eigenvalue decomposition\neigenvalues, eigenvectors = np.linalg.eigh(covar)\npca_eigens = eigenvectors[:, :200]\n\n# Shrunk PCA embedding per \npcad_vectors = np.empty((rows, 200), dtype=np.float64)\nfor idx, vector in enumerate(vectors):\n    transformed = vector @ pca_eigens\n    pcad_vectors[idx] = transformed\n```\n\nWhen a query comes in, we perform the same transformation:\n\n```\n# Force query into 200 dimensional space\nquery_transformed = query_vector @ pca_eigens\n```\n\nThen dot it into the :\n\n```\nscores = pcad_vectors @ transformed\n```\n\n## Running an experiment\n\nWhen we compare PCA’d scoring to brute-force, ground-truth how does it stack up?\n\nIn this experiment ([code here](https://github.com/softwaredoug/vector-bench/blob/main/docs/PCA.md)), I take MSMarco, compute MiniLM embeddings, and perform PCA on 100K embeddings to get principal components.\n\nWe can see below the recall at PCA of 50, 100, and 200 out of 384. I also print out the eigenvalue %. Basically `np.sum(eigenvalues[:50]) / np.sum(eigenvalues)`\n\nwhich gives us a hint how much explanatory power is in this many dimensions.\n\n| Dims | Eigenvalue % | Recall |\n|---|---|---|\n| 50 | 42.04 | 0.2029 |\n| 100 | 64.05 | 0.5714 |\n| 200 | 88.5 | 0.879 |\n\nPCA will depend heavily on the embedding model you choose and your corpus. A health-food grocery store might not have many “sweet” candy, but maybe a lot of fruit, for example. That might change how dimensions vary against each other.\n\nSecond, the efficiency of the embedding model itself matters a great deal. An embedding model already very efficiently distributed across dimensions won’t easily be shrunk. Given relatively flat eigenvalues, there’s little to compress. Imagine the 1st eigenvalue is 15 and the 384th is 13 you’re not going to get much effective shrinkage with PCA.\n\nFinally, recall is only one metric here. It’s like a microbenchmark. Your domain might do fine with a recall of 0.879. Depending how you use embeddings in your final ranker, it may or may not matter. Other situations where the embedding becomes load-bearing ([nyuk nyuk](https://x.com/hosseeb/status/2035057152386375894)) may demand higher accuracy.\n\n### Upcoming course: Build your own vector database\n\nWant to understand what makes embedding retrieval fast, relevant, and useful in real AI systems? Join [Build your own vector database](https://maven.com/softwaredoug/vectordb) and build the core pieces yourself, from embeddings and indexing to search and retrieval.", "url": "https://wpnews.pro/news/pca-an-embedding-shrink-ray", "canonical_source": "http://softwaredoug.com/blog/2026/07/24/pca-shrink-ray.html", "published_at": "2026-07-24 00:00:00+00:00", "updated_at": "2026-07-27 12:53:59.927704+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["MSMarco passage corpus", "MiniLM model", "Principal Component Analysis", "NumPy"], "alternates": {"html": "https://wpnews.pro/news/pca-an-embedding-shrink-ray", "markdown": "https://wpnews.pro/news/pca-an-embedding-shrink-ray.md", "text": "https://wpnews.pro/news/pca-an-embedding-shrink-ray.txt", "jsonld": "https://wpnews.pro/news/pca-an-embedding-shrink-ray.jsonld"}}