Learning Functional Subspaces for Neural Network Compression

A new method for neural network compression learns functional subspaces to select which low-rank components to remove from each transformer weight matrix, according to the paper "Learning Functional Subspaces for Neural Network Compression." The approach targets the memory and compute demands of modern transformers by keeping the compressed matrices dense, and therefore efficient on standard hardware. Existing low-rank weight factorization methods, by contrast, choose the subspace to remove from each weight matrix with limited regard for the function it serves.

Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keeping the matrices dense, and thus efficient on standard hardware. Existing methods, however, choose the subspace to remove from each weight matrix with lo