Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization

A research paper proposes cross-layer latent vector quantization to achieve extreme compression of Per-Layer Embeddings (PLE), which account for 35.2% of the parameters of Gemma, according to the paper's authors. The work targets the growing parameter cost of scaling embedding capacity alongside Transformer computation, a trend the authors say makes high compression rates increasingly useful. The paper's headline result is the 35.2% parameter share held by PLE in Gemma.

Scaling embedding capacity alongside Transformer computation is an emerging trend in language models. This trend makes high compression rates increasingly useful. Hence, we study extreme compression of Per-Layer Embeddings PLE , which follow this trend and account for 35.2% of the parameters of Gem