A research paper proposes cross-layer latent vector quantization to achieve extreme compression of Per-Layer Embeddings (PLE), which account for 35.2% of the parameters of Gemma, according to the paper's authors. The work targets the growing parameter cost of scaling embedding capacity alongside Transformer computation, a trend the authors say makes high compression rates increasingly useful. The paper's headline result is the 35.2% parameter share held by PLE in Gemma.
Scaling embedding capacity alongside Transformer computation is an emerging trend in language models. This trend makes high compression rates increasingly useful. Hence, we study extreme compression of Per-Layer Embeddings (PLE), which follow this trend and account for 35.2% of the parameters of Gem
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.