[Submitted on 8 Oct 2024 (
[v1](https://arxiv.org/abs/2410.06205v1)), last revised 13 May 2025 (this version, v3)]
[View PDF](/pdf/2410.06205)
[HTML (experimental)](https://arxiv.org/html/2410.06205v3)
Abstract:Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.
Submission history #
From: Federico Barbero [
[view email](/show-email/f4e10c35/2410.06205)]
**Tue, 8 Oct 2024 17:07:01 UTC (2,685 KB)**
[\[v1\]](/abs/2410.06205v1)
**Wed, 26 Feb 2025 23:03:04 UTC (3,281 KB)**
[\[v2\]](/abs/2410.06205v2)
**[v3]** Tue, 13 May 2025 14:11:59 UTC (3,325 KB)
References & Citations
...
Bibliographic Explorer
(What is the Explorer?) Connected Papers
(What is Connected Papers?) Litmaps
(What is Litmaps?) scite Smart Citations
(What are Smart Citations?) alphaXiv
(What is alphaXiv?) CatalyzeX Code Finder for Papers
(What is CatalyzeX?) DagsHub
(What is DagsHub?) Gotit.pub
(What is GotitPub?) Hugging Face
(What is Huggingface?) ScienceCast
(What is ScienceCast?) Influence Flower
(What are Influence Flowers?) CORE Recommender
(What is CORE?) arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.