What Makes Rotary Positional Encodings Useful?

A paper by Federico Barbero and co-authors, revised 13 May 2025 as v3 on arXiv, argues that Rotary Positional Encodings (RoPE) in Transformer-based large language models are not primarily useful for decaying token dependency with relative distance, as commonly believed. Studying the internals of a trained Gemma 7B model, the researchers found Gemma learns to use RoPE's highest frequencies to construct robust positional attention patterns while greatly preferring its lowest frequencies, which they suspect carry semantic information. The authors mathematically prove RoPE behaviors, verify them experimentally, and propose a modification of RoPE that fixes highlighted issues and improves performance.

Computer Science Computation and Language Submitted on 8 Oct 2024 v1 https://arxiv.org/abs/2410.06205v1 , last revised 13 May 2025 this version, v3 Title:Round and Round We Go What makes Rotary Positional Encodings useful? View PDF /pdf/2410.06205 HTML experimental https://arxiv.org/html/2410.06205v3 Abstract:Positional Encodings PEs are a critical component of Transformer-based Large Language Models LLMs , providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings RoPE , that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths. Submission history From: Federico Barbero view email /show-email/f4e10c35/2410.06205 Tue, 8 Oct 2024 17:07:01 UTC 2,685 KB \ v1\ /abs/2410.06205v1 Wed, 26 Feb 2025 23:03:04 UTC 3,281 KB \ v2\ /abs/2410.06205v2 v3 Tue, 13 May 2025 14:11:59 UTC 3,325 KB References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .