{"slug": "position-encoding-in-transformers-from-absolute-and-relative-methods-to-rotary", "title": "Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling", "summary": "A new technical survey on arXiv (2608.10021v1) provides a unified account of position encoding methods in Transformers, covering absolute and relative embeddings, Rotary Position Embeddings (RoPE), and long-context scaling techniques. The survey concludes that computing positional features beyond training length does not guarantee reliable long-context generalization, and emphasizes the need for comprehensive evaluation across multiple tasks.", "body_md": "arXiv:2608.10021v1 Announce Type: new\nAbstract: Self-attention models content-dependent interactions between tokens but does not by itself encode token order. Position encoding addresses this limitation by introducing absolute coordinates, relative distances, or position-dependent rotations into Transformer representations and attention scores. This technical survey develops a unified account of sinusoidal and learned absolute position embeddings, Shaw-style relative position representations, Transformer-XL, T5 relative position bias, ALiBi, and Rotary Position Embeddings (RoPE). We derive how RoPE converts absolute position indices into relative phase differences in Query-Key inner products and compare these methods in terms of where position is injected, computational cost, compatibility with KV caching, and length extrapolation. We then examine long-context extensions, including Position Interpolation, RoPE scaling laws, NTK-aware scaling, Dynamic NTK, NTK-by-parts, YaRN, LongRoPE, and LongRoPE2, with emphasis on frequency allocation, attention rescaling, training length, and target context length. We also summarize implementation considerations, evaluation protocols, and position-encoding choices in representative large language models. A central conclusion is that the ability to compute positional features beyond the training length does not imply reliable long-context generalization; context extension must be evaluated through short-context retention, position-wise perplexity, retrieval, reasoning, and long-context code tasks.", "url": "https://wpnews.pro/news/position-encoding-in-transformers-from-absolute-and-relative-methods-to-rotary", "canonical_source": "https://arxiv.org/abs/2608.10021", "published_at": "2026-08-12 04:00:00+00:00", "updated_at": "2026-08-12 04:09:36.103907+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["arXiv", "Transformer", "RoPE", "Transformer-XL", "T5", "ALiBi", "YaRN", "LongRoPE"], "alternates": {"html": "https://wpnews.pro/news/position-encoding-in-transformers-from-absolute-and-relative-methods-to-rotary", "markdown": "https://wpnews.pro/news/position-encoding-in-transformers-from-absolute-and-relative-methods-to-rotary.md", "text": "https://wpnews.pro/news/position-encoding-in-transformers-from-absolute-and-relative-methods-to-rotary.txt", "jsonld": "https://wpnews.pro/news/position-encoding-in-transformers-from-absolute-and-relative-methods-to-rotary.jsonld"}}