One Formula to Map the Positional Encoding Landscape

A new survey of positional encoding methods in Transformers argues that the field is best understood not as a chronological progression but as answers to a single question: where position information is injected into the attention computation. The article, drawing on the original 'Attention Is All You Need' paper and an ICLR 2025 blog post, organizes techniques into a 2×2 grid (absolute vs. relative, fixed vs. learned) and identifies three injection points: additive embeddings (yellow), query/key matrix manipulation (blue), and a third region. The survey highlights that sinusoidal encodings, used in the original Transformer, fail to capture relative position effectively and extrapolate poorly to longer sequences.

Every survey of positional encoding I have read presents the methods as a chronological parade: sinusoidal, then learned, then relative, then RoPE, then ALiBi. That framing hides the most useful insight. Almost every technique is just a different answer to one question: where, inside the attention computation, do you inject position? In this article, I summarize the landscape through that lens. First, I revisit where positional encoding sat in the original Transformer paper. Second, I expand the attention formula fully and color-code the three places position information can enter. Third, I collapse the zoo of methods into a single 2×2 grid — absolute vs. relative on one axis, fixed vs. learned on the other — that has served me better than any timeline. Self-attention is a set operation. As the ICLR 2025 blog post on positional embeddings puts it, “On its own, the Transformer architecture is position-invariant, i.e., it processes its input as an unordered set” 2 . Shuffle the tokens of a sentence and, without positional information, every attention score comes out the same. Shirley Li summarizes the fix concisely: “Positional encoding addresses, if not entirely solves, this issue by adding information about the token’s position within the sequence to its representation” 4 . The word addresses is doing real work in that sentence — as we will see, adding to the representation turned out to be only one of at least three options. In Attention Is All You Need , positional encoding is almost a footnote to the architecture: a vector p i added to each token embedding x i once, at the very bottom of the stack, before the first encoder or decoder layer. The input to the network is simply x i + p i. Vaswani et al. chose fixed sinusoidal vectors — sine and cosine waves of geometrically increasing wavelength — and hypothesized that this form “would allow the model to easily learn to attend by relative positions,” since the encoding of position pos + k is a linear function of the encoding of pos 5 . They also tried a learned lookup table instead and, in Li’s words, observed “nearly identical results” 4 . Two properties of this original design matter for everything that came after. Position is injected exactly once, and it then propagates upward entangled with the token’s semantic content. Later research questioned both choices: sinusoidal encodings turned out not to capture relative position effectively in practice 6 , and they extrapolate poorly to sequences longer than those seen in training 8 . The clearest way I know to compare methods is to stop writing attention as softmax QKᵀ/√d V and instead expand it fully for a single pair of tokens: the attention weight a mn between query token m and key token n, followed by the output z m. Every positional encoding technique touches exactly one colored region of this formula. Yellow — additive positional embeddings. Sinusoidal and learned absolute encodings modify the yellow term by replacing x m with x m + p m before the projections W Q, W K are applied. Position enters before attention and rides along inside the embedding. This is the original Transformer recipe, and also that of BERT and GPT-2. Blue — manipulating the query and key matrices. RoPE injects position during the dot product, by rotating queries and keys according to their positions. Because a rotation by mθ against a rotation by nθ leaves behind only the angle m − n θ, the score q m · k nᵀ depends on relative position by construction 7 . Nothing is added to the embeddings; the projection outputs themselves are transformed. Arun Prakash arrives at the same idea from the decomposition of the pre-attention matrix: “one can add positional information directly in the attention layer as well ” 3 . Shaw et al.’s earlier relative position embeddings live here too: instead of adding position vectors to the input embeddings, they inject trainable relative-offset embeddings into the keys and values while attention is being computed 2 , 6 . Pink — a bias on the attention score before softmax. ALiBi skips embeddings and transformations entirely and adds a scalar penalty b mn = −slope · |m − n| to the raw score, just before the softmax. Prakash captures its spirit: “The idea is very simple. Just add a bias hand-crafted after the query-key product” 3 . The farther apart two tokens are, the more their score is pushed down — a built-in recency bias that is the secret behind ALiBi’s famous length extrapolation 8 . T5’s learned relative bias occupies the same pink slot, but with a trained scalar per distance bucket instead of a hand-crafted slope. What I like about this view is that RoPE and ALiBi, usually presented as rivals, are revealed as siblings: both refuse to touch the yellow term. As the ICLR 2025 blog post argues, the philosophy they share is that positional and semantic information are different things that should not be mixed into one vector — so both methods leave the word embeddings alone and instead modify the attention weights computed at every layer 2 . The injection point tells you where position enters; two more questions tell you what kind of position it is. Is position measured from the start of the sequence absolute, as in Vaswani et al. 5 or between pairs of tokens relative, as in Shaw et al. 6 ? And is the encoding fixed — deterministic, unchanged during training — or learned, a lookup table updated by gradient descent? Irani and Metsis organize their survey of the field along exactly these lines, examining “a variety of methods, including fixed, learnable, relative, and hybrid approaches” 1 . Crossing the two questions gives a grid that fits the whole landscape on a napkin. The fixed + absolute corner holds the sinusoidal encoding of the original Transformer. The learned + absolute corner is where BERT and GPT-2 sit, trading extrapolation for task-adapted flexibility; in PyTorch this quadrant is literally one line: python import torch.nn as nn learned absolute positions: one trainable vector per positionpos embedding = nn.Embedding max seq len, d model The learned + relative corner belongs to Transformer-XL and T5, which train embeddings or scalar biases for pairwise offsets, following the direction Shaw et al. opened in 2018 6 . And the fixed + relative corner — RoPE and ALiBi — is where most modern LLMs live: relative by construction, with no positional parameters to train, and with the best length-extrapolation behavior of the four quadrants 2 . Overlaying the grid on the colored formula completes the map. The left column absolute mostly operates in yellow; the right column relative operates in blue and pink. The trend of the last several years is a steady migration from the top-left corner of the grid toward the right column — out of the embeddings and into the attention computation. Positional encoding started as a single additive vector in the 2017 Transformer and grew into a design space of its own. My summary of that space needs only two artifacts: an expanded attention formula with three colored injection points — add to the embeddings yellow , transform the queries and keys blue , or bias the score before softmax pink — and a 2×2 grid crossing absolute vs. relative with fixed vs. learned. New methods keep appearing, but so far every one I have encountered still lands in one colored region and one quadrant. If you keep those two pictures in mind, the landscape stops being a parade of papers and becomes a small set of design choices. 1 H. Irani and V. Metsis, “ Positional encoding in transformer-based time series models: A survey https://arxiv.org/pdf/2502.12370 ,” arXiv preprint arXiv:2502.12370, 2025. 2 “ Positional embeddings in transformer models: Evolution from text to vision domains https://iclr-blogposts.github.io/2025/blog/positional-embedding/ ,” ICLR Blogposts Track, 2025. 3 A. Prakash, “ Positional encoding in transformers https://arunprakash-a.github.io/2024/02/02/PositionalEncodings.html ,” Arun’s Blog, Feb. 2, 2024. 4 S. Li, “ Understanding positional encoding in transformers and beyond with code https://medium.com/@lixue421/understanding-positional-encoding-in-transformers-2c7336728be5 ,” Medium, Dec. 25, 2024. 5 A. Vaswani et al., “ Attention is all you need https://arxiv.org/abs/1706.03762 ,” in Advances in Neural Information Processing Systems NeurIPS , 2017. 6 P. Shaw, J. Uszkoreit, and A. Vaswani, “ Self-attention with relative position representations https://arxiv.org/abs/1803.02155 ,” in Proc. NAACL-HLT, 2018. 7 J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, “ RoFormer: Enhanced transformer with rotary position embedding https://arxiv.org/abs/2104.09864 ,” arXiv preprint arXiv:2104.09864, 2021. 8 O. Press, N. A. Smith, and M. Lewis, “ Train short, test long: Attention with linear biases enables input length extrapolation https://arxiv.org/abs/2108.12409 ,” in Proc. ICLR, 2022. One Formula to Map the Positional Encoding Landscape https://pub.towardsai.net/one-formula-to-map-the-positional-encoding-landscape-7534093c230c was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.