Every survey of positional encoding I have read presents the methods as a chronological parade: sinusoidal, then learned, then relative, then RoPE, then ALiBi. That framing hides the most useful insight. Almost every technique is just a different answer to one question: where, inside the attention computation, do you inject position? In this article, I summarize the landscape through that lens. First, I revisit where positional encoding sat in the original Transformer paper. Second, I expand the attention formula fully and color-code the three places position information can enter. Third, I collapse the zoo of methods into a single 2×2 grid — absolute vs. relative on one axis, fixed vs. learned on the other — that has served me better than any timeline.
Self-attention is a set operation. As the ICLR 2025 blog post on positional embeddings puts it, “On its own, the Transformer architecture is position-invariant, i.e., it processes its input as an unordered set” [2]. Shuffle the tokens of a sentence and, without positional information, every attention score comes out the same. Shirley Li summarizes the fix concisely: “Positional encoding addresses, if not entirely solves, this issue by adding information about the token’s position within the sequence to its representation” [4]. The word addresses is doing real work in that sentence — as we will see, adding to the representation turned out to be only one of at least three options.
In Attention Is All You Need, positional encoding is almost a footnote to the architecture: a vector p_i added to each token embedding x_i once, at the very bottom of the stack, before the first encoder or decoder layer. The input to the network is simply x_i + p_i. Vaswani et al. chose fixed sinusoidal vectors — sine and cosine waves of geometrically increasing wavelength — and hypothesized that this form “would allow the model to easily learn to attend by relative positions,” since the encoding of position pos + k is a linear function of the encoding of pos [5]. They also tried a learned lookup table instead and, in Li’s words, observed “nearly identical results” [4].
Two properties of this original design matter for everything that came after. Position is injected exactly once, and it then propagates upward entangled with the token’s semantic content. Later research questioned both choices: sinusoidal encodings turned out not to capture relative position effectively in practice [6], and they extrapolate poorly to sequences longer than those seen in training [8].
The clearest way I know to compare methods is to stop writing attention as softmax(QKᵀ/√d)V and instead expand it fully for a single pair of tokens: the attention weight a_mn between query token m and key token n, followed by the output z_m. Every positional encoding technique touches exactly one colored region of this formula.
Yellow — additive positional embeddings. Sinusoidal (and learned absolute) encodings modify the yellow term by replacing x_m with x_m + p_m before the projections W_Q, W_K are applied. Position enters before attention and rides along inside the embedding. This is the original Transformer recipe, and also that of BERT and GPT-2.
Blue — manipulating the query and key matrices. RoPE injects position during the dot product, by rotating queries and keys according to their positions. Because a rotation by mθ against a rotation by nθ leaves behind only the angle (m − n)θ, the score q_m · k_nᵀ depends on relative position by construction [7]. Nothing is added to the embeddings; the projection outputs themselves are transformed. Arun Prakash arrives at the same idea from the decomposition of the pre-attention matrix: “one can add positional information directly in the attention layer as well!” [3]. Shaw et al.’s earlier relative position embeddings live here too: instead of adding position vectors to the input embeddings, they inject trainable relative-offset embeddings into the keys and values while attention is being computed [2], [6].
Pink — a bias on the attention score before softmax. ALiBi skips embeddings and transformations entirely and adds a scalar penalty b_mn = −slope · |m − n| to the raw score, just before the softmax. Prakash captures its spirit: “The idea is very simple. Just add a bias (hand-crafted) after the query-key product” [3]. The farther apart two tokens are, the more their score is pushed down — a built-in recency bias that is the secret behind ALiBi’s famous length extrapolation [8]. T5’s learned relative bias occupies the same pink slot, but with a trained scalar per distance bucket instead of a hand-crafted slope.
What I like about this view is that RoPE and ALiBi, usually presented as rivals, are revealed as siblings: both refuse to touch the yellow term. As the ICLR 2025 blog post argues, the philosophy they share is that positional and semantic information are different things that should not be mixed into one vector — so both methods leave the word embeddings alone and instead modify the attention weights computed at every layer [2].
The injection point tells you where position enters; two more questions tell you what kind of position it is. Is position measured from the start of the sequence (absolute, as in Vaswani et al. [5]) or between pairs of tokens (relative, as in Shaw et al. [6])? And is the encoding fixed — deterministic, unchanged during training — or learned, a lookup table updated by gradient descent? Irani and Metsis organize their survey of the field along exactly these lines, examining “a variety of methods, including fixed, learnable, relative, and hybrid approaches” [1]. Crossing the two questions gives a grid that fits the whole landscape on a napkin.
The fixed + absolute corner holds the sinusoidal encoding of the original Transformer. The learned + absolute corner is where BERT and GPT-2 sit, trading extrapolation for task-adapted flexibility; in PyTorch this quadrant is literally one line:
import torch.nn as nn
The learned + relative corner belongs to Transformer-XL and T5, which train embeddings or scalar biases for pairwise offsets, following the direction Shaw et al. opened in 2018 [6]. And the fixed + relative corner — RoPE and ALiBi — is where most modern LLMs live: relative by construction, with no positional parameters to train, and with the best length-extrapolation behavior of the four quadrants [2].
Overlaying the grid on the colored formula completes the map. The left column (absolute) mostly operates in yellow; the right column (relative) operates in blue and pink. The trend of the last several years is a steady migration from the top-left corner of the grid toward the right column — out of the embeddings and into the attention computation.
Positional encoding started as a single additive vector in the 2017 Transformer and grew into a design space of its own. My summary of that space needs only two artifacts: an expanded attention formula with three colored injection points — add to the embeddings (yellow), transform the queries and keys (blue), or bias the score before softmax (pink) — and a 2×2 grid crossing absolute vs. relative with fixed vs. learned. New methods keep appearing, but so far every one I have encountered still lands in one colored region and one quadrant. If you keep those two pictures in mind, the landscape stops being a parade of papers and becomes a small set of design choices.
[1] H. Irani and V. Metsis, “Positional encoding in transformer-based time series models: A survey,” arXiv preprint arXiv:2502.12370, 2025.
[2] “Positional embeddings in transformer models: Evolution from text to vision domains,” ICLR Blogposts Track, 2025.
[3] A. Prakash, “Positional encoding in transformers,” Arun’s Blog, Feb. 2, 2024.
[4] S. Li, “Understanding positional encoding in transformers and beyond with code,” Medium, Dec. 25, 2024.
[5] A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017.
[6] P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” in Proc. NAACL-HLT, 2018.
[7] J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Y. Liu, “RoFormer: Enhanced transformer with rotary position embedding,” arXiv preprint arXiv:2104.09864, 2021.
[8] O. Press, N. A. Smith, and M. Lewis, “Train short, test long: Attention with linear biases enables input length extrapolation,” in Proc. ICLR, 2022.
One Formula to Map the Positional Encoding Landscape was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.