A new survey of positional encoding methods in Transformers argues that the field is best understood not as a chronological progression but as answers to a single question: where position information …
A new technical survey on arXiv (2608.10021v1) provides a unified account of position encoding methods in Transformers, covering absolute and relative embeddings, Rotary Position Embeddings (RoPE), an…
Researchers propose MDLMPE, the first positional encoding method for masked diffusion language models that explicitly tracks the evolving revealed/masked token configuration, improving performance ove…
A new study from arXiv introduces AdaRoPE, a method that assigns learnable rotation frequencies and attention scaling factors to each attention head in Transformers, outperforming standard Rotary Posi…
Thinking Machines released Inkling, a 975B parameter mixture-of-experts model with 41B active parameters and a 1M token context length, which does not use Rotary Position Embeddings (RoPE). Instead, I…
Rotary Position Embedding (RoPE), introduced by Su et al. in 2021, solves the Transformer long-context problem by rotating Query and Key vectors in 2D sub-planes, making attention depend only on relat…
A new study analyzing seven pretrained models reveals that the positional encoding scheme in transformers determines the spectral properties of attention heads, with RoPE inducing rotational spectra a…
Researchers introduced LPES, a layer-specific positional embedding scaling method that mitigates the 'lost-in-the-middle' problem in LLMs by assigning distinct scaling factors to each layer, achieving…
A developer explains the importance of positional embeddings in Transformers, comparing Absolute Positional Embedding (APE), Relative Positional Embedding (RPE), and Rotary Positional Embedding (RoPE)…
A technical analysis reveals that large language models fail at long loop tasks due to attention dilution, EOS bias, and stateless architecture, causing protocol drift where constraints are rewritten …
A new appendix maps structural parallels between DNA and large language models, identifying a shared developmental-ledger pattern of possibility, gate, commitment, ledger, inheritance, development, an…
Researchers developed XBAR, an auditable latent crossbar memory fabric that enables model-to-model communication by writing directly into a frozen transformer's KV cache. The system achieves O(1) VRAM…
A new arXiv preprint (2606.18164) by Niklas Forner and coauthors analyzes how transformers learn chain-of-thought state tracking in a solvable setting, training a simplified one-block transformer on p…
A technical analysis calculates the dollar cost per user for serving large language models at scale using napkin math, breaking down GPU resources, matrix multiplication costs, and attention mechanism…
A developer explains how the Transformer architecture works, from self-attention to modern LLMs. The key innovation is that Transformers compare tokens directly via attention rather than processing se…
Researchers have introduced Rotary Value Embeddings (RoVE), a parameter-free modification to Rotary Position Embeddings (RoPE) that makes value pathways position-sensitive by rotating values simultane…
Researchers introduced trajectory extrapolation error, a measure of how much a language model's hidden states deviate from a linear path during word processing, and found it independently predicts hum…
Researchers have introduced CosmicFish-HRM, a compact language model that uses a Hierarchical Reasoning Module (HRM) to dynamically allocate computational effort based on input complexity rather than …
Researchers have introduced LaneRoPE, a novel positional encoding method that enables multiple language model sequences to collaborate during parallel reasoning and generation tasks. The approach uses…