Researchers proposed Kronecker Product Attention (KPA), a KV-cache compression method that represents key and value activations as sums of Kronecker products of compact, token-dependent matrix factors…
An optimized training stack can reach roughly 51% model FLOPs utilization (MFU) on consumer GPUs, according to the LLMQ paper, leaving headroom for a user currently getting about 65 TFLOPS of useful c…
LayerRoPE, a depth-wise positional encoding method presented in arXiv paper 2610.09179v1, replaces all layerwise normalization weight vectors in a Transformer with a single shared vector and depth-con…
A new arXiv paper (2609.35852v1) analyzes Transformer weights at a mesoscopic level using row and column scale fields — median-centred log-RMS profiles over a matrix's channels — which, with a global …
A feasibility experiment called qwen3-ebpf ran a full Qwen3-0.6B forward pass inside Linux eBPF, with BPF programs computing all 28 decoder layers, the output logits, and the next-token argmax while a…
Moonshot AI's Kimi K3, a 2.8-trillion-parameter mixture-of-experts model with 104 billion activated parameters per token and a 1,048,576-token context window, has brought attention to the debate betwe…
A new survey of positional encoding methods in Transformers argues that the field is best understood not as a chronological progression but as answers to a single question: where position information …
A new technical survey on arXiv (2608.10021v1) provides a unified account of position encoding methods in Transformers, covering absolute and relative embeddings, Rotary Position Embeddings (RoPE), an…
Researchers propose MDLMPE, the first positional encoding method for masked diffusion language models that explicitly tracks the evolving revealed/masked token configuration, improving performance ove…
A new study from arXiv introduces AdaRoPE, a method that assigns learnable rotation frequencies and attention scaling factors to each attention head in Transformers, outperforming standard Rotary Posi…
Thinking Machines released Inkling, a 975B parameter mixture-of-experts model with 41B active parameters and a 1M token context length, which does not use Rotary Position Embeddings (RoPE). Instead, I…
Rotary Position Embedding (RoPE), introduced by Su et al. in 2021, solves the Transformer long-context problem by rotating Query and Key vectors in 2D sub-planes, making attention depend only on relat…
A new study analyzing seven pretrained models reveals that the positional encoding scheme in transformers determines the spectral properties of attention heads, with RoPE inducing rotational spectra a…
Researchers introduced LPES, a layer-specific positional embedding scaling method that mitigates the 'lost-in-the-middle' problem in LLMs by assigning distinct scaling factors to each layer, achieving…
A developer explains the importance of positional embeddings in Transformers, comparing Absolute Positional Embedding (APE), Relative Positional Embedding (RPE), and Rotary Positional Embedding (RoPE)…
A technical analysis reveals that large language models fail at long loop tasks due to attention dilution, EOS bias, and stateless architecture, causing protocol drift where constraints are rewritten …
A new appendix maps structural parallels between DNA and large language models, identifying a shared developmental-ledger pattern of possibility, gate, commitment, ledger, inheritance, development, an…
Researchers developed XBAR, an auditable latent crossbar memory fabric that enables model-to-model communication by writing directly into a frozen transformer's KV cache. The system achieves O(1) VRAM…
A new arXiv preprint (2606.18164) by Niklas Forner and coauthors analyzes how transformers learn chain-of-thought state tracking in a solvable setting, training a simplified one-block transformer on p…
A technical analysis calculates the dollar cost per user for serving large language models at scale using napkin math, breaking down GPU resources, matrix multiplication costs, and attention mechanism…