{"slug": "rope-and-nope-what-sebastian-raschka-s-2026-positional-embedding-guide-means-for", "title": "RoPE and NoPE: What Sebastian Raschka's 2026 Positional Embedding Guide Means for LLMs", "summary": "Sebastian Raschka published a long-form tutorial on November 15, 2026 covering how modern LLMs encode token positions, spanning GPT-2's learned absolute tables, full and partial RoPE, NoPE layers, and learned relative bias, with a runnable PyTorch RoPE implementation. The guide reports that RoPE-everywhere is declining in 2026 open models as hybrids rise, citing SmolLM3's NoPE pattern (every 4th block) and MiniMax-M3's partial RoPE, which rotates 64 of each 128-dim head and saves roughly 128 MiB of BF16 cosine/sine tables at a 1M-token context while KV cache still stores full keys and values. Raschka's actionable point for builders is consistency: inference engines must apply the same pairing convention, base frequency and scaling rules the checkpoint trained with, including when stretching context with YaRN.", "body_md": "**On November 15, 2026, Sebastian Raschka published a long-form tutorial on how modern LLMs represent token positions — from GPT-2's learned tables through RoPE math to NoPE-only hybrids and Inkling's learned relative bias.** If you build with open weights, fine-tune, or debug long-context failures, this is the missing bridge between our [transformer architecture primer](https://explainx.ai/blog/what-is-transformer-architecture-attention-llm-guide-2026) and per-model cards that only say \"uses RoPE\" in a bullet.\n\nRaschka's piece is intentionally dense — figures, equations, and a runnable PyTorch RoPE implementation. This explainx.ai summary extracts **what changed in 2026 open models** and **what practitioners should verify**, while pointing you to the [original article](https://magazine.sebastianraschka.com/p/understanding-positional-embeddings?ref=explainx) for proofs and code.\n\n## TL;DR: schemes and when they matter\n\n| Scheme | Where position enters | Typical models | Builder takeaway | \n|---|---|---|---|\n| Learned absolute | Added to token embeddings pre-block | GPT-2 lineage | Poor length extrapolation beyond train context | \n| RoPE (full) | Rotation on Q/K in attention | Most open LLMs since LLaMA | Default; watch `rope_theta` / scaling at long context | \n| Partial RoPE | Rotate subset of head dims | MiniMax-M3 (64 of 128 dims) | Saves small table memory; mixed benchmark impact | \n| NoPE layers | No explicit encoding; causal mask only | SmolLM3 pattern (every 4th block) | Rising in 2026 gallery per Raschka | \n| Learned relative bias | Bias added to attention logits | Inkling, historical T5 | Different inductive bias for long gaps | \n\n## Why position encoding still confuses builders\n\nSelf-attention without position information treats token pairs as unordered bags. Swap \"red fox\" and \"gray fox\" and you change the story, but not the raw ID embeddings for repeated words like \"fox.\"\n\n[Raschka's opening toy example](https://magazine.sebastianraschka.com/p/understanding-positional-embeddings?ref=explainx) walks two sentences that differ only by word order. He notes causal masking already gives **some** order signal — a token sees only prior tokens — but **not** fine-grained distance: the mask cannot distinguish one step from ten steps apart. That gap is why explicit schemes remain common even in autoregressive decoders.\n\nOur earlier [transformer guide](https://explainx.ai/blog/what-is-transformer-architecture-attention-llm-guide-2026) summarized learned embeddings versus RoPE at a high level. Raschka's November piece finally supplies **mechanistic code** and a **2026 gallery bar chart** showing RoPE-everywhere declining as hybrids rise.\n\nWave above a ruler illustrating rotary positional encoding along a token sequence\n\n## RoPE in plain language\n\n**Rotary positional embedding (RoPE)** multiplies pairs of features in queries and keys by position-dependent rotations. Dot products then depend on **relative distance**, which is what you want when attention scores must know how far apart two tokens are.\n\nRaschka implements split-half pairing (modern default) versus interleaved pairing (original paper / early LLaMA). He shows with `torch.testing.assert_close` that layouts are equivalent when you reorder features consistently — important when you port weights between codebases.\n\nFor implementers, the actionable detail is **not** the trigonometry but **consistency**: inference engines must apply the same convention, base frequency, and scaling rules the checkpoint trained with. That is where [llama.cpp and ggml releases](https://explainx.ai/blog/llama-cpp-v0-9-0-ggml-0-29-0-fp8-e4m3-multi-node-tensor-parallelism-nccl-what-changed-2026) matter when you stretch context with YaRN or similar.\n\n## Partial RoPE and memory math\n\nMiniMax-M3 rotates **half** of each 128-dim head (64 rotary dims). Raschka calculates BF16 table savings on a **1M-token** context — on the order of **128 MiB** for cosine/sine tables — while noting KV cache still stores full keys and values.\n\nHe cites the **Fractional Rotation, Full Potential?** paper finding ~10% rotation can match full RoPE on some small-model trainings, but full RoPE wins on certain evals. His editorial verdict: partial RoPE is **often not worth the complexity** unless you are squeezing last MiB at megacontext.\n\n## NoPE layers and SmolLM3's pattern\n\nThe [NoPE paper](https://arxiv.org/abs/2305.19466?ref=explainx) showed causal attention alone can encode position via features that average differently as prefix length grows. Raschka reproduces the toy \"one hot The token\" example where uniform attention yields **1/prefix_length** statistics downstream layers can read.\n\nAt scale, results are mixed: tiny models generalize length on synthetic tasks; 1.3B code models drift at extreme lengths. In **2026 open weights**, Raschka highlights **SmolLM3 (3B)** applying RoPE in three layers and **skipping every fourth block** — authors reported improved long-context behavior.\n\nThat connects to explainx.ai's [Kimi K3 architecture notes](https://explainx.ai/blog/kimi-k3-architecture-raschka-latentmoe-nope-july-2026), where Raschka previously documented **NoPE everywhere** in a frontier open stack. The lesson: **do not assume RoPE** when you pick a fine-tune base — read the gallery card.\n\nAttention mechanism spotlight illustrating how RoPE changes query-key geometry in transformers\n\n## Inkling and learned relative bias\n\nBefore the gallery summary, Raschka covers **Inkling's learned relative position bias** — add a term to scaled Q·K based on hidden state and token distance — and links it to **T5's simpler bucketing**. [Thinking Machines' Inkling post](https://explainx.ai/blog/inkling-thinking-machines-open-weights-july-2026) on explainx.ai already noted relative embeddings instead of RoPE at the announcement layer; Raschka's section supplies the attention-diagram intuition.\n\n## What 2026 open models are doing (per gallery)\n\nSection 6.4 aggregates Raschka's **LLM Architecture Gallery** snapshots:\n\n- **RoPE in every softmax layer** remains the plurality but**share fell from 2025 into 2026** .\n- **Mixed RoPE + NoPE** blocks grew — still a minority but directionally interesting.\n- **NoPE-only** entries in the snapshot were**recurrent hybrids** without classic softmax everywhere.\n- **Other** includes architectures like**xLSTM** (no softmax attention) and**Inkling** (learned bias).\n\nFor builders choosing a base model in November 2026, treat the chart as **release-bias data**, not a performance ranking: galleries track **what shipped**, not **what won every benchmark**.\n\n## What people search after seeing \"RoPE explained\"\n\nSearch logs cluster on **\"rotary positional embeddings rope explained\"** and **\"positional embedding vs rope\"** — both ask the same practical question: *If I already know transformers, what do I change in code?*\n\nRaschka's answer: stop adding position vectors before block 0 unless you are debugging GPT-2; instead trace **where rotations hook into Q/K**, whether **partial dims** are frozen, and whether later **NoPE blocks** still inherit position from earlier rotations.\n\nSecondary searches on **\"llm meaning\"** are noise for this article — ignore them in implementation work.\n\n## Split-half versus interleaved RoPE in production code\n\nRaschka spends meaningful space on a detail most blogs skip: **original RoPE** interleaved feature pairs, while **most 2026 implementations** use split-half pairing because vectorized sine/cosine ops are simpler. Weight ports break when frameworks disagree on ordering — not when the math differs.\n\nIf you fine-tune with Hugging Face Transformers then deploy with a C++ runtime, verify the export script's `rope_scaling` JSON matches both sides. Our [local GLM tensor-parallel write-up](https://explainx.ai/blog/glm-5-3-two-dgx-stations-vllm-tensor-parallel-753b-nvfp4-607-tokens-per-second-guide-2026) is a reminder that deployment blogs must mention **implementation parity**, not just peak tokens/sec.\n\n## From GPT-2 learned vectors to rotation inside attention\n\nSection 2 of Raschka's article revisits **learned absolute** tables — the approach in GPT-2 and BERT. They are expressive within train context but famously weak at extrapolation. Section 3–5 build RoPE intuition with frequency bases, wavelength metaphors, and the PyTorch `rope_split_halves` function.\n\nReaders who only ever invoked `AutoModel.from_pretrained` still benefit: when a model card says **\"2048 train context, 128k inference,\"** the magic is usually RoPE scaling plus KV tricks, not a bigger embedding table.\n\n## Causal masking alone is not enough — but NoPE exploits it\n\nRaschka's NoPE discussion is careful not to overclaim. Causal masks provide **prefix statistics** exploitable by later layers — his uniform-attention toy model shows how a feature counting `The` yields **1/n** signals. Real models do not use uniform attention, but the ablations in the NoPE paper convinced authors that **some** layers can drop explicit encodings, especially when earlier layers already rotated Q/K.\n\nSmolLM3's **every fourth block** pattern is the pragmatic compromise: keep most layers familiar to tooling, experiment where long-context evals demanded it.\n\n## Gallery methodology caveat\n\nThe bar chart in Section 6.4 counts **architecture cards** in Raschka's gallery, not market share by download volume. A recurrent hybrid with NoPE-only softmax layers counts once even if its checkpoint is niche. Use the gallery to **audit your own assumptions**, not to declare RoPE \"dead.\"\n\n## How to use this when debugging long context\n\n| Symptom | Check | \n|---|---|\n| Garbage after 32k tokens | RoPE scaling / YaRN params vs model card | \n| Divergence after quantizing KV | Partial RoPE dims and cache layout | \n| Fine-tune collapse on order-sensitive task | Accidental NoPE block in base model | \n| Port weights between frameworks | Split-half vs interleaved pairing | \n\nPair those checks with hardware context from [local inference guides](https://explainx.ai/blog/airllm-run-70b-llm-4gb-gpu-inference-2026) only after the math matches — faster GPUs do not fix wrong `rope_theta`.\n\n## Related reading\n\n- [What is a transformer?](https://explainx.ai/blog/what-is-transformer-architecture-attention-llm-guide-2026) — attention foundations\n- [Kimi K3 architecture and NoPE](https://explainx.ai/blog/kimi-k3-architecture-raschka-latentmoe-nope-july-2026) — prior Raschka coverage\n- [Inkling open weights](https://explainx.ai/blog/inkling-thinking-machines-open-weights-july-2026) — relative bias in production model\n- [Vals AI eval transparency](https://explainx.ai/blog/vals-ai-cyberoffense-gpt-6-sol-daybreak-deepseek-v4-1-flash-open-weight-cyber-range-2026) — how labs publish benchmark context alongside architecture posts\n- [GLM local tensor parallel](https://explainx.ai/blog/glm-5-3-two-dgx-stations-vllm-tensor-parallel-753b-nvfp4-607-tokens-per-second-guide-2026) — when architecture details meet deployment\n\n**Primary source:** [Understanding Positional Embeddings in Modern LLMs — Sebastian Raschka, Nov 15, 2026](https://magazine.sebastianraschka.com/p/understanding-positional-embeddings?ref=explainx)\n\n## Classroom use and follow-on articles\n\nRaschka notes the piece is a **foundation for an upcoming LLM architecture trends article** — expect cross-links when that publishes. Educators can assign Section 1–5 for graduate labs and Section 6 for seminar discussion on inductive biases. Substack supporters get gallery image exports; open readers still benefit from the public figures embedded in the web view.\n\nIf you teach from [Build a Large Language Model (From Scratch)](https://github.com/rasbt/LLMs-from-scratch?ref=explainx), this article is the positional-encoding companion chapter the book's attention module points toward.\n\n*Gallery counts and model examples reflect Raschka's article as published November 15, 2026.*", "url": "https://wpnews.pro/news/rope-and-nope-what-sebastian-raschka-s-2026-positional-embedding-guide-means-for", "canonical_source": "https://explainx.ai/blog/rope-positional-embeddings-modern-llms-raschka-guide-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-15 00:00:00+00:00", "updated_at": "2026-11-15 15:23:33.860379+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "machine-learning", "ai-research"], "entities": ["Sebastian Raschka", "RoPE", "NoPE", "GPT-2", "LLaMA", "MiniMax-M3", "SmolLM3", "Inkling"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/rope-and-nope-what-sebastian-raschka-s-2026-positional-embedding-guide-means-for", "markdown": "https://wpnews.pro/news/rope-and-nope-what-sebastian-raschka-s-2026-positional-embedding-guide-means-for.md", "text": "https://wpnews.pro/news/rope-and-nope-what-sebastian-raschka-s-2026-positional-embedding-guide-means-for.txt", "jsonld": "https://wpnews.pro/news/rope-and-nope-what-sebastian-raschka-s-2026-positional-embedding-guide-means-for.jsonld"}}