{"slug": "how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every", "title": "How AI Reads a Whole Book at Once — Long Context, KV-Cache & The New Tower (Every Number Measured)", "summary": "A Latent AI Labs experiment on a 96 GB NVIDIA RTX PRO 6000 Blackwell GPU running the open-weights Qwen3-4B model (36 layers, 32 query heads, 8 key-value heads) retrieved a 10-word passcode hidden at token #16,357 of a 32,768-token document in 20 out of 20 trials across 5 needle depths and 4 context lengths, answering \"7492\" in 2.04 seconds with 100.0% confidence. The lab measured that scaling context 64× from 512 to 32,768 tokens grows attention work 4,096×, which would require 2,304 GB of VRAM if the 32,768 × 32,768 attention score tables were written out in bfloat16, yet peak working memory rose only 14.26 GB because FlashAttention (Tri Dao et al., arXiv:2205.14135) avoids materializing them. The lab frames the results as evidence that 2025–2026 architectures including Google Gemma 3, Alibaba Qwen3-Next and DeepSeek-V3/R1 are replacing the original 2017 Transformer design.", "body_md": "*By Latent AI Labs (@LatentAILabs)*\n\nIn our [first article](https://arunp0.medium.com/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number-measured-0c6499992573) ([YouTube video](https://youtu.be/7HBtSz3wSSM)), our AI confidently invented a fictional “Kesharpur Tower in Bihar.” In our [second article](https://arunp0.medium.com/why-ai-says-wait-how-thinking-models-actually-work-every-step-measured-b1baff6c592d) ([YouTube video](https://youtu.be/XXFt9s6cJAg)), we measured how thinking models catch their own mistakes by writing a scratchpad — and saw that every single token an AI reads or writes adds **144 kilobytes** to its working memory.\n\nSo for Video #3, we ran a simple experiment on a **96 GB NVIDIA RTX PRO 6000 Blackwell GPU** using the open-weights **Qwen3–4B** model (36 layers, 32 query heads, 8 key-value heads):\n\nWe hid a single 10-word sentence — ***“The secret passcode to the Kesharpur Vault is 7492.”*** — dead center at token #16,357 inside a **32,768-token** document (~25,000 words, the length of a short book) filled with dense technical chapters on astronomy, oceanography, thermodynamics, and architecture.\n\nWe tested 5 different needle depths (5%, 25%, 50%, 75%, 95%) across 4 context lengths (2,048, 8,192, 16,384, and 32,768 tokens). **20 out of 20 times**, in **2.04 seconds**, the model answered **7492** with **100.0% confidence**.\n\nThen we opened all 36 floors of the Transformer — **1,152 attention heads** in total — to see *who* actually read the book.\n\nAlmost none of them did:\n\nHere is the complete architectural breakdown of how modern AI reads a whole book at once — and why 2025–2026 architectures (**Google Gemma 3**, **Alibaba Qwen3-Next**, **DeepSeek-V3/R1**) are tearing down the original 2017 Transformer tower.\n\nIn standard self-attention, every token at position *t* sends out a Query vector and computes a dot product against the Key badge of every earlier token 1 … t. That forms a lower-triangular grid of N(N + 1) / 2 causal comparisons per attention head.\n\nHere is what we measured on our GPU as we scaled the book length by **64×** from 512 tokens to 32,768 tokens (m2_scaling):\n\n```\nContext (N) │ Causal Pairs/Head │ Total Pairs (1,152 Heads) │ Naive Matrix (bf16) │ Peak VRAM Delta │ Prefill Time────────────┼───────────────────┼───────────────────────────┼─────────────────────┼─────────────────┼─────────────        512 │           131,328 │            151.3 Million  │             0.56 GB │         91.1 MB │      35.1 ms      1,024 │           524,800 │            604.6 Million  │             2.25 GB │        182.3 MB │      53.1 ms      2,048 │         2,098,176 │             2.42 Billion  │             9.00 GB │        364.5 MB │      78.4 ms      4,096 │         8,390,656 │             9.67 Billion  │            36.00 GB │        729.0 MB │     158.1 ms      8,192 │        33,558,528 │            38.66 Billion  │           144.00 GB │      1,722.2 MB │     341.0 ms     16,384 │       134,225,920 │           154.63 Billion  │           576.00 GB │      5,253.2 MB │     772.9 ms     32,768 │       536,887,296 │           618.49 Billion  │   2,304.00 GB (2.3T)│ 14,264 MB(14.3G)│ 2,035.8ms(2s)\n```\n\nWhen the prompt grows by **64×**, the attention work grows by **64 × 64 = 4,096×**. If the GPU actually wrote out those 32,768 × 32,768 attention score tables in main memory at 2 bytes per number (bfloat16), a single floor would require **64 GB**, and all 36 floors would demand **2,304 GB (2.3 Terabytes)** of VRAM.\n\nYet on our GPU, reading the entire 32,768-token book only added **14.26 GB** of peak working memory and finished in **2.04 seconds**.\n\nWhat saved it is **FlashAttention** (Tri Dao et al., *arXiv:2205.14135*):\n\nFlashAttention solves reading the prompt (prefill). But as soon as the model starts generating tokens one by one, a second memory bottleneck appears: the **KV-Cache**.\n\nEvery token already read leaves behind two vectors on every floor: its **Key** badge and its **Value** content:\n\n```\nKV Bytes / Token = 2 × n_layers × n_kv_heads × d_head × 2 bytes\n```\n\nBecause a Transformer reads all 32,768 words in parallel, plain self-attention is a bag of words. Yet when we asked Qwen3-4B:\n\nHow does the attention dot product know word order across 32,000 tokens?\n\nModern models use **Rotary Position Embeddings (RoPE)** (Jianlin Su et al., *arXiv:2104.09864*). Before a query and key take a dot product, RoPE groups each head’s 128 numbers into **64 pairs of 2D coordinates** and rotates pair *i* at token position *m* by angle m · θ_i, where:\n\n```\nθ_i = base^(−2i / d),    where base = rope_theta = 1,000,000\n```\n\nBecause the dot product of two rotated 2D vectors depends only on the difference between their angles, absolute positions cancel out: ⟨R_m q, R_n k⟩ is a function purely of the relative distance (m − n)!\n\nWe measured the exact rotation speeds of all **64 clock hands** inside Qwen3-4B (m4_rope):\n\n```\nClock Hand (2D Pair)              │ Head Dims │ Speed (°/token)  │ 360° Wavelength   │ Role──────────────────────────────────┼───────────┼──────────────────┼───────────────────┼───────────────────────────────────Pair #0  (Fastest Second Hand)    │ [0, 1]    │ 57.30° / token   │ 6.28 tokens       │ Immediate neighbor syntax (1..6)Pair #16                          │ [32, 33]  │ 0.637° / token   │ 565.5 tokens      │ Paragraph-level distancePair #31 (Middle Minute Hand)     │ [62, 63]  │ 0.0711° / token  │ 5,063.3 tokens    │ Chapter-level distancePair #48                          │ [96, 97]  │ 0.0040° / token  │ 90,009 tokens     │ Multi-chapter distancePair #63 (Slowest Hour Hand)      │ [126, 127]│ 0.000071° / token│ 5,063,256 tokens  │ Book-scale monotonic order\n```\n\nEarly open models like LLaMA-1 used rope_theta = 10,000, so their slowest hands spun **100× faster** and wrapped into unseen angles past 4,096 tokens. When we forced Qwen3-4B's position clock to wrap mod 512 on an 8,192-token book, its probability of retrieving \"7492\" collapsed from **100.0% to** **0.0%**. Combined with **YaRN** (Bowen Peng et al., *arXiv:2309.00071*) — which leaves the fast high-frequency hands untouched while interpolating the slow low-frequency hands — Qwen3 stretches its context window from 32,768 to **131,072 tokens**.\n\nWhen we inspected the attention matrix of all **1,152 heads** (36 layers × 32 heads) while answering the vault question on a 2,048-token book (m5_attention_sinks_and_retrieval_heads), we uncovered a dramatic division of labor:\n\nOn Floor 8, the average attention head puts **76.9%** of its focus on the **first 4 tokens** (<|im_start|>user\\nA). On Floor 29, it is **72.0%**. Across the entire model, **735 out of 1,152 heads (63.8%)** put over 80% of their attention on just the first 4 tokens plus the last 64 local tokens.\n\nWhy? As Guangxuan Xiao et al. (*StreamingLLM, arXiv:2309.17453*) proved, **Softmax must sum to 100%**. When a head has nothing useful to match in the middle of the book, it needs a neutral “no-op” parking lot to dump its unused probability mass — and because Token #0 is visible to every token in causal attention, the model turns the opening tokens into an **Attention Sink**.\n\nWe verified this directly on our GPU (sink_streaming_test.json):\n\nWhile two-thirds of the heads park on the sink, who found \"7492\" at token #997? A tiny squad of **Retrieval Heads** (Wenhao Wu et al., *arXiv:2404.15558*):\n\nWhen we performed surgical GPU ablation (extra_results.json → 2_matched_layer_ablation):\n\nIf two-thirds of heads only ever look at the first 4 sink tokens and the last few dozen local words — and only a handful of upper floors scan the whole book — why pay to store a 32,768-token KV-cache on **all 36 floors**?\n\nThat insight drives **the new hybrid tower** used in 2025–2026 frontier models:\n\n*Watch the* *full 20-minute animated GPU breakdown on YouTube**, and check out* *Part 1: How ChatGPT Actually Works* *(**YouTube**) and* *Part 2: Why AI Says “Wait”* *(**YouTube**).*\n\n[How AI Reads a Whole Book at Once — Long Context, KV-Cache & The New Tower (Every Number Measured)](https://pub.towardsai.net/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every-number-measured-3c92dcc8d451) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every", "canonical_source": "https://pub.towardsai.net/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every-number-measured-3c92dcc8d451?source=rss----98111c9905da---4", "published_at": "2026-10-22 05:01:21+00:00", "updated_at": "2026-10-22 05:10:10.170513+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research", "machine-learning"], "entities": ["Latent AI Labs", "NVIDIA RTX PRO 6000 Blackwell", "Qwen3-4B", "FlashAttention", "Tri Dao", "Google Gemma 3", "Alibaba Qwen3-Next", "DeepSeek-V3/R1"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every", "markdown": "https://wpnews.pro/news/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every.md", "text": "https://wpnews.pro/news/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every.txt", "jsonld": "https://wpnews.pro/news/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every.jsonld"}}