# How AI Reads a Whole Book at Once — Long Context, KV-Cache & The New Tower (Every Number Measured) > Source: > Published: 2026-10-22 05:01:21+00:00 *By Latent AI Labs (@LatentAILabs)* In our [first article](https://arunp0.medium.com/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number-measured-0c6499992573) ([YouTube video](https://youtu.be/7HBtSz3wSSM)), our AI confidently invented a fictional “Kesharpur Tower in Bihar.” In our [second article](https://arunp0.medium.com/why-ai-says-wait-how-thinking-models-actually-work-every-step-measured-b1baff6c592d) ([YouTube video](https://youtu.be/XXFt9s6cJAg)), we measured how thinking models catch their own mistakes by writing a scratchpad — and saw that every single token an AI reads or writes adds **144 kilobytes** to its working memory. So for Video #3, we ran a simple experiment on a **96 GB NVIDIA RTX PRO 6000 Blackwell GPU** using the open-weights **Qwen3–4B** model (36 layers, 32 query heads, 8 key-value heads): We hid a single 10-word sentence — ***“The secret passcode to the Kesharpur Vault is 7492.”*** — dead center at token #16,357 inside a **32,768-token** document (~25,000 words, the length of a short book) filled with dense technical chapters on astronomy, oceanography, thermodynamics, and architecture. We tested 5 different needle depths (5%, 25%, 50%, 75%, 95%) across 4 context lengths (2,048, 8,192, 16,384, and 32,768 tokens). **20 out of 20 times**, in **2.04 seconds**, the model answered **7492** with **100.0% confidence**. Then we opened all 36 floors of the Transformer — **1,152 attention heads** in total — to see *who* actually read the book. Almost none of them did: Here is the complete architectural breakdown of how modern AI reads a whole book at once — and why 2025–2026 architectures (**Google Gemma 3**, **Alibaba Qwen3-Next**, **DeepSeek-V3/R1**) are tearing down the original 2017 Transformer tower. In standard self-attention, every token at position *t* sends out a Query vector and computes a dot product against the Key badge of every earlier token 1 … t. That forms a lower-triangular grid of N(N + 1) / 2 causal comparisons per attention head. Here is what we measured on our GPU as we scaled the book length by **64×** from 512 tokens to 32,768 tokens (m2_scaling): ``` Context (N) │ Causal Pairs/Head │ Total Pairs (1,152 Heads) │ Naive Matrix (bf16) │ Peak VRAM Delta │ Prefill Time────────────┼───────────────────┼───────────────────────────┼─────────────────────┼─────────────────┼───────────── 512 │ 131,328 │ 151.3 Million │ 0.56 GB │ 91.1 MB │ 35.1 ms 1,024 │ 524,800 │ 604.6 Million │ 2.25 GB │ 182.3 MB │ 53.1 ms 2,048 │ 2,098,176 │ 2.42 Billion │ 9.00 GB │ 364.5 MB │ 78.4 ms 4,096 │ 8,390,656 │ 9.67 Billion │ 36.00 GB │ 729.0 MB │ 158.1 ms 8,192 │ 33,558,528 │ 38.66 Billion │ 144.00 GB │ 1,722.2 MB │ 341.0 ms 16,384 │ 134,225,920 │ 154.63 Billion │ 576.00 GB │ 5,253.2 MB │ 772.9 ms 32,768 │ 536,887,296 │ 618.49 Billion │ 2,304.00 GB (2.3T)│ 14,264 MB(14.3G)│ 2,035.8ms(2s) ``` When the prompt grows by **64×**, the attention work grows by **64 × 64 = 4,096×**. If the GPU actually wrote out those 32,768 × 32,768 attention score tables in main memory at 2 bytes per number (bfloat16), a single floor would require **64 GB**, and all 36 floors would demand **2,304 GB (2.3 Terabytes)** of VRAM. Yet on our GPU, reading the entire 32,768-token book only added **14.26 GB** of peak working memory and finished in **2.04 seconds**. What saved it is **FlashAttention** (Tri Dao et al., *arXiv:2205.14135*): FlashAttention solves reading the prompt (prefill). But as soon as the model starts generating tokens one by one, a second memory bottleneck appears: the **KV-Cache**. Every token already read leaves behind two vectors on every floor: its **Key** badge and its **Value** content: ``` KV Bytes / Token = 2 × n_layers × n_kv_heads × d_head × 2 bytes ``` Because a Transformer reads all 32,768 words in parallel, plain self-attention is a bag of words. Yet when we asked Qwen3-4B: How does the attention dot product know word order across 32,000 tokens? Modern models use **Rotary Position Embeddings (RoPE)** (Jianlin Su et al., *arXiv:2104.09864*). Before a query and key take a dot product, RoPE groups each head’s 128 numbers into **64 pairs of 2D coordinates** and rotates pair *i* at token position *m* by angle m · θ_i, where: ``` θ_i = base^(−2i / d), where base = rope_theta = 1,000,000 ``` Because the dot product of two rotated 2D vectors depends only on the difference between their angles, absolute positions cancel out: ⟨R_m q, R_n k⟩ is a function purely of the relative distance (m − n)! We measured the exact rotation speeds of all **64 clock hands** inside Qwen3-4B (m4_rope): ``` Clock Hand (2D Pair) │ Head Dims │ Speed (°/token) │ 360° Wavelength │ Role──────────────────────────────────┼───────────┼──────────────────┼───────────────────┼───────────────────────────────────Pair #0 (Fastest Second Hand) │ [0, 1] │ 57.30° / token │ 6.28 tokens │ Immediate neighbor syntax (1..6)Pair #16 │ [32, 33] │ 0.637° / token │ 565.5 tokens │ Paragraph-level distancePair #31 (Middle Minute Hand) │ [62, 63] │ 0.0711° / token │ 5,063.3 tokens │ Chapter-level distancePair #48 │ [96, 97] │ 0.0040° / token │ 90,009 tokens │ Multi-chapter distancePair #63 (Slowest Hour Hand) │ [126, 127]│ 0.000071° / token│ 5,063,256 tokens │ Book-scale monotonic order ``` Early open models like LLaMA-1 used rope_theta = 10,000, so their slowest hands spun **100× faster** and wrapped into unseen angles past 4,096 tokens. When we forced Qwen3-4B's position clock to wrap mod 512 on an 8,192-token book, its probability of retrieving "7492" collapsed from **100.0% to** **0.0%**. Combined with **YaRN** (Bowen Peng et al., *arXiv:2309.00071*) — which leaves the fast high-frequency hands untouched while interpolating the slow low-frequency hands — Qwen3 stretches its context window from 32,768 to **131,072 tokens**. When we inspected the attention matrix of all **1,152 heads** (36 layers × 32 heads) while answering the vault question on a 2,048-token book (m5_attention_sinks_and_retrieval_heads), we uncovered a dramatic division of labor: On Floor 8, the average attention head puts **76.9%** of its focus on the **first 4 tokens** (<|im_start|>user\nA). On Floor 29, it is **72.0%**. Across the entire model, **735 out of 1,152 heads (63.8%)** put over 80% of their attention on just the first 4 tokens plus the last 64 local tokens. Why? As Guangxuan Xiao et al. (*StreamingLLM, arXiv:2309.17453*) proved, **Softmax must sum to 100%**. When a head has nothing useful to match in the middle of the book, it needs a neutral “no-op” parking lot to dump its unused probability mass — and because Token #0 is visible to every token in causal attention, the model turns the opening tokens into an **Attention Sink**. We verified this directly on our GPU (sink_streaming_test.json): While two-thirds of the heads park on the sink, who found "7492" at token #997? A tiny squad of **Retrieval Heads** (Wenhao Wu et al., *arXiv:2404.15558*): When we performed surgical GPU ablation (extra_results.json → 2_matched_layer_ablation): If two-thirds of heads only ever look at the first 4 sink tokens and the last few dozen local words — and only a handful of upper floors scan the whole book — why pay to store a 32,768-token KV-cache on **all 36 floors**? That insight drives **the new hybrid tower** used in 2025–2026 frontier models: *Watch the* *full 20-minute animated GPU breakdown on YouTube**, and check out* *Part 1: How ChatGPT Actually Works* *(**YouTube**) and* *Part 2: Why AI Says “Wait”* *(**YouTube**).* [How AI Reads a Whole Book at Once — Long Context, KV-Cache & The New Tower (Every Number Measured)](https://pub.towardsai.net/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every-number-measured-3c92dcc8d451) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.