How AI Reads a Whole Book at Once — Long Context, KV-Cache & The New Tower (Every Number Measured)

A Latent AI Labs experiment on a 96 GB NVIDIA RTX PRO 6000 Blackwell GPU running the open-weights Qwen3-4B model (36 layers, 32 query heads, 8 key-value heads) retrieved a 10-word passcode hidden at token #16,357 of a 32,768-token document in 20 out of 20 trials across 5 needle depths and 4 context lengths, answering "7492" in 2.04 seconds with 100.0% confidence. The lab measured that scaling context 64× from 512 to 32,768 tokens grows attention work 4,096×, which would require 2,304 GB of VRAM if the 32,768 × 32,768 attention score tables were written out in bfloat16, yet peak working memory rose only 14.26 GB because FlashAttention (Tri Dao et al., arXiv:2205.14135) avoids materializing them. The lab frames the results as evidence that 2025–2026 architectures including Google Gemma 3, Alibaba Qwen3-Next and DeepSeek-V3/R1 are replacing the original 2017 Transformer design.

By Latent AI Labs @LatentAILabs In our first article https://arunp0.medium.com/strawberry-has-2-rs-how-chatgpt-actually-works-under-the-hood-every-number-measured-0c6499992573 YouTube video https://youtu.be/7HBtSz3wSSM , our AI confidently invented a fictional “Kesharpur Tower in Bihar.” In our second article https://arunp0.medium.com/why-ai-says-wait-how-thinking-models-actually-work-every-step-measured-b1baff6c592d YouTube video https://youtu.be/XXFt9s6cJAg , we measured how thinking models catch their own mistakes by writing a scratchpad — and saw that every single token an AI reads or writes adds 144 kilobytes to its working memory. So for Video 3, we ran a simple experiment on a 96 GB NVIDIA RTX PRO 6000 Blackwell GPU using the open-weights Qwen3–4B model 36 layers, 32 query heads, 8 key-value heads : We hid a single 10-word sentence — “The secret passcode to the Kesharpur Vault is 7492.” — dead center at token 16,357 inside a 32,768-token document ~25,000 words, the length of a short book filled with dense technical chapters on astronomy, oceanography, thermodynamics, and architecture. We tested 5 different needle depths 5%, 25%, 50%, 75%, 95% across 4 context lengths 2,048, 8,192, 16,384, and 32,768 tokens . 20 out of 20 times , in 2.04 seconds , the model answered 7492 with 100.0% confidence . Then we opened all 36 floors of the Transformer — 1,152 attention heads in total — to see who actually read the book. Almost none of them did: Here is the complete architectural breakdown of how modern AI reads a whole book at once — and why 2025–2026 architectures Google Gemma 3 , Alibaba Qwen3-Next , DeepSeek-V3/R1 are tearing down the original 2017 Transformer tower. In standard self-attention, every token at position t sends out a Query vector and computes a dot product against the Key badge of every earlier token 1 … t. That forms a lower-triangular grid of N N + 1 / 2 causal comparisons per attention head. Here is what we measured on our GPU as we scaled the book length by 64× from 512 tokens to 32,768 tokens m2 scaling : Context N │ Causal Pairs/Head │ Total Pairs 1,152 Heads │ Naive Matrix bf16 │ Peak VRAM Delta │ Prefill Time────────────┼───────────────────┼───────────────────────────┼─────────────────────┼─────────────────┼───────────── 512 │ 131,328 │ 151.3 Million │ 0.56 GB │ 91.1 MB │ 35.1 ms 1,024 │ 524,800 │ 604.6 Million │ 2.25 GB │ 182.3 MB │ 53.1 ms 2,048 │ 2,098,176 │ 2.42 Billion │ 9.00 GB │ 364.5 MB │ 78.4 ms 4,096 │ 8,390,656 │ 9.67 Billion │ 36.00 GB │ 729.0 MB │ 158.1 ms 8,192 │ 33,558,528 │ 38.66 Billion │ 144.00 GB │ 1,722.2 MB │ 341.0 ms 16,384 │ 134,225,920 │ 154.63 Billion │ 576.00 GB │ 5,253.2 MB │ 772.9 ms 32,768 │ 536,887,296 │ 618.49 Billion │ 2,304.00 GB 2.3T │ 14,264 MB 14.3G │ 2,035.8ms 2s When the prompt grows by 64× , the attention work grows by 64 × 64 = 4,096× . If the GPU actually wrote out those 32,768 × 32,768 attention score tables in main memory at 2 bytes per number bfloat16 , a single floor would require 64 GB , and all 36 floors would demand 2,304 GB 2.3 Terabytes of VRAM. Yet on our GPU, reading the entire 32,768-token book only added 14.26 GB of peak working memory and finished in 2.04 seconds . What saved it is FlashAttention Tri Dao et al., arXiv:2205.14135 : FlashAttention solves reading the prompt prefill . But as soon as the model starts generating tokens one by one, a second memory bottleneck appears: the KV-Cache . Every token already read leaves behind two vectors on every floor: its Key badge and its Value content: KV Bytes / Token = 2 × n layers × n kv heads × d head × 2 bytes Because a Transformer reads all 32,768 words in parallel, plain self-attention is a bag of words. Yet when we asked Qwen3-4B: How does the attention dot product know word order across 32,000 tokens? Modern models use Rotary Position Embeddings RoPE Jianlin Su et al., arXiv:2104.09864 . Before a query and key take a dot product, RoPE groups each head’s 128 numbers into 64 pairs of 2D coordinates and rotates pair i at token position m by angle m · θ i, where: θ i = base^ −2i / d , where base = rope theta = 1,000,000 Because the dot product of two rotated 2D vectors depends only on the difference between their angles, absolute positions cancel out: ⟨R m q, R n k⟩ is a function purely of the relative distance m − n We measured the exact rotation speeds of all 64 clock hands inside Qwen3-4B m4 rope : Clock Hand 2D Pair │ Head Dims │ Speed °/token │ 360° Wavelength │ Role──────────────────────────────────┼───────────┼──────────────────┼───────────────────┼───────────────────────────────────Pair 0 Fastest Second Hand │ 0, 1 │ 57.30° / token │ 6.28 tokens │ Immediate neighbor syntax 1..6 Pair 16 │ 32, 33 │ 0.637° / token │ 565.5 tokens │ Paragraph-level distancePair 31 Middle Minute Hand │ 62, 63 │ 0.0711° / token │ 5,063.3 tokens │ Chapter-level distancePair 48 │ 96, 97 │ 0.0040° / token │ 90,009 tokens │ Multi-chapter distancePair 63 Slowest Hour Hand │ 126, 127 │ 0.000071° / token│ 5,063,256 tokens │ Book-scale monotonic order Early open models like LLaMA-1 used rope theta = 10,000, so their slowest hands spun 100× faster and wrapped into unseen angles past 4,096 tokens. When we forced Qwen3-4B's position clock to wrap mod 512 on an 8,192-token book, its probability of retrieving "7492" collapsed from 100.0% to 0.0% . Combined with YaRN Bowen Peng et al., arXiv:2309.00071 — which leaves the fast high-frequency hands untouched while interpolating the slow low-frequency hands — Qwen3 stretches its context window from 32,768 to 131,072 tokens . When we inspected the attention matrix of all 1,152 heads 36 layers × 32 heads while answering the vault question on a 2,048-token book m5 attention sinks and retrieval heads , we uncovered a dramatic division of labor: On Floor 8, the average attention head puts 76.9% of its focus on the first 4 tokens <|im start| user\nA . On Floor 29, it is 72.0% . Across the entire model, 735 out of 1,152 heads 63.8% put over 80% of their attention on just the first 4 tokens plus the last 64 local tokens. Why? As Guangxuan Xiao et al. StreamingLLM, arXiv:2309.17453 proved, Softmax must sum to 100% . When a head has nothing useful to match in the middle of the book, it needs a neutral “no-op” parking lot to dump its unused probability mass — and because Token 0 is visible to every token in causal attention, the model turns the opening tokens into an Attention Sink . We verified this directly on our GPU sink streaming test.json : While two-thirds of the heads park on the sink, who found "7492" at token 997? A tiny squad of Retrieval Heads Wenhao Wu et al., arXiv:2404.15558 : When we performed surgical GPU ablation extra results.json → 2 matched layer ablation : If two-thirds of heads only ever look at the first 4 sink tokens and the last few dozen local words — and only a handful of upper floors scan the whole book — why pay to store a 32,768-token KV-cache on all 36 floors ? That insight drives the new hybrid tower used in 2025–2026 frontier models: Watch the full 20-minute animated GPU breakdown on YouTube , and check out Part 1: How ChatGPT Actually Works YouTube and Part 2: Why AI Says “Wait” YouTube . How AI Reads a Whole Book at Once — Long Context, KV-Cache & The New Tower Every Number Measured https://pub.towardsai.net/how-ai-reads-a-whole-book-at-once-long-context-kv-cache-the-new-tower-every-number-measured-3c92dcc8d451 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.