By Latent AI Labs (@LatentAILabs)
In our first article (YouTube video), our AI confidently invented a fictional “Kesharpur Tower in Bihar.” In our second article (YouTube video), we measured how thinking models catch their own mistakes by writing a scratchpad — and saw that every single token an AI reads or writes adds 144 kilobytes to its working memory.
So for Video #3, we ran a simple experiment on a 96 GB NVIDIA RTX PRO 6000 Blackwell GPU using the open-weights Qwen3–4B model (36 layers, 32 query heads, 8 key-value heads):
We hid a single 10-word sentence — “The secret passcode to the Kesharpur Vault is 7492.” — dead center at token #16,357 inside a 32,768-token document (~25,000 words, the length of a short book) filled with dense technical chapters on astronomy, oceanography, thermodynamics, and architecture.
We tested 5 different needle depths (5%, 25%, 50%, 75%, 95%) across 4 context lengths (2,048, 8,192, 16,384, and 32,768 tokens). 20 out of 20 times, in 2.04 seconds, the model answered 7492 with 100.0% confidence.
Then we opened all 36 floors of the Transformer — 1,152 attention heads in total — to see who actually read the book.
Almost none of them did:
Here is the complete architectural breakdown of how modern AI reads a whole book at once — and why 2025–2026 architectures (Google Gemma 3, Alibaba Qwen3-Next, DeepSeek-V3/R1) are tearing down the original 2017 Transformer tower.
In standard self-attention, every token at position t sends out a Query vector and computes a dot product against the Key badge of every earlier token 1 … t. That forms a lower-triangular grid of N(N + 1) / 2 causal comparisons per attention head.
Here is what we measured on our GPU as we scaled the book length by 64× from 512 tokens to 32,768 tokens (m2_scaling):
Context (N) │ Causal Pairs/Head │ Total Pairs (1,152 Heads) │ Naive Matrix (bf16) │ Peak VRAM Delta │ Prefill Time────────────┼───────────────────┼───────────────────────────┼─────────────────────┼─────────────────┼───────────── 512 │ 131,328 │ 151.3 Million │ 0.56 GB │ 91.1 MB │ 35.1 ms 1,024 │ 524,800 │ 604.6 Million │ 2.25 GB │ 182.3 MB │ 53.1 ms 2,048 │ 2,098,176 │ 2.42 Billion │ 9.00 GB │ 364.5 MB │ 78.4 ms 4,096 │ 8,390,656 │ 9.67 Billion │ 36.00 GB │ 729.0 MB │ 158.1 ms 8,192 │ 33,558,528 │ 38.66 Billion │ 144.00 GB │ 1,722.2 MB │ 341.0 ms 16,384 │ 134,225,920 │ 154.63 Billion │ 576.00 GB │ 5,253.2 MB │ 772.9 ms 32,768 │ 536,887,296 │ 618.49 Billion │ 2,304.00 GB (2.3T)│ 14,264 MB(14.3G)│ 2,035.8ms(2s)
When the prompt grows by 64×, the attention work grows by 64 × 64 = 4,096×. If the GPU actually wrote out those 32,768 × 32,768 attention score tables in main memory at 2 bytes per number (bfloat16), a single floor would require 64 GB, and all 36 floors would demand 2,304 GB (2.3 Terabytes) of VRAM.
Yet on our GPU, reading the entire 32,768-token book only added 14.26 GB of peak working memory and finished in 2.04 seconds.
What saved it is FlashAttention (Tri Dao et al., arXiv:2205.14135):
FlashAttention solves reading the prompt (prefill). But as soon as the model starts generating tokens one by one, a second memory bottleneck appears: the KV-Cache.
Every token already read leaves behind two vectors on every floor: its Key badge and its Value content:
KV Bytes / Token = 2 × n_layers × n_kv_heads × d_head × 2 bytes
Because a Transformer reads all 32,768 words in parallel, plain self-attention is a bag of words. Yet when we asked Qwen3-4B:
How does the attention dot product know word order across 32,000 tokens?
Modern models use Rotary Position Embeddings (RoPE) (Jianlin Su et al., arXiv:2104.09864). Before a query and key take a dot product, RoPE groups each head’s 128 numbers into 64 pairs of 2D coordinates and rotates pair i at token position m by angle m · θ_i, where:
θ_i = base^(−2i / d), where base = rope_theta = 1,000,000
Because the dot product of two rotated 2D vectors depends only on the difference between their angles, absolute positions cancel out: ⟨R_m q, R_n k⟩ is a function purely of the relative distance (m − n)!
We measured the exact rotation speeds of all 64 clock hands inside Qwen3-4B (m4_rope):
Clock Hand (2D Pair) │ Head Dims │ Speed (°/token) │ 360° Wavelength │ Role──────────────────────────────────┼───────────┼──────────────────┼───────────────────┼───────────────────────────────────Pair #0 (Fastest Second Hand) │ [0, 1] │ 57.30° / token │ 6.28 tokens │ Immediate neighbor syntax (1..6)Pair #16 │ [32, 33] │ 0.637° / token │ 565.5 tokens │ Paragraph-level distancePair #31 (Middle Minute Hand) │ [62, 63] │ 0.0711° / token │ 5,063.3 tokens │ Chapter-level distancePair #48 │ [96, 97] │ 0.0040° / token │ 90,009 tokens │ Multi-chapter distancePair #63 (Slowest Hour Hand) │ [126, 127]│ 0.000071° / token│ 5,063,256 tokens │ Book-scale monotonic order
Early open models like LLaMA-1 used rope_theta = 10,000, so their slowest hands spun 100× faster and wrapped into unseen angles past 4,096 tokens. When we forced Qwen3-4B's position clock to wrap mod 512 on an 8,192-token book, its probability of retrieving "7492" collapsed from 100.0% to 0.0%. Combined with YaRN (Bowen Peng et al., arXiv:2309.00071) — which leaves the fast high-frequency hands untouched while interpolating the slow low-frequency hands — Qwen3 stretches its context window from 32,768 to 131,072 tokens.
When we inspected the attention matrix of all 1,152 heads (36 layers × 32 heads) while answering the vault question on a 2,048-token book (m5_attention_sinks_and_retrieval_heads), we uncovered a dramatic division of labor:
On Floor 8, the average attention head puts 76.9% of its focus on the first 4 tokens (<|im_start|>user\nA). On Floor 29, it is 72.0%. Across the entire model, 735 out of 1,152 heads (63.8%) put over 80% of their attention on just the first 4 tokens plus the last 64 local tokens.
Why? As Guangxuan Xiao et al. (StreamingLLM, arXiv:2309.17453) proved, Softmax must sum to 100%. When a head has nothing useful to match in the middle of the book, it needs a neutral “no-op” parking lot to dump its unused probability mass — and because Token #0 is visible to every token in causal attention, the model turns the opening tokens into an Attention Sink.
We verified this directly on our GPU (sink_streaming_test.json):
While two-thirds of the heads park on the sink, who found "7492" at token #997? A tiny squad of Retrieval Heads (Wenhao Wu et al., arXiv:2404.15558):
When we performed surgical GPU ablation (extra_results.json → 2_matched_layer_ablation):
If two-thirds of heads only ever look at the first 4 sink tokens and the last few dozen local words — and only a handful of upper floors scan the whole book — why pay to store a 32,768-token KV-cache on all 36 floors?
That insight drives the new hybrid tower used in 2025–2026 frontier models:
Watch the full 20-minute animated GPU breakdown on YouTube*, and check out* Part 1: How ChatGPT Actually Works (YouTube) and Part 2: Why AI Says “Wait” (YouTube).
How AI Reads a Whole Book at Once — Long Context, KV-Cache & The New Tower (Every Number Measured) was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.