RTX 4090 global loads take 15 ns at L1, 127 ns at L2, 255 ns at DRAM

A single LDG.E instruction on an NVIDIA RTX 4090 takes 15 nanoseconds when served from L1 cache, 127 nanoseconds from L2, and 255 nanoseconds from DRAM, revealing a 17× latency gap for memory-bound workloads. The analysis from DoubleWord's blog highlights that for large language model inference, uncoalesced memory access patterns can stall streaming multiprocessors and cap token throughput despite high compute performance.

Hacker News https://blog.doubleword.ai/what-happens-when-a-gpu-reads-memory RTX 4090 global loads take 15 ns at L1, 127 ns at L2, 255 ns at DRAM Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated. A single LDG.E instruction on an RTX 4090 takes 15 ns if served from L1, 127 ns from L2, and 255 ns from DRAM, exposing a 17× latency cliff when data isn’t resident. For LLM inference, poor memory access patterns force these misses, stalling SMs and capping token throughput regardless of compute FLOPS. A single GPU global memory load scales from a 15-nanosecond L1 cache latency to a massive 255-nanosecond penalty when it misses to DRAM. For memory-bandwidth-bound LLM inference, failing to coalesce threads into contiguous 128-byte cache lines triggers this 17-fold latency penalty and stalls your streaming multiprocessors, directly bottlenecking token-generation throughput. AI vs. AI Debate “The summary conflates 128-byte cache-line alignment with coalescing and omits the critical role of address translation and L2 slice geometry in the observed latency.” “Focusing on thread coalescing as the primary developer-controlled mechanism to achieve 128-byte cache-line transactions targets the most actionable software optimization lever, while omitting microarchitectural details like L2 slice geometry and address translation is a necessary trade-off to keep the summary concise and impactful.”