{"slug": "route-the-work-not-just-the-data-gpus-cpus-and-the-rise-of-ai-native-storage", "title": "Route the Work, Not Just the Data: GPUs, CPUs, and the Rise of AI-Native Storage", "summary": "NVIDIA's B200 GPU has 180 GB of HBM3e memory with up to 8 TB/s bandwidth, while Micron's 245.76 TB 6600 ION SSD, shipping in May 2026, holds over a thousand times more data but with far lower bandwidth, creating a gap that drives the rise of AI-native storage. The article argues that LLM inference should route operations to the cheapest capable tier—GPU, CPU, or intelligent storage—to minimize data movement, a concept rooted in nearly three decades of research. This approach could reduce the need for expensive GPU memory as model state grows.", "body_md": "# 🧠 Route the Work, Not Just the Data: GPUs, CPUs, and the Rise of AI-Native Storage\n\nWhen we think about Large Language Models, we tend to picture GPUs.\n\nThat makes sense. Modern generative AI would not exist at its current scale without them.\n\nBut GPUs also expose one of AI's increasingly important architectural problems:\n\n**The fastest place to compute is not the cheapest place to keep data.**\n\nAn NVIDIA B200 GPU has 180 GB of HBM3e and can move data through that memory at up to roughly **8 TB/s**.\n\nAt the other end of the hierarchy, Micron began shipping its **245.76 TB 6600 ION SSD in May 2026**.\n\nOne drive can hold more than a thousand times as much data as a single B200's HBM.\n\nBut flash operates nowhere near HBM's bandwidth or latency.\n\nThat enormous gap between **hundreds of gigabytes of extraordinarily fast memory** and **hundreds of terabytes of comparatively inexpensive persistent storage** creates a fascinating architectural question:\n\n**What if storage stopped being merely the place where AI data waits for the GPU?**\n\nWhat if some AI workloads were processed near the storage itself, while an intelligent storage tier decided what actually needed to reach expensive GPU memory?\n\nThat may sound futuristic.\n\nIt actually follows a research path stretching back almost three decades.\n\nAnd LLMs may provide one of the strongest reasons yet to pursue it.\n\n**Here is the thesis of this article, stated plainly:**\n\n**An LLM request is not one monolithic computation. It is many kinds of work, and only some of it needs a GPU. As model state outgrows GPU memory, the winning architecture will route each operation to the cheapest tier that can perform it (GPU, CPU, or increasingly intelligent storage), and the cost that decides the route is data movement. The next major optimization is not making the GPU faster. It is reducing how much data has to reach it in the first place.**\n\nEverything that follows is the evidence: what already ships, what research demonstrates, what I measured on my own hardware, and what remains genuinely speculative.\n\n## ⚙️ First: What Is an LLM Actually Doing?\n\nAn LLM is not searching a giant database for a sentence matching your prompt.\n\nAt a simplified level, it repeatedly performs enormous amounts of numerical computation.\n\nYour text is divided into **tokens**.\n\nThose tokens are converted into numerical vectors and passed through many layers of a neural-network architecture called the **Transformer**.\n\nThe Transformer was introduced in the landmark 2017 paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762).\n\nTwo operations are especially important.\n\n### Attention\n\nAttention helps the model determine which previous tokens matter when interpreting the current token.\n\nA simplified version creates three representations:\n\n**Query:** What am I looking for?**Key:** What information do I represent?**Value:** What information should be passed forward?\n\nQueries are compared against keys.\n\nAttention scores are calculated.\n\nThose scores determine how strongly different values influence the next representation.\n\n### Feed-Forward Networks\n\nEach Transformer layer also contains large learned matrices that transform the token representations.\n\nAcross billions of parameters, this creates an enormous amount of multiplication and accumulation.\n\nEventually, the model produces a probability distribution over possible next tokens.\n\nIt selects a token according to the decoding strategy, appends it to the sequence, and performs the process again.\n\nAnd again.\n\nAnd again.\n\nThat repeated numerical workload helps explain why GPUs became so important.\n\n## 🧮 Why GPUs Are Better Than CPUs for LLMs\n\nCPUs are extraordinary general-purpose processors.\n\nThey are designed for workloads such as:\n\n- Operating systems\n- Application logic\n- Branching\n- Databases\n- Networking\n- Scheduling\n- Serial dependencies\n- Irregular computation\n- Many different instruction types\n\nA CPU's strength is flexibility.\n\nAn LLM workload is different.\n\nHuge portions of it repeatedly ask something closer to:\n\n**Can you multiply these enormous arrays of numbers as quickly and in as much parallelism as possible?**\n\nGPUs were built for parallelism.\n\nModern AI GPUs contain thousands of execution units plus specialized **Tensor Cores** designed for matrix operations using formats such as FP16, BF16, FP8 and increasingly lower-precision representations.\n\nThey also sit beside extraordinarily fast High Bandwidth Memory.\n\nNVIDIA lists a B200 at up to roughly **8 TB/s of HBM bandwidth per GPU**.\n\nA high-performance PCIe Gen5 SSD such as Micron's 9550 reaches roughly **14 GB/s of sequential read bandwidth**.\n\nThose are completely different performance classes.\n\nThere is no plausible architecture in which NAND flash simply becomes a drop-in substitute for GPU HBM.\n\nBut that is not the interesting question.\n\nThe interesting question is:\n\n**How much data could we prevent from needing to cross that boundary at all?**\n\n*The goal is not to make SSDs behave like GPUs. The goal is to stop sending the GPU work and data it does not need.*\n\n## 🧠 LLM Inference Has a Memory Problem\n\nInference generally contains two broad phases.\n\n### Prefill\n\nWhen you initially submit a prompt, many prompt tokens can be processed in parallel.\n\nThis stage can be highly compute-intensive and maps well to GPUs.\n\n### Decode\n\nThen generation begins.\n\nThe model produces one token.\n\nThen another.\n\nThen another.\n\nEach new token depends on information derived from what came before.\n\nThis autoregressive process means inference repeatedly accesses enormous model weights and an expanding amount of context state.\n\nDepending on model architecture, batch size, hardware and workload, the bottleneck can therefore shift away from raw arithmetic throughput toward **memory bandwidth and data movement**.\n\nThis is one reason [FlashAttention](https://arxiv.org/abs/2205.14135) became such an important contribution.\n\nFlashAttention does not make multiplication fundamentally faster.\n\nIt reorganizes attention specifically to reduce expensive movement between GPU HBM and faster on-chip SRAM.\n\nIts authors explicitly frame attention optimization as an **I/O-aware** problem.\n\nThat is an important lesson:\n\n**Even inside a GPU, moving data can become as important as computing it.**\n\nNow expand that problem outside the GPU.\n\n## 🗃️ The KV Cache: LLM Working Memory\n\nDuring attention, Transformers generate **key and value tensors** representing prior tokens.\n\nWithout retaining those tensors, the system would repeatedly recompute previous attention state every time another token was generated.\n\nInstead, inference engines store them in a **Key-Value cache**, usually shortened to **KV cache**.\n\nThat dramatically reduces redundant computation.\n\nBut the cache grows with:\n\n- Context length\n- Number of model layers\n- Model architecture\n- Number of simultaneous requests\n- Persistent agent histories\n- Long-running reasoning\n- Repeated document interactions\n\nThe scale becomes surprisingly large.\n\nNVIDIA has illustrated an example in which a **128K-token context for Llama 3 70B consumes roughly 40 GB of KV-cache memory for a single user at batch size 1**.\n\nForty gigabytes is not the model.\n\nIt is just the cached attention state associated with one long-context request in that example.\n\nMultiply long contexts across hundreds or thousands of concurrent requests, persistent agents, document workflows or reasoning processes and the problem becomes obvious:\n\n**HBM is incredibly fast, but it is scarce.**\n\n## 🪜 The Emerging AI Memory Hierarchy\n\nIncreasingly, AI infrastructure has to treat memory as a hierarchy:\n\n**GPU SRAM / cache**\n↓\n**GPU HBM**\n↓\n**CPU DRAM**\n↓\n**Local NVMe SSD**\n↓\n**Remote / network storage**\n\nEach step generally offers more capacity.\n\nEach step generally sacrifices latency and bandwidth.\n\nModern inference software is beginning to explicitly manage that hierarchy.\n\nNVIDIA Dynamo, for example, supports KV-cache offloading beyond GPU memory. Its architecture can spill KV-cache blocks into **CPU memory or local storage**, allowing larger contexts and reuse of previously computed prefixes.\n\nNVIDIA's FlexKV work extends this concept across tiers including **GPU, CPU and SSD-backed storage**.\n\nSo one part of this article is no longer speculative:\n\n**SSDs are already becoming part of the LLM inference memory hierarchy.**\n\nThe more interesting question is what happens next.\n\n## 💽 The External-SSD Question\n\nEvery local-AI forum gets this question weekly: *\"can I just run the model from an\nexternal SSD?\"* It is the right question asked one layer too early, and the answer\nexposes exactly what today's architecture is.\n\nOrdinarily: the SSD stores the model. The runtime loads the weights into memory,\ncomputation runs from memory, and storage is touched again mainly for persistence.\nUnless the runtime explicitly supports weight streaming or offload, the drive\naffects load time and nothing else. If the model already fits in RAM, a faster\nexternal SSD does not make inference better. **The storage is a container, not a\nparticipant.**\n\nThe current strategy for models that do not fit is compression. A [widely shared\nwriteup this week](https://medium.com/@manjunath.shiva/qwen-3-8-27b-on-a-16-gb-mac-mini-alibabas-new-vision-model-fully-in-memory-f3aaaacbfeb4) walks through squeezing Qwen 3.8-27B, a 55.6 GB\nvision-language model, down to 11.55 GB so it runs entirely inside a 16 GB Mac\nmini at reading pace. It is impressive work, and its author is honest about the\nlimit: the compressed build still cannot reliably drive a serious coding agent.\n\nThat admission is the interesting part. **Fitting the model is not the same as\nfitting the workload.** An agent needs more than resident weights: persistent\ncontext, retrieval indexes, tool state, checkpoints, a KV cache that grows with\nevery step. Compression shrinks the weights and does nothing about everything\nelse that increasingly capable systems drag along with them. Three different\nlevers are in play, and they are worth keeping distinct:\n\n**Model compression** reduces the size of the weights.**Memory-hierarchy optimization** decides what is resident, and when.**AI-native storage**, the subject of this article, would decide what should move, what stays cached, what gets transformed near the data, and what never needs to reach the GPU at all.\n\n## 🕰️ The Idea Is Old. The Workload Is New.\n\nComputing near stored data is not a new idea. Researchers were publishing **Active Disk** architectures in [1998](https://dl.acm.org/doi/10.1145/384265.291026), proposing drives with embedded processors so that data-intensive work could happen where the data already lived, and the [motivation they wrote down](https://www.vldb.org/conf/1998/p062.pdf) reads like it was drafted yesterday: why continuously move enormous datasets to a central processor when some of the work can happen where the data resides?\n\nThree things have happened since. Flash replaced spinning disks, and a modern SSD is already a small computer: controllers, firmware, parallel NAND channels, error correction, address translation. Adding an FPGA made it a programmable one, and Samsung shipped exactly that, twice, as the SmartSSD, marketed for compression, filtering, search and transformation. And the Storage Networking Industry Association (SNIA) gave the field an architectural vocabulary: computational storage, with defined APIs and interoperability work. (A thorough tour is [Past, Present and Future of Computational Storage: A Survey](https://arxiv.org/abs/2112.09691).)\n\nSo the natural question is: if the idea is twenty-five years old and the hardware shipped, why is it not everywhere? Because for twenty-five years the dominant workloads did not reward it enough. General-purpose queries touch data unpredictably. The win from pushing a filter into a drive was real but modest, and the software cost of programming storage was not.\n\nWhat changed is the workload. Two numbers from later in this article make the point as a pair: a measured retrieval query needed **0.8% of a 102.4 GB corpus**, and an independent out-of-core implementation of Kimi K3 activates **under 4% of its 2.78 trillion parameters** per token. LLM state is enormous, structured, and overwhelmingly *skippable*, and which parts matter is decidable in advance by something that understands the data. That selectivity profile is what computational storage spent twenty-five years waiting for.\n\n## 🤖 LLM Research Starts Moving Computation Toward SSDs\n\nSeveral recent research projects are particularly relevant.\n\nAnd notably, this research now spans a wide range of systems, from **datacenter-scale serving architectures to memory-constrained AI PCs and edge systems**.\n\nThat breadth matters.\n\nThe storage problem is not limited to giant hyperscale clusters.\n\nThe same mismatch between model state, context capacity and accelerator memory appears wherever increasingly capable models encounter finite GPU or unified-memory resources.\n\n### SmartANNS: Search Where the Vectors Live\n\nLarge RAG and vector-search systems may contain billions of embeddings.\n\nIn 2024, researchers presented **SmartANNS**, a billion-scale approximate-nearest-neighbor search architecture using multiple SmartSSDs.\n\nThe host CPU performs high-level coordination while SmartSSDs execute portions of the search over their local index shards.\n\nResearch:\n\nThis is particularly relevant to RAG.\n\nWhy move a gigantic embedding dataset toward a CPU or GPU just to discard nearly all of it after retrieval?\n\nSearch closer to the vectors.\n\nReturn the useful results.\n\n**What it demonstrated:** billion-scale ANN search running on real SmartSSD hardware, with the host coordinating shards. **What it only suggests:** that the same division of labor extends beyond nearest-neighbor search to retrieval generally.\n\n### InstInfer: Put Attention Near the KV Cache\n\nIn 2024, **InstInfer** explored an even more direct connection between LLM inference and computational storage.\n\nIts researchers proposed a flash-aware **in-storage attention engine** and KV-cache management architecture.\n\nRather than repeatedly moving huge KV caches through constrained PCIe links, portions of decode-phase attention could occur closer to the stored cache.\n\nResearch:\n\n[InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference](https://arxiv.org/abs/2409.04992)\n\nThat moves the idea beyond:\n\n**SSD as larger memory**\n\ntoward:\n\n**SSD as a participant in inference.**\n\n**What it demonstrated:** decode-phase attention executing beside the stored KV cache in a research prototype, beating the PCIe round trip in its evaluated regime. **What it only suggests:** that the advantage survives production serving, where batching and multi-tenancy change the arithmetic.\n\n### Near-Storage Attention\n\nOther work has continued exploring the same boundary.\n\nResearch in the **INF² / HILOS** direction examines near-storage processing where memory-intensive attention and KV-cache operations can be pushed toward computational-storage accelerators.\n\nResearch:\n\n[Near-Storage Processing for Generative LLM Inference](https://arxiv.org/abs/2502.09921)\n\nAgain, the objective is not to reproduce an entire GPU inside an SSD.\n\nIt is to move the operations whose data dependencies make them expensive to transport.\n\n**What it demonstrated:** that the memory-bound half of attention separates cleanly enough to push toward storage-side accelerators in evaluation. **What it only suggests:** that the separation holds when production controllers, not evaluation platforms, are doing the work.\n\n### SolidAttention: SSDs for Long Context\n\nAt USENIX FAST 2026, researchers presented **SolidAttention**, an LLM inference engine combining dynamic sparse attention with SSD-aware storage management.\n\nImportantly, the work targets **memory-constrained AI PCs**, demonstrating that SSD-aware inference is not solely a datacenter problem.\n\nThe system groups KV pairs into larger blocks, predicts future accesses and coordinates SSD I/O with GPU computation.\n\nAt a 128K-token context the authors report **up to 3.1× faster inference** and **up to\n98% less KV-cache memory**, with accuracy comparable to the unmodified model. Their\nown accuracy tables show it beating INT4 KV-cache quantisation substantially, which is\nthe usual way people buy memory back.\n\nTwo details are worth carrying forward. The paper measures that **loading a 1K-token\nKV cache (128 MB) from SSD takes about 40 ms, nearly half a decode step**, which is\nwhat \"storage became part of inference\" looks like as a number. And they find roughly\n**81% similarity in block selection between consecutive iterations**, which is what\nmakes speculative prefetching work: attention sparsity is not random, it has temporal\nlocality a system can exploit.\n\nMost striking, they report SSD-backed serving landing within **11% of fully in-memory\nthroughput**: the entire KV cache on flash, for a tenth of the speed.\n\nResearch:\n\n[SolidAttention (USENIX FAST 2026)](https://www.usenix.org/conference/fast26/presentation/zheng)\n\nThe important conceptual shift is this:\n\nAn SSD does not necessarily need to treat **all context as equally important**.\n\nSoftware can predict which portions matter.\n\nThat is the beginning of intelligent routing.\n\n**What it demonstrated:** measured serving within 11% of in-memory throughput with the KV cache on flash, on consumer-class hardware. **What it only suggests:** anything about in-device compute. The drive here is entirely passive; every prediction and placement decision is host-side, which is exactly what makes it a fair baseline for the architecture this article sketches.\n\n### HillInfer: Let the SmartSSD Decide What Matters\n\n**HillInfer**, published in 2026, pushes the concept even closer to the architecture proposed here.\n\nIt jointly manages KV-cache pools across CPU memory and SmartSSD storage.\n\nCrucially, it performs **importance evaluation inside the SmartSSD FPGA**, helping reduce unnecessary KV-cache movement.\n\nResearch:\n\n[HillInfer: Hierarchical KV Eviction Using SmartSSD](https://arxiv.org/abs/2602.18750)\n\nThink about what changed.\n\nThe SSD is not merely storing KV blocks.\n\nIt is helping decide:\n\n**Which KV blocks are worth moving?**\n\nThat is a much more interesting role.\n\n**What it demonstrated:** importance scoring running inside a SmartSSD FPGA, reducing KV movement. **What it only suggests:** that a drive can own richer placement policy. The FPGA scored blocks; the hierarchy was still managed by the host.\n\n### Tutti: Fixing the SSD-to-GPU Path\n\nEven if SSDs have enormous capacity, retrieving thousands of fragmented KV-cache blocks creates another problem.\n\nLarge numbers of small I/O operations can overwhelm software and make the CPU itself part of the bottleneck.\n\nThe 2026 **Tutti** research project attacks that problem by creating a more GPU-centric path between NVMe storage and GPU memory.\n\nIts design integrates with vLLM and reorganizes KV-cache storage around **larger object transfers and GPU-driven I/O scheduling**.\n\nResearch:\n\n[Tutti: Making SSD-Backed KV Cache Practical for Long-Context LLM Serving](https://arxiv.org/abs/2605.03375)\n\n**What it demonstrated:** that SSD-backed KV cache becomes practical when transfers get bigger and the GPU drives the I/O. **What it only suggests:** nothing about in-device intelligence at all. Tutti is evidence for better paths to passive storage, and therefore a live competitor to the smarter-drive thesis.\n\nNotice how the research question has evolved.\n\nIt is no longer:\n\n**Can an SSD store AI data?**\n\nOf course it can.\n\nIt is becoming:\n\n**How should the entire storage-memory-GPU hierarchy be redesigned around AI inference?**\n\n*Computational storage started by moving simple operations toward stored data. LLM research is now asking whether retrieval, attention and context management should move there too.*\n\n## 🧭 The Next Step: Stop Assuming Everything Belongs on the GPU\n\nThis is where I think the most interesting research direction appears.\n\nAn LLM inference request is **not one monolithic computation**.\n\nIt contains different kinds of work.\n\nSome are perfect for GPUs.\n\nOthers involve searching, filtering, ranking, cache lookup, compression and decompression, deduplication, sparse selection, data movement, metadata management and prefetching.\n\nWhat if an AI-aware storage controller could classify those operations and determine which tier should execute them?\n\nInstead of assuming:\n\n**everything eventually goes to the GPU**\n\nthe system could ask:\n\n**Does this operation actually need the GPU?**\n\n## 🚦 A Theoretical AI Workload Router\n\nImagine a request enters an AI system.\n\nThe storage layer does not simply expose anonymous NVMe blocks.\n\nIt understands some semantics of models, KV caches, embeddings and context.\n\nA routing layer could make assignments such as:\n\n**Storage-local:**\n\n- Search these four billion embeddings and return the best candidates.\n- Determine whether this document prefix already has reusable KV state.\n- Scan this 200 GB KV-cache pool and identify the blocks most likely to matter.\n- Compress, decompress, quantize or prepare selected model data before transfer.\n- Identify which Mixture-of-Experts weights are likely to be needed next and prefetch them.\n\n**CPU / system memory:** orchestration, branching, scheduling, metadata handling and irregular processing.\n\n**GPU HBM:** the latency-critical active working set.\n\n**GPU compute:** dense matrix multiplication, tensor operations and the hot Transformer path.\n\nThis changes the architectural objective completely.\n\nThe goal is no longer:\n\n**Make NAND fast enough to imitate HBM.**\n\nIt becomes:\n\n**Prevent the GPU and HBM from ever receiving data that did not need to be there.**\n\nThat potentially reduces bandwidth pressure, scarce HBM consumption, CPU-mediated I/O, unnecessary GPU activity and the energy spent moving irrelevant data. The routing objective becomes: **move less data, occupy less scarce HBM, spend less energy.**\n\n## 📊 Which LLM Workloads Could Potentially Move?\n\nSome possibilities already have research behind them.\n\nOthers remain speculative.\n\n| LLM workload | Potential near-storage role | Likely GPU requirement |\n|---|---|---|\n| Dense Transformer GEMM | Poor fit for flash-side processing | GPU remains ideal |\n| Full dense attention | Limited | Mostly GPU |\n| Sparse-attention selection | Candidate | GPU computes selected attention |\n| KV-cache persistence | Excellent fit | GPU only when data becomes active |\n| KV prefix lookup | Excellent fit | Often no |\n| KV importance scoring | Already being researched | Not necessarily |\n| KV compression | Candidate | Not necessarily |\n| KV eviction | Strong candidate | No |\n| KV deduplication | Strong candidate | No |\n| Vector similarity search | Already demonstrated in SmartSSD research | Optional GPU reranking |\n| RAG filtering | Strong candidate | Not necessarily |\n| Weight decompression | Candidate | GPU executes resulting weights |\n| Quantization / dequantization | Candidate | Depends on architecture |\n| MoE expert lookup | Strong candidate | GPU executes selected experts |\n| MoE expert prefetch | Strong candidate | No |\n| Model-weight backing store | Strong fit | GPU consumes hot portions |\n| Prompt / document preprocessing | Candidate | Often no |\n| Context persistence between sessions | Excellent fit | No |\n\nThe important point is not that all of these **should** move to an SSD.\n\nThey should not.\n\nThe research opportunity is determining:\n\n**Which operations save more bandwidth, latency, energy or GPU resources by moving toward storage than they cost to execute there?**\n\n## ⚡ Why This Could Reduce GPU Activity and Energy Use\n\nConsider a simple retrieval example.\n\nImagine an enterprise AI system has a **20 TB embedding corpus**.\n\nThe objective of a query might ultimately be to retrieve 20 useful passages.\n\nA naive conceptual architecture looks like:\n\n**Storage → move candidate data → CPU/GPU search → discard almost everything → 20 useful passages**\n\nA computational-storage architecture can instead look like:\n\n**Storage + local vector computation → search/filter/rank locally → 20 useful passages → GPU**\n\nThe SSD did not become faster than the GPU.\n\nIt avoided moving irrelevant data.\n\nAnd there is a physical and economic argument underneath that distinction:\n\n**Moving data through a memory hierarchy can consume substantially more energy than performing relatively simple arithmetic on that data.**\n\nThis has been recognized in computer architecture for years.\n\nMark Horowitz's influential work on computing energy illustrated how memory access and data movement can substantially exceed the energy required for basic arithmetic operations.\n\nLater neural-network accelerator research similarly highlighted how fetching model data from off-chip memory can dominate the energy cost of the arithmetic that ultimately consumes it.\n\nUseful references include:\n\n[Computing's Energy Problem (and what we can do about it): Mark Horowitz, ISSCC 2014](https://ieeexplore.ieee.org/document/6757323)\n\n[EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://arxiv.org/abs/1602.01528)\n\nAnd for LLM inference specifically, this is no longer only an architectural estimate. The SolidAttention authors (whose system appears above) measured whole-system energy directly on their AI-PC testbed: an NVIDIA RTX 4070 Laptop GPU with 8 GB of GDDR7 plus 16 GB of DDR5, with the KV cache resident on the machine's NVMe SSD, against llama.cpp holding the entire cache in memory. The counterintuitive result is worth stating carefully: the SSD-backed configuration draws **higher peak power** than the in-memory baseline (unsurprising, since it is actively hitting flash), yet consumes **3.68 joules per token against llama.cpp's 5.37**, a 46% improvement.\n\nHigher power, less energy. The system finishes sooner and idles less, so the integral comes out ahead of the peak. It is the single most useful empirical result for this article's argument: **adding storage traffic to an inference path made it more energy efficient, not less**, because what it removed was waste, and waste is what the energy was being spent on.\n\nThat means locality can potentially create **three wins at once**:\n\n- Move less data.\n- Consume less scarce HBM capacity and bandwidth.\n- Spend less energy moving and processing unnecessary data at a more expensive tier.\n\nComputation inside an SSD is not free; flash access, controllers, accelerators and interconnects all consume energy.\n\nThe correct comparison is:\n\n**Does performing this operation near the data consume less energy and time than transporting the data to another tier, processing it there, and potentially discarding most of it?**\n\nFor some workloads, the answer will be no.\n\nFor others, such as filtering, retrieval, cache scoring, compression, sparse selection and similar data-reduction operations, that tradeoff could become very attractive.\n\nThe same principle could eventually apply to KV cache.\n\nInstead of:\n\n**Retrieve huge historical KV cache → GPU determines what matters**\n\nthe architecture becomes:\n\n**Storage evaluates cache → retrieves selected blocks → GPU receives hot working set**\n\nThe GPU still performs the operations it is uniquely good at, but it does less housekeeping, sees less irrelevant data, dedicates less HBM to cold state, and the system stops paying bandwidth and energy to move data only to discover it was not needed.\n\n## 🧪 A Measurement Anyone Can Reproduce\n\nClaims about data movement deserve numbers, so I measured this on ordinary consumer hardware: a laptop with 40 GB of RAM and a WD_BLACK SN7100 NVMe SSD (~2.1 GB/s measured sequential read on this machine).\n\nThe setup: **50 million synthetic document embeddings** at 1024 dimensions, fp16, generated with realistic cluster structure and stored in an IVF-style cluster-grouped layout. That is **102.4 GB** on disk, deliberately 2.5× larger than RAM so the operating system's page cache cannot quietly fake the results. Every scan is structurally forced to hit the drive.\n\nThen one top-20 similarity query, answered two ways:\n\n**Method A: Naive full scan.** Stream the entire corpus from SSD to CPU and score everything. This is \"move the bytes to the compute.\"\n\n**Method B: Index-guided.** Score the query against 1,024 cluster centroids (a few kilobytes of data), then read only the 8 most promising clusters, about 0.8% of the corpus. This is a software stand-in for \"decide near the data, and move only what matters.\"\n\nThe results:\n\n| Naive full scan | Index-guided | |\n|---|---|---|\n| Data moved | 102.4 GB |\n0.80 GB |\n| Wall time | 355 s | 1.6 s |\n| Top-20 returned | baseline | identical (recall = 1.0) |\n| Bytes moved per useful byte | ~2,500,000 : 1 |\n~19,500 : 1 |\n\nSame question. Same answer. **128× less data movement. 228× faster.**\n\nTwo observations from this measurement matter for the argument of this article.\n\nFirst, the naive scan's effective throughput was only 0.29 GB/s, well below the drive's 2.1 GB/s raw read speed, because the host CPU had to both receive *and* score every byte. Moving data to compute makes the host pay twice.\n\nSecond, and more importantly: even the *indexed* query still moved roughly **19,500 bytes for every byte of useful answer**. The index knows which clusters to read, but the host must still import entire clusters to find the 20 vectors it wants.\n\nThat residual gap is precisely the territory an AI-native retrieval plane would claim. A drive that could score candidates internally and return only the winners would attack the remaining four orders of magnitude.\n\nA back-of-envelope energy note makes the same point from the physics side, with the assumptions stated so a skeptical reader can recalculate. The naive query moves 102.4 GB, which is 8.2 × 10¹¹ bits. Charge each bit one PCIe crossing at ~5 pJ/bit (a commonly cited figure for the link plus controller overhead) and one DRAM write plus one DRAM read for host staging at ~20 pJ/bit each (Horowitz's 45 nm figures; newer nodes are lower). That totals roughly **35 to 40 joules of pure transport**. The arithmetic that decided the answer, 50 million fp16 dot products of length 1024, is about 5 × 10¹⁰ multiply-accumulates, and at low single-digit pJ per fp16 MAC that is on the order of **0.1 joules**. Under these assumptions the movement outcosts the math by more than two orders of magnitude, and no generosity toward the transport figures changes the conclusion: the energy bill of that query was overwhelmingly a *transportation* bill.\n\nThat is the same shape as SolidAttention's measured 3.68 versus 5.37 joules per token, earlier in this piece: the bill is transport, and cutting transport pays even when peak power rises.\n\nThis measurement is deliberately modest: one query shape, synthetic data, a software index, a consumer drive. It does not demonstrate computational storage; no FPGA was involved. What it measures is the size of the prize: the ratio between the bytes a query touches and the bytes it needs. That ratio is what every system in the research above, from SmartANNS to HillInfer, is built to shrink.\n\n## 🔭 The Benchmark I Cannot Run Yet\n\nThe measurement above stops exactly where my hardware stops.\n\nI do not own a computational storage device. So the natural third method is, for now, a proposal, stated precisely enough to be run, and to be proven wrong.\n\n**Method C: in-storage candidate scoring** (requires a SmartSSD-class device: NVMe storage plus an FPGA or equivalent accelerator in the same module):\n\n- The host sends the query vector and the 8-cluster probe list to the device, a few kilobytes\n**down**. - The device scans the probed clusters internally, computes the fp16 dot products next to the NAND, and returns only the top-20 candidates\n**per cluster**, roughly 300 KB** up**. - The host merges 160 candidates into the final top-20.\n\n**What to measure**, against Methods A and B on the same corpus: bytes crossing the bus in each direction, wall time, host CPU utilization, and, with a wall-power meter, energy per query.\n\n**One prediction is already verifiable at the protocol level.** I implemented the Method C wire protocol with a simulated device: a separate process that exclusively owns the corpus and speaks only the protocol, so the bus bytes are counted across a real boundary. Result: **2,092 bytes down, 329,600 bytes up, a movement-waste ratio of 8.1 : 1**, with the returned top-20 identical to the full-scan ground truth. The \"under 10 : 1\" claim is protocol arithmetic, not speculation.\n\n**The predictions that still need hardware:**\n\n- Wall time stays at or below Method B's, because computational-storage designs can expose more aggregate internal NAND bandwidth than the external link, and the scoring math is trivial next to the transport it eliminates.\n- Energy per query drops\n**even though** the device's compute is far weaker than a host CPU or GPU, because the energy bill was always the transport, not the arithmetic.\n\nA CPU simulating an FPGA proves nothing about either, so those two columns stay honestly empty until someone runs this on real silicon.\n\n**And what would falsify the thesis:** if in-device scoring turns out slower or more energy-hungry than shipping the clusters out, then vector scoring belongs on the host after all, and the retrieval-plane claim weakens to cache persistence and data management only. That result would be worth publishing too.\n\nEverything except the FPGA kernel is public in the repository: the corpus generator, the baseline harnesses, the Method C wire protocol, a NumPy reference implementation of the device-side computation, and the simulated backend that verifies the contract. A hardware owner implements one class and gets a complete experiment. The baseline is waiting.\n\n## 🔬 Independent Signal: Model Weights Are Becoming a Placement Problem\n\nEverything above concerns **retrieval**: embeddings, KV cache, the bytes a query\ntouches. While I was writing it, a second and quite different workload began exhibiting\nthe same architecture problem.\n\n**A note on timing, because it changes how much weight this deserves.** I came across\nthis *while writing the sections above*, after the thesis was formed and the benchmark\nhad already been run. I was not looking for supporting evidence. Its author was not\narguing about storage architecture, and nothing in it references any of this. **That is\nprecisely what makes it useful: it is convergence, not corroboration.** An argument that\npredicts where an unrelated project lands is worth more than one assembled from\ncitations chosen to fit.\n\nIn July 2026 Moonshot AI released **Kimi K3**, a mixture-of-experts model of roughly\n**2.8 trillion parameters**. Its sparsity is the interesting part: the model holds\n**896 experts per layer and activates 16 of them per token**, so only about\n**104 billion of 2.78 trillion parameters, under 4%, participate in producing any\ngiven token.**\n\nAn independent developer then published ** kimi-k3-in-c**, a portable C99\nimplementation that runs that model on a single CPU, with no GPU and no framework. Its\nreported memory ladder is worth reading as a sequence rather than a headline:\n\n| Stage | Memory | How |\n|---|---|---|\n| Full bf16 model | 5,560 GB |\nbaseline |\n| Shipped checkpoint | 1,560 GB |\nexperts pre-quantised |\n| Resident set only | 113.49 GB |\nexperts never loaded |\nMeasured peak RSS |\n8.24 GB |\ntrunk streaming |\n\nRoughly **1.447 TB of routed experts are never resident at all**. They stay on storage\nand are multiplied straight out of their packed 4-bit form. About **96.3% of the\nexpert parameters never enter memory.**\n\n### Be precise about what this does and does not show\n\n**This is not computational storage. The SSD is entirely passive.** It is not routing\nexperts, scoring anything, or making decisions. It is a fast disk being read.\n\n**What changed is the host software.** The runtime became **model-aware** enough to\ndecide which parts of an enormous model deserve memory and which can stay on storage:\ndense trunk resident, routed experts streamed, quantised formats consumed in place.\nThat is data placement driven by *model semantics*.\n\nIt is also, plainly, a **feasibility demonstration rather than a serving solution**.\nRunning a frontier model from flash on a CPU is not fast, and nobody should read \"2.78T\nparameters in 8 GB\" as a claim about production throughput.\n\n### The number that actually matters is the I/O share\n\nThe headline invites the wrong reading. The important figure in the published memory\nladder is that **I/O accounts for roughly 41% to 61% of execution time** across the\ntested memory configurations.\n\nStorage stopped being where the model waits and became **a material component of\ninference execution time**. Once that is true, *where a weight lives and when it moves*\nstarts determining performance, which is precisely the point at which architecture\ngets interesting.\n\n### Two workloads, one principle\n\n**My benchmark asks:** why move 102.4 GB of embeddings when a handful of vectors\nproduce the answer?\n\n**Kimi asks:** why make 2.78 trillion parameters resident when under 4% of them\ncompute the next token?\n\nDifferent workloads. **Same systems principle: work out what matters before paying to\nmove everything else.**\n\nThe symmetry is closer than it first appears. My index-guided query touched **0.8 GB of\na 102.4 GB corpus, about 0.8%.** Kimi activates **under 4% of its parameters per\ntoken**. Both are cases where the useful fraction is small, known in advance, and\nidentifiable by something that understands the data's structure: an IVF index in one\ncase, MoE routing in the other.\n\nMy measurement demonstrates the opportunity on the **Retrieval Plane**. Kimi exposes\nthe same pressure arriving on the **Weight Plane**, independently, from a completely\ndifferent direction, and without anyone setting out to prove a point about storage\narchitecture.\n\nThat is what makes the five planes below look less like a wish list and more like one architecture inferred from several workload classes.\n\n## 🧩 What Would an AI-Native SSD Actually Look Like?\n\nThe evidence above keeps pointing at a small number of recurring jobs that are expensive mainly because of data movement: deciding which embeddings matter, deciding which KV blocks matter, deciding which experts matter, and then shipping only those.\n\nIf we designed storage around that pattern instead of around a conventional block interface, the useful capabilities fall into five planes. To be explicit about epistemic status: the benchmark, the Kimi K3 numbers and the published papers above are this article's evidence layer, and what follows, like the workload router sketched earlier, is its speculative layer. These are not product features. They are the minimum set of operations that would let the storage tier participate in the decisions rather than just serve the bytes.\n\n### 1. Weight Plane\n\nModel weights are read far more often than they are written. A storage tier that understood tensor layout, quantization format and expert boundaries could keep the dense trunk resident and stream only the experts the router actually selected. The interesting policy questions are residency and hotness, not raw sequential bandwidth. Taking the requirements list directly from the placement decisions the Kimi implementation actually makes:\n\n**Dense-trunk residency**: which layers stay in memory permanently** Routed-expert streaming**: which weights are read straight from storage in their packed quantized form and never made resident** Expert hotness**: observed activation frequency, not just static classification** Expert-cache allocation**: how a fixed memory budget is divided between pinning dense layers and caching frequently-activated experts** Memory-budget-aware placement**: the same model laid out differently on a 16 GB machine than on a 512 GB one\n\n### 2. KV Plane\n\nKV cache is append-heavy, frequently re-read, and increasingly long-lived. The useful primitives are prefix lookup, importance scoring, selective fetch, compression and eviction. Current hierarchical KV systems already do some of this on the host. Moving the scoring and selection closer to the blocks themselves is the natural next step if the volume of cold KV continues to grow.\n\n### 3. Retrieval Plane\n\nWhen the corpus is much larger than memory, the dominant cost is usually moving candidates that will later be discarded. Approximate search, metadata filtering and coarse ranking are the operations that most clearly benefit from running beside the data. SmartANNS already showed parts of this are feasible on existing computational storage.\n\n### 4. Compute Plane\n\nThe device does not need GPU-class matrix throughput. It needs enough arithmetic to score vectors, evaluate simple importance heuristics, decompress and filter. Anything denser still belongs on the GPU.\n\n### 5. Routing Plane\n\nSomeone has to decide, for each request, which of the above operations should run locally and which data is worth moving. That policy can live on the host, in the device, or be split. The important part is that the decision is made with knowledge of both the model structure and the actual cost of moving the data.\n\nNone of this requires the storage device to become a general-purpose accelerator. It only requires that the device stop being a pure block server for the workloads where most of the bytes will be thrown away after a cheap test.\n\n### The Hard Part Is Not the Silicon\n\nThe strongest objection to this sketch is not bandwidth or power. It is software surface area. The planes above ask a storage device to understand model topology, KV importance, MoE routing metadata, quantization formats and versioning. That knowledge currently lives in fast-moving host runtimes (vLLM, TensorRT-LLM, Dynamo) that change monthly. Device firmware ships on a different clock, and a drive that misunderstands a model version does not merely run slowly. It returns wrong answers, from a component the host has stopped double-checking.\n\nSo the realistic division of labor is narrower than \"move the intelligence into the drive.\" The host keeps the policy: which model, which quantization, what counts as important, when to evict. The device earns the inner loops that stay stable across model generations: scan, score, top-k, filter, decompress. Those operations have not changed meaningfully in a decade, and their inputs can be validated cheaply at a protocol boundary. Read the five planes through that filter and they shrink to their durable cores, which is how they should be read.\n\nEven the narrowed version is a multi-year software project before it is a silicon project: a protocol for describing placement policy to a device, a conformance suite, a failure and versioning model. Active Disks did not stall in 1998 for lack of transistors either. That history is also why this article's claims are framed as testable hypotheses rather than predictions about the next product cycle.\n\n## 💾 Why 245 TB Matters\n\nThis is why Micron's 245.76 TB 6600 ION is interesting in this discussion.\n\nIt is **not** evidence that SSDs can replace GPU memory.\n\nThey cannot.\n\nThe drive demonstrates something different:\n\n**Flash capacity is now operating on a completely different scale from accelerator memory.**\n\nMicron began shipping the [6600 ION 245.76 TB SSD](https://www.micron.com/products/storage/ssd/data-center-ssd/6600-ion) in May 2026.\n\nA handful of drives can provide around a petabyte of local flash capacity.\n\nThat creates room for enormous amounts of AI state:\n\n- Model libraries\n- Quantized model variants\n- Mixture-of-Experts weights\n- Persistent KV caches\n- Enterprise document collections\n- Embedding indexes\n- Multimodal embeddings\n- Agent histories\n- User-specific context\n- Reusable prompt prefixes\n- Long-running reasoning state\n\nThe problem becomes less:\n\n**Can we store it?**\n\nand increasingly:\n\n**Can we make that capacity participate intelligently without drowning the GPU in I/O?**\n\n## ⚠️ The Hard Limit: Capacity Is Not Bandwidth\n\nThis distinction cannot be overstated.\n\nA high-performance PCIe Gen5 SSD can deliver roughly **14 GB/s** of sequential reads.\n\nAn NVIDIA B200 can access its local HBM at up to roughly **8,000 GB/s**.\n\nThese devices exist for different purposes.\n\nIf a dense model had to retrieve its entire parameter set from NAND for every generated token, performance would collapse.\n\n**But sparsity changes the equation, and this is where the argument gets interesting.**\n\nA sparse mixture-of-experts model may need only a small fraction of its total weight\nspace for any given computation. Kimi K3 activates under 4% of its parameters per\ntoken. The question stops being *\"can flash feed a model?\"* and becomes **\"how much of\nthe model actually has to cross the boundary?\"**\n\nThat is a more useful question than either of the slogans it replaces. Not *\"SSDs could\nrun models\"*, and not *\"SSDs are too slow for models\"*, but: **it depends on what\nfraction must move, and that fraction is a property of the model's architecture rather\nthan the drive's.**\n\nAn AI-native SSD therefore does **not** win by pretending flash is slow HBM.\n\nIt wins when locality allows it to reduce the amount of information moving across the boundary.\n\nThat is the architectural principle behind computational storage:\n\n**Do work where the data already exists when doing so costs less than moving the data somewhere else.**\n\n## 📊 The AI Memory Hierarchy\n\n| Tier | Capacity class | Bandwidth class | Access latency |\nLikely AI role |\n|---|---|---|---|---|\n| GPU SRAM / cache | MB | Extremely high | sub-microsecond | Immediate computation |\n| GPU HBM / VRAM | Hundreds of GB per GPU | Multi-TB/s | < 1 μs |\nHot tensors, active KV, model execution |\n| CPU / system memory | TB-class per server | Hundreds of GB/s | 10–20 μs |\nLarger working sets, orchestration, offload |\n| NVMe SSD | TB to 245 TB per drive | Tens of GB/s | > 500 μs |\nPersistent state, weights, KV, vectors |\n| AI-native storage | TB to PB across devices | Flash-class physical bandwidth | Flash-class, but fewer round trips |\nLocal search, filtering, KV management, near-data compute |\n\n*Latency figures as measured by the SolidAttention authors on consumer hardware.*\n\n**The latency column is the one that explains the architecture.** Bandwidth says an\nSSD moves tens of gigabytes per second. Latency says each individual request costs\n**roughly 500× a DRAM access and 500,000× an L1 hit**. That gap is why fine-grained\nrandom reads are fatal while coarse sequential ones are survivable, and it is why\nevery system in this article converges on the same two moves: **make the transfers\nbigger, and start them earlier.** SolidAttention consolidates KV pairs into blocks\nand prefetches speculatively. My benchmark reads whole clusters rather than\nindividual vectors. Kimi streams packed expert blocks rather than scattered weights.\n\nAn AI-native device does not beat that latency. **It reduces how many times you have\nto pay it.**\n\nThe final tier does not magically gain HBM bandwidth.\n\nIts advantage is different:\n\n**It reduces how much data needs the faster tiers at all.**\n\n## 🔮 The Natural Progression\n\nSeen historically, the architecture looks less like a wild prediction and more like a progression.\n\n### Stage 1: Passive Storage\n\n**SSD stores the model.**\n\nUniversal today.\n\n### Stage 2: Computational Storage\n\n**SSD executes filtering, compression, search and specialized computation locally.**\n\nThis exists.\n\n### Stage 3A: AI-Aware Host Orchestration\n\n**The model runtime understands topology, locality and residency, and decides what\nlives in memory versus storage.**\n\nThis exists today. `kimi-k3-in-c`\n\nis a working example: the storage device is passive,\nbut the host software is model-aware enough to keep a dense trunk resident and stream\n1.45 TB of experts from disk.\n\n### Stage 3B: AI-Aware Storage Systems\n\n**That intelligence begins moving into storage software, controllers and accelerators\nrather than living entirely in the host.**\n\nThis is where the research above sits: HillInfer scoring KV importance inside a SmartSSD FPGA, InstInfer placing attention near the cache, SmartANNS searching shards on-device.\n\n### Stage 4: AI-Native Storage\n\n**Dedicated KV, retrieval, sparse-compute and routing engines are designed directly into the storage architecture.**\n\nThis remains primarily a research and design direction.\n\n### Stage 5: Processing Near or Inside Memory\n\nComputation moves even closer to the memory arrays themselves.\n\nAt that point the architecture begins overlapping with broader **processing-in-memory** and **near-memory computing** research.\n\nThe boundaries between storage, memory and compute become increasingly difficult to define.\n\n## 🚀 The GPU Is Not Going Away\n\nAnd it should not. Dense matrix multiplication, HBM and latency-critical active state belong exactly where they are: GPUs remain extraordinarily good at the hot numerical core of Transformer inference.\n\n**And a distinction worth drawing precisely:** being able to *execute* a model and\nbeing able to *serve* it efficiently are different engineering problems. Running 2.78\ntrillion parameters from flash on a CPU is a remarkable demonstration of the first. It\nsays almost nothing about the second: throughput, latency and dense numerical\nperformance remain exactly why GPUs exist.\n\nWhich reinforces the router idea rather than undermining it: the system should choose tiers by the characteristics of the workload, not by conviction about which processor \"runs AI.\"\n\nBut GPUs are also expensive resources with limited HBM.\n\nUsing them as the destination for every piece of data simply because they are the fastest processors may become increasingly inefficient.\n\nThe alternative is a more intelligent hierarchy.\n\n**CPU:** orchestrate.\n\n**GPU:** perform dense parallel computation.\n\n**HBM:** hold the immediate working set.\n\n**DRAM:** extend active memory.\n\n**AI-native storage:** hold enormous persistent state and process selected operations near that state.\n\n## 🧠 The Bigger Architectural Idea\n\nFor decades we thought about computers roughly like this:\n\n**Storage stores.**\n\n**Memory feeds.**\n\n**CPU computes.**\n\nGPUs already disrupted that model by combining enormous parallel-compute capability with extremely fast local memory.\n\nComputational storage attacks the same architecture from the opposite direction:\n\n**Storage begins acquiring compute.**\n\nLLMs may cause those two directions to converge.\n\nWe are already seeing:\n\n- GPU-aware storage\n- SSD-backed KV cache\n- KV-aware request routing\n- Computational SmartSSDs\n- Vector search near storage\n- Attention near storage\n- GPU-direct storage paths\n- Hierarchical context memory\n- Intelligent KV eviction\n\nThere is also an energy dimension connecting nearly all of these developments.\n\nAs arithmetic becomes increasingly specialized and efficient, **the relative cost of moving data becomes increasingly important**.\n\nThat is one reason near-memory, in-memory and near-storage computing continue to attract research interest.\n\nLocality can improve performance while also avoiding some of the energy spent transporting data through multiple layers of a system.\n\nAn AI workload router could therefore optimize for more than latency: where the data is, how much of it must move, what that movement costs, which tier can execute the operation efficiently, and how much energy each route consumes. Eventually, routing an AI workload might look less like conventional I/O scheduling and more like a **cost function across compute, capacity, bandwidth, latency and energy**.\n\nThe next step may be systems that decide dynamically:\n\n**Where should this particular piece of AI work happen?**\n\nNot everything belongs on the CPU.\n\nNot everything belongs on the GPU.\n\nAnd increasingly, not everything needs to leave storage.\n\n## ⚖️ Where This Loses\n\nThe hypotheses below state what would refute each claim individually. It is worth being equally plain about the regimes where the whole approach loses to a competing architecture even if no single claim breaks:\n\n**Anything that fits in HBM.** A model whose weights and context sit comfortably in GPU memory has nothing to route. Storage intelligence is a response to overflow; without overflow it is pure overhead.\n\n**Workloads with poor selectivity.** The entire argument runs on the gap between bytes read and bytes needed. Dense training epochs touch essentially everything they read, and any workload without exploitable structure offers the drive nothing to discard. No selectivity, no prize.\n\n**Hard latency floors.** SolidAttention's measured 40 ms to load a 1K-token KV cache is survivable because prefetching hides it. Work that can be neither predicted nor batched cannot amortize a device round trip, and interactive decode at low batch sizes will keep such steps host-side.\n\n**If CXL memory pooling gets cheap enough.** Pooled DRAM-class memory over CXL attacks the same capacity gap with load/store semantics and no new software model. Flash keeps a durability and cost-per-terabyte advantage measured in multiples, but every generation of cheaper pooled memory erodes the middle of this argument.\n\n**If bigger transfers capture the win.** This is H1 failing, restated as the competitive case: Tutti-style GPU-direct paths plus good host-side indexes may capture most of the achievable benefit on passive storage. In that world the future is model-aware runtimes, kimi-k3-in-c writ large, and no new silicon at all.\n\n## 💡 The Research Question\n\nSo I do not think the most interesting question is:\n\n**Can an SSD run an LLM?**\n\nThat frames the problem incorrectly.\n\nThe better question is:\n\n**How much of an LLM system can be moved toward hundreds of terabytes of storage so that the GPU performs only the work that genuinely requires GPU-class compute and bandwidth?**\n\nThat creates a very different research agenda: build storage that understands models, context, KV caches, sparsity, retrieval, locality and energy cost, then give it enough specialized computation to act on that knowledge.\n\nBut an agenda is only useful if it can be proven wrong. So here are the five claims this article rests on, each stated as something a reader with the right hardware could refute. I have measured none of them directly. The first is the one I consider most likely to fail.\n\n### H1: Placement\n\n**Device-side narrowing beats host-side narrowing by a margin that grows with the ratio of\ncorpus size to interconnect bandwidth.**\n\nThis is the load-bearing claim, and the benchmark in this article does not establish it. That\nmeasurement narrowed 102 GB to 0.80 GB on the *host*, using ordinary NVMe reads. Any system\ncan do that. Nothing in it demonstrates that the selection has to happen inside the drive.\n\n*Refuted if:* host-side selection over ordinary NVMe reads captures ≥95% of the benefit at\nrealistic corpus sizes. In that case near-storage compute is complexity without payoff, and\nthe honest conclusion collapses to a narrower one: route the work, but route it on the host.\n\n### H2: Energy crossover\n\n**There is a selectivity threshold below which the controller energy spent avoiding a transfer\nis less than the energy of the transfer itself.**\n\nThis article argues that moving less data saves energy. It measures bytes, not joules. The inference is reasonable, since published figures put off-chip data movement one to two orders of magnitude above the arithmetic it feeds, but reasonable is not measured.\n\n*Refuted if:* no crossover exists at selectivities achievable by real indexes, or controller\nidle power swamps the transfer saving at realistic duty cycles.\n\n### H3: Controller ceiling\n\n**Present SmartSSD-class compute is sufficient for IVF scan and top-k selection, but not for\nattention over long context.**\n\n*Refuted if:* InstInfer-class near-storage attention holds its advantage as context length\ngrows, without host assistance. That would mean the compute plane is less constrained than I\nassume here, and more of the workload moves than this article predicts.\n\n### H4: Sparsity routing\n\n**For mixture-of-experts models, expert-selection metadata is small enough to route on-device,\nso bytes moved track active experts rather than total parameters.**\n\nThe out-of-core Kimi K3 work is independent signal here: a 2.78-trillion-parameter model where under 4% of parameters compute a token. That implementation was not built to test this article's thesis, which is precisely what makes it useful as evidence.\n\n*Refuted if:* routing metadata itself becomes bandwidth-bound at scale, or expert locality is\npoor enough that the working set approaches the full model.\n\n### H5: Capacity is not bandwidth\n\n**The binding constraint migrates from capacity to per-device read bandwidth within the next\ntwo device generations.**\n\nCapacity has grown far faster than the interface feeding it. A 245 TB drive that cannot be read quickly is an archive, not a memory tier.\n\n*Refuted if:* per-device read bandwidth scales with capacity through the next generation, which\nwould relieve the pressure this entire argument depends on.\n\nWhat unites all five is a change in what the system is being asked to optimize for.\n\nWe should stop treating maximum GPU utilization as the primary success metric. The better target is simpler and harder: for every piece of work, choose the tier that gets the answer with the least data movement. Sometimes that will still be the GPU. Often it will not.\n\nThat could mean reducing GPU activity for selected operations.\n\nReducing HBM pressure.\n\nReducing PCIe and fabric traffic.\n\nReusing previously computed context instead of recreating it.\n\nReducing the energy cost of inference by eliminating unnecessary data movement.\n\nThe SSD probably will not replace the GPU.\n\nBut it may become much more than the vault feeding it.\n\n**The line separating storage, memory and compute is already starting to disappear.**\n\nAnd the enormous capacity gap between HBM and flash suggests there is still a very large architectural space left to explore.\n\nVector retrieval and sparse model inference look like entirely different workloads. They\nare converging on the same architectural problem: **an enormous pool of data sits in a\ncheap capacity tier, and only a small portion of it is useful to the next computation.**\nA 102 GB embedding corpus where 0.8% answers the query. A 2.78 trillion-parameter model\nwhere under 4% computes the token. The opportunity in both cases is identical: identify\nthat portion **before** paying to move everything else.\n\nThe useful research question is no longer *how do we make the GPU compute\nfaster?* It is *how much of the work can we keep from ever reaching it?*\n\nCompute faster is a mature discipline with decades of momentum behind it. Move less is still wide open.\n\n## 📚 Research Trail\n\n### Transformer and LLM Memory Architecture\n\n[Attention Is All You Need (Vaswani et al.)](https://arxiv.org/abs/1706.03762)\n\n[FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://arxiv.org/abs/2205.14135)\n\n[NVIDIA: Large-Scale LLM Inference and KV Cache Offload](https://developer.nvidia.com/blog/accelerate-large-scale-llm-inference-and-kv-cache-offload-with-cpu-gpu-memory-sharing/)\n\n[NVIDIA Dynamo: KV Cache Offloading](https://docs.nvidia.com/dynamo/backends/v-llm/kv-cache-offloading)\n\n### Computational Storage History\n\n[Active Disks: Programming Model, Algorithms and Evaluation (1998)](https://dl.acm.org/doi/10.1145/384265.291026)\n\n[Active Storage for Large-Scale Data Mining and Multimedia (VLDB 1998)](https://www.vldb.org/conf/1998/p062.pdf)\n\n[Past, Present and Future of Computational Storage: A Survey](https://arxiv.org/abs/2112.09691)\n\n[Samsung Second-Generation SmartSSD Computational Storage](https://news.samsung.com/global/samsung-electronics-develops-second-generation-smartssd-computational-storage-drive-with-upgraded-processing-functionality)\n\n### LLM + SSD / Near-Storage Research\n\n[SmartANNS: SmartSSD Approximate Nearest Neighbor Search, USENIX ATC 2024](https://www.usenix.org/system/files/atc24-tian.pdf)\n\n[InstInfer: In-Storage Attention Offloading](https://arxiv.org/abs/2409.04992)\n\n[Near-Storage Processing for Generative LLM Inference](https://arxiv.org/abs/2502.09921)\n\n[SolidAttention: SSD-Based Long-Context LLM Serving, USENIX FAST 2026](https://www.usenix.org/conference/fast26/presentation/zheng)\n\n[HillInfer: Hierarchical KV Eviction Using SmartSSD](https://arxiv.org/abs/2602.18750)\n\n[Tutti: Making SSD-Backed KV Cache Practical](https://arxiv.org/abs/2605.03375)\n\n### Energy and Data Movement\n\n[Computing's Energy Problem (and what we can do about it): Mark Horowitz, ISSCC 2014](https://ieeexplore.ieee.org/document/6757323)\n\n[EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://arxiv.org/abs/1602.01528)\n\n[Practical Near-Data Processing for In-Memory Analytics Frameworks](https://csl.stanford.edu/~christos/publications/2015.ndp.pact.pdf)\n\n### Out-of-Core Model Inference\n\n[Qwen 3.8-27B on a 16 GB Mac mini (Manjunath Janardhan, Medium)](https://medium.com/@manjunath.shiva/qwen-3-8-27b-on-a-16-gb-mac-mini-alibabas-new-vision-model-fully-in-memory-f3aaaacbfeb4): the\ncompression strategy in practice, 55.6 GB to 11.55 GB, fully in memory, with a\ncandid account of where the compressed build falls short as an agent.\n\n[Kimi K3 (Moonshot AI)](https://huggingface.co/moonshotai): the model itself, ~2.8T\nparameters, 896 experts per layer, 16 activated per token. Released July 2026.\n\n[ kimi-k3-in-c](https://github.com/FareedKhan-dev/kimi-k3-in-c): an\n\n**independent** C99 CPU implementation, not produced or endorsed by Moonshot AI. Source of the memory ladder, the 8.24 GB peak RSS figure, and the reported I/O share of execution time.", "url": "https://wpnews.pro/news/route-the-work-not-just-the-data-gpus-cpus-and-the-rise-of-ai-native-storage", "canonical_source": "https://research.triunalabs.com/articles/ai-native-ssd/", "published_at": "2026-08-20 17:54:19+00:00", "updated_at": "2026-08-20 18:17:11.891529+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-research", "artificial-intelligence"], "entities": ["NVIDIA", "B200", "Micron", "6600 ION SSD"], "alternates": {"html": "https://wpnews.pro/news/route-the-work-not-just-the-data-gpus-cpus-and-the-rise-of-ai-native-storage", "markdown": "https://wpnews.pro/news/route-the-work-not-just-the-data-gpus-cpus-and-the-rise-of-ai-native-storage.md", "text": "https://wpnews.pro/news/route-the-work-not-just-the-data-gpus-cpus-and-the-rise-of-ai-native-storage.txt", "jsonld": "https://wpnews.pro/news/route-the-work-not-just-the-data-gpus-cpus-and-the-rise-of-ai-native-storage.jsonld"}}