{"slug": "high-bandwidth-flash-the-full-report", "title": "High Bandwidth Flash: The Full Report", "summary": "High Bandwidth Flash (HBF), a NAND-based memory stack with read bandwidth comparable to HBM4 but 10x the capacity, is set to sample from Sandisk in late 2026 and appear in AI inference devices by early 2027. By stacking NAND dies with TSVs and parallel reads, HBF overcomes NAND's slow latency to serve inference decode workloads, potentially disrupting DRAM and NAND markets.", "body_md": "# High Bandwidth Flash: The Full Report\n\n### Why HBF for inference decode fits, what it does to the DRAM and NAND markets, a worked cost model, who moves first, and risks\n\nThere’s been a lot of chatter about High Bandwidth Flash (HBF) recently.\n\nIf you’re not up to speed, the basic idea is that HBF is a stack of NAND dies built the way an HBM stack is built; dies stacked vertically, wired together with through-silicon vias (TSVs), and sitting right next to the GPU on the package interposer:\n\nWhat’s interesting is that it has the same read bandwidth as an HBM4 stack, but with roughly 10x the capacity. And it’s made of NAND, not DRAM like HBM. (*NAND is the cheap stuff.)*\n\nThe first samples of the memory itself are expected very soon from Sandisk, sometime in the second half of 2026. Samples of the first AI inference devices built with HBF follow in early 2027.\n\nEverything has tradeoffs, flash too. We’ll look at why those tradeoffs aren’t so bad for inference decode workloads, and the impact HBF will have on the memory market.\n\n**Table of contents**\n\nFlash is storage. But could it work for memory?\n\nWeight memory is what drives GPU count\n\nDRAM is no longer scaling well. But NAND is\n\nCMOS directly Bonded to Array: how the HBF stack is built (for the device physics layer, see Vik’s\n\n[High Bandwidth Flash: NAND’s Bid for AI Memory](https://www.viksnewsletter.com/p/high-bandwidth-flash-nands-bid-for-ai))[Paid] Inference decode fits this profile; training does not\n\n[Paid] Four ways HBF attaches to a GPU, each with a different supply chain winner\n\n[Paid] The disaggregated inference cost model\n\n[Paid] What HBF does to the memory markets\n\n[Paid] Nvidia is the certification gate; custom silicon moves first\n\n[Paid] Competitive landscape: Sandisk, SK Hynix, Samsung, YMTC\n\n[Paid] The patent: a processor bonded to NAND\n\n[Paid] HBM is infeasible for edge devices; HBF is not\n\n[Paid] Timeline and risks\n\n## Flash is storage. But could it work for memory?\n\nNAND flash has traditionally been used for information *storage*, i.e. where data lives when it’s not being used. *Cheap, dense, and non-volatile. But slooooooowwwww....*\n\nBut *memory* is where the accelerator keeps information handy during computation, and it has to be fast enough that compute never waits. SRAM is used for memory and can be read very quickly, on the order of about a nanosecond. Next in the memory hierarchy is DRAM, which is slower to read at ~100 nanoseconds.\n\nBut storage like NAND takes about a hundred microseconds, roughly 1,000x slower than DRAM. By definition that’s *storage* and not *memory*, right? It takes way too long to read, but good for long term storage. That won’t work for computation... way too slow! Compute would just be sitting idle, waiting forever for data.\n\n**So how could NAND flash possibly be used as memory?**\n\nWell, could there be a way to get the right data to the accelerator in time for computation, even with NAND’s slow reads? *If you start the read it WAY before the accelerator needs the data, it could work?*\n\nIn addition to intelligently scheduling when data is requested, one can also try to achieve high bandwidth from NAND. Bandwidth is the amount of data that arrives per unit of time. So if you know the read is going to take a long time, well, might as well have a bunch of reads running in parallel, if possible, right?\n\nHBF unlocks high bandwidth by stacking NAND dies to increase the “width” or the parallelism. Each die is divided into many sub-arrays, which are small blocks of NAND that can each be read at the same time, independently of one another. An HBF stack places 16 of these dies behind a single interface, thousands of bits wide, so a huge number of sub-arrays are available to read at once. Any single read still takes thousands of nanoseconds, but thousands of reads can run in parallel, so a lot of data can be moved simultaneously.\n\nThe 16 flash dies are stacked and connected with through-silicon vias (TSVs). A controller logic die is then bonded directly onto the NAND array. That logic die schedules the parallel sub-array reads and drives the results out over the wide interface to the GPU. The bonded die is called “CMOS directly Bonded to Array” or CBA:\n\nThe logic + HBF stack is on a high-bandwidth interposer. The resulting HBF delivers 1.6 TB/s of read bandwidth, which is the same as an HBM4 stack at the JEDEC spec’s 6.4 Gb/s operating point:\n\nTo underscore the importance of what the packaging is doing here, compare against the NAND you can buy today. Conventional flash ships about 14 GB/s behind a PCIe 5.0 NVMe controller. Packaged as HBF, the same material delivers 1.6 TB/s. Roughly 100x the bandwidth, from packaging alone. *Bullish advanced packaging!*\n\nAgain, HBF’s latency is still 10-100x slower than HBM. But if the accelerator knows what data it needs in advance, it can prefetch it and avoid waiting on any single slow read. The argument for HBF is that inference decode is the perfect workload. *I’ll explain in more detail later for paid subscribers.*\n\n## Weight memory is what drives GPU count\n\n**What are the economic implications of HBF?**\n\nInference cost scales with GPU count, and for today’s massive frontier models, GPU count is often driven by memory capacity per GPU.\n\n*Why?*\n\nWell, you need enough HBM to hold all the weights, but HBM is co-packaged with the accelerator; a fixed amount of HBM is bonded into each GPU package. So you can’t add memory without adding GPUs. Hence, bigger models mean more GPUs.\n\nOf course, weights aren’t the only thing the accelerator needs to store in memory and acccess quickly and often. The KV cache and activations sit in memory too, and both stay in HBM or DRAM.\n\nBut the KV cache takes new writes every token; NAND’s endurance can’t handle that. NAND has much lower write endurace.\n\nAnd activations need low-latency random access that NAND is too slow to give.\n\nSo HBF is for storing model weights.\n\nAnd frontier weights are huge and always wanting to be even bigger. *Wouldn’t it be nice to hold the model in significantly cheaper memory than HBM?*\n\nRecall that a 70B parameter model at fp16 (2 bytes per parameter) requires 70 × 10⁹ × 2 = 140 GB just for weights. A 1T parameter model thus needs 1,000 × 10⁹ × 2 = 2 TB. But that’s a lot of HBM; today’s shipping HBM4 stacks hold 36 GB (12-Hi); 16-Hi parts push 48 GB, and [the JEDEC spec tops out at 64 GB](https://www.jedec.org/news/pressreleases/jedec%C2%AE-and-industry-leaders-collaborate-release-jesd270-4-hbm4-standard-advancing). So a large model needs many stacks, which means many GPUs. *Expensive!*\n\nThat also means more interconnect (to move data around all those GPUs), which requires more power and, ultimately, a higher cost per output token (watts and $).\n\nBut HBF can provide 512 GB of capacity per stack!\n\nSo 512 GB per stack versus 48 GB for HBM4 is roughly 10x the capacity at the same bandwidth. Sandisk states up to 8-16x across the HBM family, depending on which generation you compare against.\n\nInteresting! Super promising. Consider the implications of storing frontier models in a few HBF stacks rather than spread across many HBM stacks across many GPUs in a server or rack.\n\n*New opportunities arise too! Could on-premises, air-cooled deployments at enterprises have frontier-model weights capacity without needing so many expensive GPUs? *\n\n## CMOS directly Bonded to Array (CBA)\n\n**Let’s dig into the manufacturing of HBF a bit more.** *Dang, I seriously didn’t mean to be punny with that bit.*\n\nSandisk stacks 16 BiCS NAND dies (BiCS, or Bit Cost Scalable, is just its brand of 3D NAND) with TSVs (*the same way HBM is built*) and bonds the controller onto the array:\n\nThe result matches HBM4’s footprint, height, and power envelope, so it can drop into the same interposer slot an HBM stack sits in today. It speaks nearly the same electrical interface, too, though [the host memory controller has to change](https://blocksandfiles.com/2025/02/12/sandisk-spills-its-technolgy-futures-beans/), so it isn’t quite drop-in. And being NAND, it’s non-volatile (it holds data with the power off), so unlike DRAM it needs no constant refresh power.\n\nOne open question is the NAND cell type. NAND can pack more than one bit into each memory cell:\n\nHighly recommend this explainer from Vik:\n\nNote that the fewer bits each NAND cell packs, the faster and more durable the cell is, but at the tradeoff of density.\n\nQLC (quad-level cell) holds four bits, cheap and dense, but is the quickest to wear out. SLC (single-level cell) holds one, yet is far more durable.\n\nSandisk hasn’t said which HBF uses, but [Irrational Analysis](https://irrationalanalysis.substack.com/p/market-memo-hot-summer-topics) reported recently that industry sources tell him it will be SLC, which would improve HBF’s write endurance by an order of magnitude, but at the cost of density.\n\nSandisk makes the NAND, but turning 16 dies into a stack that matches HBM’s footprint, thermals, and interface is advanced packaging work... the same thing the big 3 HBM folks have already perfected. So SK Hynix is the stacking partner.\n\nSandisk brings the flash, SK Hynix brings the stack.\n\nThe two [began an OCP standardization effort in February 2026](https://www.sandisk.com/company/newsroom/press-releases/2026/2026-02-25-sandisk-and-sk-hynix-begin-global-standardization-of-next-generation-memory-solution-high-bandwidth-flash-hbf).\n\n*Standardization?*\n\nYou might expect Sandisk to want a monopoly on HBF, right? Actually, it wants other suppliers, so long as Sandisk is the biggest. After all, hyperscalers won’t design a sole-source part into a billion-dollar platform, so being the only supplier is a risk to HBF adoption.\n\nA ratified standard with multiple suppliers removes that risk and gets HBF designed in. Again, Sandisk would rather own a big share of a market that exists than all of one that never does. *It reminds me of with Credo and active electrical cables. Grow the category into a real market, and win the largest slice of it.*\n\n**So how does HBF compare to HBM? **\n\nHBF has the edge when it comes to capacity vs HBM4, but trails in most other respects:\n\n*Note the 512 GB capacity is 14x the 36 GB 12-Hi HBM4 shipping today, ~10.7x the 48 GB 16-Hi flagship, and 8x the 64 GB JEDEC max.*\n\nFor the same bandwidth, HBF has up to 8-16x the capacity at roughly 2x the power.\n\nYes, HBF has worse bandwidth per watt, but for model weight storage one can argue the metrics that matter most are capacity and capacity per watt, where HBF wins.\n\nSanDisk also claims a [similar cost to an HBM stack](https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/collateral/company/Sandisk-HBF-Fact-Sheet.pdf) despite 8-16x the capacity, which works out to roughly 10x lower cost per GB.\n\n*I thought NAND was way cheaper?!*\n\nThe raw NAND is cheap per bit, but an HBF *stack* isn’t cheap. Most of its cost is the same advanced packaging that makes HBM expensive like the 16-high TSV stacking, the CBA logic die, and the interposer. So you get HBM-stack pricing with ~10x the bits, not SSD pricing.\n\nOf course, HBF has weaknesses compared to HBM. *It’s all tradeoffs.*\n\nMicrosecond latency against HBM’s ~100 ns, a minimum read 128x larger, and write endurance that rules out anything write-heavy.\n\nObjections to HBF are primarily those three issues. But the painfulness of those weaknesses depends on the workload. And for decode inference, HBF isn’t so bad! *We’ll cover it more below.*\n\nInterestingly, John Carmack has been thinking about this too; yesterday’s X post has over 1M views:\n\nClick on it and read the whole thing! Here’s an important snippet:\n\n“model inference can have a deterministic memory access pattern. You don’t need ‘random access memory’ at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth.\n\nClearly, there’s merit to flash for weights for inference.\n\nThere are lots of nuances too.\n\nAnd many big questions. *How does HBF impact HBM demand? How does HBF impact the already hot NAND market?* I’ll address those and more.\n\n**What paid subscribers get in the rest of this piece:**\n\n**Why decode doesn’t care about NAND’s weaknesses****The four ways HBF attaches to a GPU:** direct HBM replacement, mixed HBM+HBF slots, HBM-as-cache, and disaggregated prefill/decode. Each implies a different GPU attachment rate and a different set of supply chain winners, and the earliest route to production isn’t the obvious one.**The disaggregated cost model, worked:** a table pricing the model weight tier for Llama 405B, DeepSeek-V3, and a 1T-class model. HBM vs HBF**What HBF does to the DRAM and NAND markets****Route to market:** Nvidia, hyperscaler custom silicon, what about ASIC startups and other merchant GPU vendors?**A Sandisk patent that hints at the long term roadmap for HBF****Timeline and risks**\n\nAnd more!", "url": "https://wpnews.pro/news/high-bandwidth-flash-the-full-report", "canonical_source": "https://www.chipstrat.com/p/high-bandwidth-flash-the-full-report", "published_at": "2026-07-07 20:30:12+00:00", "updated_at": "2026-07-07 20:39:04.856030+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "ai-research"], "entities": ["Sandisk", "SK Hynix", "Samsung", "YMTC", "Nvidia", "HBM4", "NAND", "DRAM"], "alternates": {"html": "https://wpnews.pro/news/high-bandwidth-flash-the-full-report", "markdown": "https://wpnews.pro/news/high-bandwidth-flash-the-full-report.md", "text": "https://wpnews.pro/news/high-bandwidth-flash-the-full-report.txt", "jsonld": "https://wpnews.pro/news/high-bandwidth-flash-the-full-report.jsonld"}}