{"slug": "meta-reinvents-ai-metadata-flow-and-kv-caching", "title": "Meta reinvents AI metadata flow and KV caching", "summary": "Meta engineers have redesigned their AI storage architecture to eliminate metadata bottlenecks that cause GPU stalls, collapsing multiple metadata layers into a unified schema backed by ZippyDB and implementing a fat client SDK for direct data streaming. The new system, detailed in Meta's AI Storage Blueprint at Scale, achieves an 80% cache hit rate and 1-2 ms metadata access, addressing the storage performance lag that has been a primary contributor to GPU stalls in AI workloads.", "body_md": "# Meta reinvents AI metadata flow and KV caching\n\n[Meta](https://www.blocksandfiles.com/ai-ml/2024/03/13/meta-hooks-up-with-hammerspace-for-advanced-ai-infrastructure-project/1612092) says storage is acting as a brake on AI because its performance increase has been slow compared to AI compute speed tripling every two years. Metadata access flows and data ingestion schemes need reinventing to remove the brakes\n\nDirector of Engineering Sidharth Bajaj and SW Development Engineer Venkatraghavan Srinivasan, writing in Meta’s [AI Storage Blueprint at Scale](https://engineering.fb.com/2026/07/01/data-infrastructure/metas-ai-storage-blueprint-at-scale/) document, say: “As a result, storage bottlenecks continue to be one of the primary contributors to GPU stalls for AI workloads, directly impacting expenditures and time to market.” Meta’s BLOB storage architecture has evolved to maximize GPU utilization and also maximize research velocity by reducing the amount of time researchers spend ingesting and moving data across regions, thus speeding their research efforts.\n\nThey say “Meta operates hundreds of exabyte-scale storage clusters that serve all of Meta’s external and internal products, including Facebook, Instagram, Reality Labs, Meta AI, Ads, Data Warehouse, and internal Databases. Our storage service exposes object storage, file systems, and block-device APIs, and these API abstractions are built on top of a horizontally scalable foundational block layer called Tectonic.\n\n“The Tectonic layer is a regional, multi-tenant storage fabric that provides high durability and availability leveraging erasure-coding techniques, supports tiering across media types (e.g., HDD and flash), and manages smart placement of hot, cold, and warm data for efficient utilization of I/O across tenants.”\n\nMeta trained Llama directly over the Tectonic block layer by exposing an NFS-like FileSystem interface on top of it. It has been moving away from file to BLOB (Binary Large OBject) storage because of the need for unified storage access to massive data lakes and high performance. BLOB storage layers operate on top of Tectonic and “expose a global, infinitely scalable storage fabric”\n\nWhen multiple GPUs are operating an AI data-loading pipeline from storage drives, high-latency metadata fetches by the GPU host can stall a GPU, delaying the overall pipeline. The getObject API request flow in this case is depicted in the flow chart below;\n\nThe two engineers say: “After the request arrives at the API server, the server does many metadata lookups across the namelayer, volumeslayer, and containerlayer before resolving the path to a set of (blockId, offset, size) tuples.” (A tuple is an ordered sequence of elements)\n\nAfter the lookups, the API server proxies (sends) the data from the Tectonic layer to the client.\n\nMeta engineers rewrote the metadata subsystem and collapsed the metadata spread across different layers into one unified and flat schema backed by [ZippyDB](https://engineering.fb.com/2021/08/06/core-infra/zippydb/). This reduced the number of metadata lookups. The API server no longer proxies the data to the client. Instead they built “a fat client SDK that is capable of streaming bytes directly from storage servers to the clients.” IT has a Tectonic BlockClient embedded within it and can stream data from these blocks directly from Tectonic.\n\nTo cope with distributed regional deployment “we now deploy a regional BLOB-storage stack colocated with GPUs in every AI region.”\n\nThe new request flow is depicted in the right-hand diagram above.\n\nTo avoid hot traffic spikes, they developed a distributed data cache, using “spare memory on the GPU hosts as a distributed data cache for frequently and concurrently accessed data.” This resulted in an average cache hit rate of 80 percent on the distributed data cache, and the read-plan cache provides 1-2 ms access to metadata.\n\nThey also added hedged reads on the client side and dynamic concurrency control on the client SDK. The net result is that “the new BLOB-storage stack is now capable of serving AI workloads without causing GPU stalls.”\n\n**Comment**\n\nHyperscaler Meta could rewrite the getObject API request flow because it was capable of the low-level engineering involved. General enterprises cannot do this and are dependent on schemes produced by collaborating GPU and storage suppliers, such as KV caching, which may not be optimized so well for GPU utilization and research velocity.\n\n**The data ingestion delay problem**\n\nMeta’s AI training job submitters, its researchers, need to send training dataset snapshots from BLOB storage to a regional GPU center, in a data loading file format, and this data ingestion can take hours. The datasets have a write-once, read-many format and this “rang a bell. What if we think of storage as a disk in a planet-scale computer and borrow ideas from the operating-system world? When a Linux process running on a CPU core attempts to read a file from disk, the operating system transparently hydrates data on demand across the various layers of the cache—page cache in memory and L2 and L1 CPU caches.”\n\nThe BLOB storage foundation uses disk drives. Data is copied from the devices to fast flash (SSDs).The two engineers say: “The core idea is to leverage the various on-host and off-host storage resources as a tiered cache with global BLOB-storage fabric backed by HDDs as the ultimate source of truth. Specifically, we leverage the memory and flash on the GPU host as L1 and L2 caches. And we leverage the regional BLOB-storage fabric backed by flash as the L3 cache.” The effects were dramatic;\n\nThe previous 150 minute data ingestion time shrank to 10 minutes and a prior 89 hour ingestion job was reduced to just over 3 hours.\n\nMeta’s prior disk-based metadata subsystem and non-tiered data ingestion was slowing its AI development work down. Its engineers say: ”By rebuilding the metadata subsystem and by adopting a tiered caching architecture with prefetching/on-demand hydration, we are able to meet the needs of today’s workloads effectively.”\n\n##### Comment\n\nMeta's multi-tiered data ingestion scheme is similar in principle to Nvidia's [KV caching scheme](https://www.blocksandfiles.com/ai-ml/2026/03/30/nvidia-and-its-partners-kv-cache-extenders/5209284) but operates at a. regional scale and not just inside a single AI Factory data center.\n\nCitrini analyst Jukan [points out](https://x.com/jukan05/status/2081175404396921164) that \"traditional high-capacity storage has primarily been evaluated in terms of $/TB—in other words, how cheaply it can store large volumes of data.\" But if data arrives too slowly from storage and leaves GPUs idle, the cost of wasted GPU time may exceed any savings achieved on storage.\n\nIf the incremental cost of flash is less than the GPU idle time eliminated by flash times the cost per GPU-hour then \"using flash is economically rational despite its higher cost per terabyte.\" Therefore \"GPU-adjacent storage tiers should therefore be evaluated not only by $/TB, but also by the total system cost per effective GPU-hour. In other words, up to a certain ceiling, it could make economic sense to spend several times more on flash than we do today.\"", "url": "https://wpnews.pro/news/meta-reinvents-ai-metadata-flow-and-kv-caching", "canonical_source": "https://www.blocksandfiles.com/object/2026/07/27/meta-reinvents-ai-metadata-flow-and-kv-caching/5278805", "published_at": "2026-07-27 16:18:33+00:00", "updated_at": "2026-07-27 16:23:11.638192+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-research"], "entities": ["Meta", "Sidharth Bajaj", "Venkatraghavan Srinivasan", "Tectonic", "ZippyDB", "Llama", "Facebook", "Instagram"], "alternates": {"html": "https://wpnews.pro/news/meta-reinvents-ai-metadata-flow-and-kv-caching", "markdown": "https://wpnews.pro/news/meta-reinvents-ai-metadata-flow-and-kv-caching.md", "text": "https://wpnews.pro/news/meta-reinvents-ai-metadata-flow-and-kv-caching.txt", "jsonld": "https://wpnews.pro/news/meta-reinvents-ai-metadata-flow-and-kv-caching.jsonld"}}