Meta says storage is acting as a brake on AI because its performance increase has been slow compared to AI compute speed tripling every two years. Metadata access flows and data ingestion schemes need reinventing to remove the brakes
Director of Engineering Sidharth Bajaj and SW Development Engineer Venkatraghavan Srinivasan, writing in Meta’s AI Storage Blueprint at Scale document, say: “As a result, storage bottlenecks continue to be one of the primary contributors to GPU stalls for AI workloads, directly impacting expenditures and time to market.” Meta’s BLOB storage architecture has evolved to maximize GPU utilization and also maximize research velocity by reducing the amount of time researchers spend ingesting and moving data across regions, thus speeding their research efforts.
They say “Meta operates hundreds of exabyte-scale storage clusters that serve all of Meta’s external and internal products, including Facebook, Instagram, Reality Labs, Meta AI, Ads, Data Warehouse, and internal Databases. Our storage service exposes object storage, file systems, and block-device APIs, and these API abstractions are built on top of a horizontally scalable foundational block layer called Tectonic.
“The Tectonic layer is a regional, multi-tenant storage fabric that provides high durability and availability leveraging erasure-coding techniques, supports tiering across media types (e.g., HDD and flash), and manages smart placement of hot, cold, and warm data for efficient utilization of I/O across tenants.”
Meta trained Llama directly over the Tectonic block layer by exposing an NFS-like FileSystem interface on top of it. It has been moving away from file to BLOB (Binary Large OBject) storage because of the need for unified storage access to massive data lakes and high performance. BLOB storage layers operate on top of Tectonic and “expose a global, infinitely scalable storage fabric”
When multiple GPUs are operating an AI data- pipeline from storage drives, high-latency metadata fetches by the GPU host can stall a GPU, delaying the overall pipeline. The getObject API request flow in this case is depicted in the flow chart below;
The two engineers say: “After the request arrives at the API server, the server does many metadata lookups across the namelayer, volumeslayer, and containerlayer before resolving the path to a set of (blockId, offset, size) tuples.” (A tuple is an ordered sequence of elements)
After the lookups, the API server proxies (sends) the data from the Tectonic layer to the client.
Meta engineers rewrote the metadata subsystem and collapsed the metadata spread across different layers into one unified and flat schema backed by ZippyDB. This reduced the number of metadata lookups. The API server no longer proxies the data to the client. Instead they built “a fat client SDK that is capable of streaming bytes directly from storage servers to the clients.” IT has a Tectonic BlockClient embedded within it and can stream data from these blocks directly from Tectonic.
To cope with distributed regional deployment “we now deploy a regional BLOB-storage stack colocated with GPUs in every AI region.”
The new request flow is depicted in the right-hand diagram above.
To avoid hot traffic spikes, they developed a distributed data cache, using “spare memory on the GPU hosts as a distributed data cache for frequently and concurrently accessed data.” This resulted in an average cache hit rate of 80 percent on the distributed data cache, and the read-plan cache provides 1-2 ms access to metadata.
They also added hedged reads on the client side and dynamic concurrency control on the client SDK. The net result is that “the new BLOB-storage stack is now capable of serving AI workloads without causing GPU stalls.”
Comment
Hyperscaler Meta could rewrite the getObject API request flow because it was capable of the low-level engineering involved. General enterprises cannot do this and are dependent on schemes produced by collaborating GPU and storage suppliers, such as KV caching, which may not be optimized so well for GPU utilization and research velocity.
The data ingestion delay problem
Meta’s AI training job submitters, its researchers, need to send training dataset snapshots from BLOB storage to a regional GPU center, in a data file format, and this data ingestion can take hours. The datasets have a write-once, read-many format and this “rang a bell. What if we think of storage as a disk in a planet-scale computer and borrow ideas from the operating-system world? When a Linux process running on a CPU core attempts to read a file from disk, the operating system transparently hydrates data on demand across the various layers of the cache—page cache in memory and L2 and L1 CPU caches.”
The BLOB storage foundation uses disk drives. Data is copied from the devices to fast flash (SSDs).The two engineers say: “The core idea is to leverage the various on-host and off-host storage resources as a tiered cache with global BLOB-storage fabric backed by HDDs as the ultimate source of truth. Specifically, we leverage the memory and flash on the GPU host as L1 and L2 caches. And we leverage the regional BLOB-storage fabric backed by flash as the L3 cache.” The effects were dramatic;
The previous 150 minute data ingestion time shrank to 10 minutes and a prior 89 hour ingestion job was reduced to just over 3 hours.
Meta’s prior disk-based metadata subsystem and non-tiered data ingestion was slowing its AI development work down. Its engineers say: ”By rebuilding the metadata subsystem and by adopting a tiered caching architecture with prefetching/on-demand hydration, we are able to meet the needs of today’s workloads effectively.”
Comment
Meta's multi-tiered data ingestion scheme is similar in principle to Nvidia's KV caching scheme but operates at a. regional scale and not just inside a single AI Factory data center.
Citrini analyst Jukan points out that "traditional high-capacity storage has primarily been evaluated in terms of $/TB—in other words, how cheaply it can store large volumes of data." But if data arrives too slowly from storage and leaves GPUs idle, the cost of wasted GPU time may exceed any savings achieved on storage.
If the incremental cost of flash is less than the GPU idle time eliminated by flash times the cost per GPU-hour then "using flash is economically rational despite its higher cost per terabyte." Therefore "GPU-adjacent storage tiers should therefore be evaluated not only by $/TB, but also by the total system cost per effective GPU-hour. In other words, up to a certain ceiling, it could make economic sense to spend several times more on flash than we do today."