Data movement across modern AI workloads
Demand for AI compute is growing faster than modern infrastructure can keep up with [1]. Labs are hungry for GPUs, and oftentimes their workloads must run wherever capacity is available. In a world where every GPU hour carries opportunity cost, this limitation creates painful inefficiencies across storage and compute.
Running GPU workloads across different clusters, regions, cloud providers, and on-premises facilities introduces challenges around time, cost, and operations. Each new configuration brings security and reliability risks, along with often-overlooked egress costs. When data or artifacts are not available where a workload runs, expensive accelerators can sit idle while teams diagnose and recover from the mismatch. This complexity burdens AI researchers whose attention should be on experimental methods and model outcomes, not infrastructure overhead.
More Compute Creates a Placement Problem
More Compute Creates a Placement Problem
AI data-center capacity is expanding rapidly, increasing the number of environments where workloads may run [1][2]. This flexibility matters: teams can choose capacity based on availability, cost, and performance requirements. But compute capacity and the data needed to unlock it are not co-located by default.
Datasets, model weights, container images, and other assets are typically anchored in massive storage systems that have matured over many years. The authoritative copy of a given asset may reside in an object store, a data lake, or a registry. As a result, a data-hungry GPU’s utilization is dependent on interactions with these backing data stores. When selecting a cluster, the challenge is not simply finding available GPUs, but also determining whether inputs can reach the cluster in the form the workload expects.
Even after reserving a cluster, internal accelerators will sit idle until required workload data arrives. If data arrives slowly, or in incomplete or unreliable forms, compute capacity will not be used productively. The emerging constraint is not whether compute exists, but whether the right data is available at the right time and place.
The Data Lifecycle of AI Workloads
The Data Lifecycle of AI Workloads
AI training has two flavors: pre-training builds a base model’s broad capabilities from large corpora, while post-training refines the model for specific behaviors, capabilities, and safety objectives. Research teams run pre-training and post-training workloads across reserved and on-demand GPU clusters in a wide range of compute environments. Training can shift across runs to new capacity, or to a different cluster from the reserved set, based on available GPUs.
Source-of-truth storage provides a current, shared record of datasets, model artifacts, evaluations, and outputs across researchers, experiments, and compute environments. Without that shared record, teams accumulate fragmented copies, uncertain versions, and results that are difficult to reuse or compare. That shared record creates coordination value only when workloads can quickly and reliably obtain the correct version in the environment where work will occur, then return the outputs for others to reuse.
Training therefore has a consistent data path: inputs flow from a common source into the environment where work occurs, and checkpoints, logs, and artifacts flow back so the organization can coordinate future work from the same up-to-date foundation.
Pre-training
Pre-training begins with a dependency set: a large corpus, base-model artifacts, containers, configuration, and prior checkpoints must be available near the selected cluster. The job is ready only when those inputs are complete, correct, and available at the required throughput; a missing shard, inaccessible checkpoint, or slow container-image pull can delay its start. The right placement strategy depends on corpus size and shape, allocation duration and capacity, and the storage systems already available.
Compute-adjacent caches can support highly stable runs, but complexity increases when data or checkpoints change, caches must be warmed and evicted, or another cluster’s job needs the same data. The selected compute needs a coherent version of the workload’s inputs and predictable I/O performance throughout the run; storage performance can otherwise stall the training pipeline [3][4]. Tokenization, preprocessing, data s, and intra-cluster I/O can still limit training throughput, but only after the necessary inputs are in place.
Post-training
Post-training is more iterative and ranges from supervised or preference optimization over fixed datasets to reinforcement-learning loops that continuously generate and score trajectories. In agentic RL, rollout engines interact with task environments, sandboxes, and tools; verifiers or judges produce rewards; training workers update the model; and updated weights may then need to reach the rollout fleet. Teams can colocate training and generation on the same GPUs or disaggregate them across separate pools [5][6][7]. Still, either approach requires task environments, model versions, trajectories, rewards, checkpoints, and evaluation results to be available at the right stage. The bottleneck is therefore not only initial dataset hydration, but keeping this coupled loop supplied, synchronized, and recoverable.
Research agents increasingly operate inside this experimental loop. They can accelerate it by preparing inputs, running evaluations, inspecting failures, and generating follow-on work. But these agents also multiply the number of short-lived environments that need access to data and artifacts. Their usefulness depends on quickly creating isolated working copies, preserving only the results that matter, and avoiding turning storage shared across agents into an uncontrolled write surface.
Across both pre-training and post-training, the input path and output path must work together. Checkpoints let a run resume after an interruption or continue on another cluster, while logs, intermediate artifacts, evaluation results, and outputs support comparison and follow-on work.
Depending on the environment, staging data and artifacts can materially delay the interval between job submission and execution. Reducing these bottlenecks makes each GPU-hour go further and increases research velocity by shortening the path from an experiment to the evidence needed for the next model design decision.
Inference’s Continuous Loop
While training commonly runs in isolated compute environments, where a single run can tolerate setup time and resume after failure recovery, inference serves active customer traffic, so its environment has stricter performance and availability requirements. A serving process needs many artifacts: model weights, tokenizer, configuration, adapters, runtime libraries, and more. Caching can make steady-state serving relatively straightforward in the simple case. The harder problem is keeping complete model bundles available and consistent during cold starts, autoscaling, failover, deployments, rollbacks, and shifts in the active model set. Autoscaling, failover, and cold starts make model placement a continuous operational concern, not an exceptional event. When demand grows, a new serving instance starts, or a serving environment becomes unavailable, capacity may shift to another cluster. A platform serving a large model catalog cannot assume every model and its dependencies are already resident everywhere; hosting every available model on every GPU node is impractical. The operational question is whether the requested model and its complete serving bundle can load quickly enough onto replacement or newly scaled capacity to serve traffic.
Model rollouts and rollbacks create a related requirement. A new version must be placed alongside its tokenizer, configuration, adapters, and other supporting artifacts wherever it will serve traffic. Because a given rollout might be reversed, the prior version must remain available as well. The challenge is not only choosing a version, but making the complete serving bundle available consistently across the capacity that may receive requests.
Batch inference follows a different timing pattern but has the same data-movement dependency. Large input datasets must be staged near available workers, which then generate predictions, scores, embeddings, or other outputs that return to the durable store for later use. Since batch work can be assigned to whichever capacity is available, the speed and reliability of this movement affect both how quickly a run can start and which compute pool can run it.
No One Size Fits All Storage System
No single storage product can serve all AI workload shapes. Instead, teams maintain heterogeneous storage topologies to provide distinct advantages depending on a given workload’s needs. Durable object stores hold authoritative data, compute-adjacent systems support active working sets, and repositories distribute artifacts or serve structured data. The right question is not which interface is best overall, but which tradeoffs fit the work: durability, performance, structure, access semantics, and cost.
Storage systems and tradeoffs:
| Storage system | What it is good at | Tradeoff for AI workloads |
|---|---|---|
| Object storage | Durable source of truth with a flat namespace for large amounts of unstructured data, such as AI artifacts. | Accessed through a remote object API. Large parallel reads can achieve high throughput, but latency-sensitive, small, or metadata-heavy access patterns can be slow. |
| Network file systems | A shared POSIX-style working set for many workers near a cluster, including active training data and checkpoints. | Performance depends on the file system's location, provisioned capacity, network path, and metadata capacity. Creates, opens, listings, and other metadata-heavy operations can become a separate bottleneck due to the hierarchical structure. |
| Node-local NVMe or ephemeral disks | Fast scratch space, caches, shuffles, etc. | Capacity is fixed-size and tied to an individual node. Ephemeral disks can lose data when a node fails or is terminated, while maintaining enough local capacity across a fleet adds cost. |
| Model and container registries | Versioned, immutable distribution of packaged models, images, and related artifacts. | Registries are designed around manifests and content-addressed blobs, not fine-grained training-data access. They solve artifact management, not general data placement. |
| Operational databases | Mutable metadata, task state, permissions, catalogs, and indexed record access. | They are usually not the right delivery layer for multi-terabyte corpora used in AI; transactional, connection-oriented access differs from bulk data delivery. |
| Data lakes and lakehouse tables | Large raw or curated datasets, snapshots, and shared analytical or training source data. | They can be an excellent source of truth, but file layout and table metadata matter. Workers may still need data materialized, sharded, cached, or staged in a runtime-friendly form. |
| Data warehouses | SQL analytics, transformations, aggregations, and the creation of curated evaluation or training inputs. | They optimize for analytical queries, not necessarily for exposing a training-ready file layout to a GPU cluster. Exporting or materializing results can become another placement step. |
These systems are complementary, not interchangeable. Sources [8]-[11] below describe object-store performance and locality, ephemeral-storage lifecycle, registry semantics, and warehouse materialization. A dataset may be authoritative in object storage or a data lake, actively read from a shared file system or local cache, and accompanied by models distributed through a registry. Data movement connects those roles so that a training run, inference service, or agent can use the right data in the right environment.
Compute Is Useful Only When the Inputs Are Ready
More AI compute across more locations creates more potential execution locations and more complexity for teams to manage. This capacity becomes useful only once a workload can start, access its input data efficiently, preserve recovery state, return its results, and clean up afterward.
Data placement is not a separate chore. It is an essential part of the workload lifecycle. When done well, it enables AI development to scale with GPU capacity.
Stay informed
Get technical insights on AI infrastructure and occasional company announcements. Join the mailing list.
Appendix
[1] International Energy Agency, Energy and AI (2025): https://www.iea.org/reports/energy-and-ai/energy-demand-from-ai. Data-center electricity consumption is projected to roughly double by 2030; the report also notes that energy infrastructure typically has longer planning and construction cycles than data centers.
[2] JLL, Global data center sector to nearly double to 200 GW amid AI infrastructure boom (2026): https://www.jll.com/en-us/newsroom/global-data-center-sector-to-nearly-double-to-200gw-amid-ai-infrastructure-boom.
[3] Meta Engineering, Meta’s AI Storage Blueprint at Scale (2026): https://engineering.fb.com/2026/07/01/data-infrastructure/metas-ai-storage-blueprint-at-scale/. Describes training-data ingestion that can take hours, storage-fetch tail latency that stalls GPUs, and caches used to reduce regional data-access delays.
[4] Svogor et al., Profiling and Improving the PyTorch Data for High-Latency Storage (2022): https://arxiv.org/abs/2211.04908. A workload-specific experiment showing data- and remote-storage latency can materially increase GPU idle time.
[5] Laminar: A Scalable Asynchronous RL Post-Training Framework (2025): https://arxiv.org/abs/2510.12633. Discusses asynchronous RL post-training, rollout generation, and the cost of global weight synchronization.
[6] ECHO-2: A Large Scale Distributed Rollout Framework for Cost-efficient Reinforcement Learning (2026): https://arxiv.org/abs/2602.02192. Describes repeated rollout generation, reward evaluation, centralized learning, and weight dissemination.
[7] Mercor, Training Frontier Knowledge Work Agents (2026): https://www.mercor.com/blog/training-frontier-knowledge-work-agents-a-397b-rl-training-guide-with-skyrl/. A production-oriented example of agentic RL using task environments, sandboxed trials, verifiers, rollouts, and asynchronous training.
[8] AWS, Performance Guidelines for Amazon S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html. Covers regional locality, concurrent access, range requests, caching, and latency-sensitive request handling.
[9] Kubernetes, Ephemeral Volumes: https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/. Documents node-local ephemeral storage and its lifecycle tied to a scheduled workload.
[10] Open Container Initiative, Distribution Specification: https://specs.opencontainers.org/distribution-spec/. Defines registries as manifest-and-blob distribution systems.
[11] Snowflake, Unload into a Snowflake Stage: https://docs.snowflake.com/en/user-guide/data-unload-snowflake. Documents the materialization of warehouse data into staged files before local use.