{"slug": "multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo", "title": "Multi-Region training with Amazon SageMaker HyperPod and Qumulo", "summary": "Amazon Web Services and Qumulo reported that a SageMaker HyperPod training cluster in the US West (Oregon) Region (us-west-2) reading data remotely from the US East (Ohio) Region (us-east-2) through Qumulo's Cloud Data Fabric matched the throughput of a co-located hub cluster, reaching 115–117 samples/sec despite 60 ms of network latency. The validation used Cloud Native Qumulo for storage, VPC peering for cross-Region access, and Qumulo's NeuralCache predictive caching, with throughput converging within the first 100–150 batches and GPU utilization rising from 80–90 percent during warmup to 98–100 percent afterward. Qumulo said the architecture delivers 99 percent GPU utilization and sub-3 ms data operations on a single-Region deployment, letting teams train on datasets in another AWS Region or on-premises without copying petabytes of data or changing training code.", "body_md": "## [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Multi-Region training with Amazon SageMaker HyperPod and Qumulo\n\nWith [Amazon SageMaker HyperPod](https://aws.amazon.com/sagemaker/ai/hyperpod/) and [Qumulo](https://qumulo.com/product/aws), you can place training compute in one AWS Region and keep your dataset in another. Training large AI models requires massive GPU capacity, but your ideal compute resources and your training data don’t always reside in the same AWS Region. Accessing data across Regions adds network latency and transfer costs. Teams face a choice: either replicate petabytes of data across Regions, or absorb cross-Region latency on every read and accept slower training. This pairing can help tackle that trade-off, letting teams keep frontier models current without moving data or sacrificing throughput.\n\nIn this post, we present a solution to this challenge, explain the architecture, and share validation results from a cross-Region training run. By pairing Amazon SageMaker HyperPod with [Cloud Native Qumulo (CNQ)](https://qumulo.com/product/aws) and Qumulo’s [Cloud Data Fabric (CDF)](https://qumulo.com/product/cloud-data-fabric/), training jobs can read datasets stored in another AWS Region or on-premises without copying the data, changing the code, or sacrificing throughput. We validated the approach by running the same training job independently on two clusters. The hub cluster runs in the US East (Ohio) Region (us-east-2), co-located with the data. The spoke cluster runs in the US West (Oregon) Region (us-west-2) and reads the data remotely through CDF with 60 ms of network latency. We then compared their throughput. After a short warmup, the spoke cluster matched the hub’s performance.\n\nPairing Amazon SageMaker HyperPod with CNQ delivers two measurable outcomes:\n\n- **Single-Region training performance:** CNQ’s cloud-native architecture allows it to scale its performance independently of its data storage size. This results in full 99 percent GPU utilization and p5.48xlarge network saturation with sub-3 ms data operations.\n- **Remote clusters train at optimal utilization:** A HyperPod cluster running in a different Region from its data reaches the same throughput as a cluster co-located with the data (115–117 samples/sec) with no additional data orchestration needed. Throughput converges within the first 100–150 batches, and that one-time warmup is negligible at scale: under 1 percent of wall-clock time at 10,000+ batches, and under 0.1 percent at over 100,000. This ultimately results in 80–90 percent GPU utilization for the initial phase of training for 100–150 batches and then converges on 98–100 percent GPU utilization.\n\n## Solution overview\n\nAmazon SageMaker HyperPod provides the managed training infrastructure including resilient clusters with automatic health checks, node replacement, and checkpoint recovery. The solution pairs SageMaker HyperPod with Qumulo’s storage layer. We validated the solution with the following components:\n\n- **Single source of truth** : Store your training dataset in one AWS Region using Cloud Native Qumulo (CNQ).\n- **Cross-Region access** : Each HyperPod cluster mounts its local CNQ instance over Network File System (NFS). The CNQ spoke cluster retrieves data from the hub cluster over[Virtual Private Cloud (VPC)](https://aws.amazon.com/vpc/) peering, making the full dataset available without cross-Region access from the compute nodes.\n- **Intelligent data placement** : Qumulo’s Cloud Data Fabric (CDF) uses predictive caching through[NeuralCache](https://qumulo.com/product/neural-cache) . It learns your data loader access patterns and serves data from local Non-Volatile Memory Express (NVMe), making cross-Region latency transparent after an initial warmup period.\n\nFigure 1 shows the validated architecture. In the spoke Region, machine learning developers submit jobs through an [Amazon Elastic Kubernetes Service (Amazon EKS)](https://aws.amazon.com/eks/) orchestrator, which runs them on the HyperPod cluster compute nodes. Those nodes mount a local CNQ spoke over NFS. The CNQ hub in the second Region holds the single copy of the training data, and CDF projects that dataset to the spoke over VPC peering, so the compute nodes read from the same local mount path in both Regions.\n\n### About Qumulo Cloud Data Fabric\n\nCDF makes a single dataset on a *hub* Cloud Native Qumulo (CNQ) instantly visible and accessible to multiple *spoke* CNQ instances. This holds even on network links with more than 100 ms of latency. The primary component is Qumulo’s NeuralCache, which uses an AI model to predict the next 4 KB blocks a job will need and pre-caches them at the spoke. Combined with CNQ’s elastic performance, the dataset performs like local storage for the spoke clusters in this validated configuration, without the costs of full data replication.\n\nWe ran the same training job (1.02 billion-parameter LLaMA v3, two ml.p5.48xlarge instances per Amazon SageMaker HyperPod cluster, 16 H100 GPUs total) independently on each cluster. The following table compares the performance:\n\n| **Configuration** | **Training speed** | **Time to 999 batches** | **NeuralCache state** | \n| Hub (us-east-2 to us-east-2) | 116–117 samples/sec | **18.5 minutes** | N/A (local) | \n| Spoke, Warm Cache (us-east-2 to us-west-2) | 115–116 samples/sec | **18.5 minutes** | Converged | \n| Spoke, Cold Start (us-east-2 to us-west-2) | 95–115, then 116 samples/sec | ~19–20 minutes* | Warming then Converged | \n\n** Worst-case scenario with no prior runs. Batches 0–150 run 15–20 percent slower during NeuralCache warmup, then converge to hub-level performance within the first epoch.*\n\n## Understanding the architecture\n\nThe following sections introduce the validated architecture with Qumulo configured in the same Region as the HyperPod cluster as a hub and also as a spoke.\n\n### How Cloud Data Fabric and NeuralCache work\n\nCloud Data Fabric (CDF) keeps a single authoritative copy of your dataset on the hub cluster and projects it to spoke clusters as a unified Portable Operating System Interface (POSIX) namespace. When a spoke is created, CDF replicates the filesystem metadata to it, so the full namespace is browsable within seconds, before file data has moved. When the spoke mounts the filesystem, CDF services each read from the nearest valid source: local NVMe cache on a hit, the hub on a miss. CDF also uses advanced congestion control, which paces to measured bottleneck bandwidth and round-trip propagation time rather than backing off on packet loss. This is what keeps a long-haul link full where loss-based algorithms collapse. CDF sustains near-line-rate throughput on paths with round-trip times (RTTs) as high as 900 ms.\n\nNeuralCache is the predictive caching layer inside CDF. It observes the sequential 4 KB block reads issued by the training data loader and learns the access pattern. It then places data predictively before the job issues the request. Within the first 100–150 batches, the model learns to predict which blocks will be requested next and pre-fetches them from the hub into local NVMe storage on the spoke. Once warmed, 94–96 percent of reads are served from local cache at sub-5 ms latency. This results in lower latency operations for applications.\n\n### Hub configuration\n\nData and compute share the same Region. Qumulo delivers 1.0–1.3 GBps sustained read throughput (saturating the 10 Gbps network link) at 2–3 ms max read latency, keeping GPUs fully utilized throughout training. Co-locating data and compute matters because every data loader read travels only within the Region, so there’s no wide-area network hop on the critical path. CNQ sustains these numbers because its cloud-native architecture scales throughput with the size of the cluster rather than the size of the dataset. This is why a 4-node cluster can saturate the network link feeding the GPUs. Figure 2 shows this single-Region layout, with the HyperPod cluster, the Amazon EKS orchestrator, and the CNQ hub that stores the dataset all in one Region.\n\n### Spoke configuration (cold start)\n\nYour Amazon SageMaker HyperPod cluster runs in a different Region from where the data originates. It mounts the local Qumulo spoke, which handles fetching data from the hub. NeuralCache observes your data loader access patterns in real time and begins caching data locally. During the initial 100–150 batches, cross-Region latency is visible and training runs at 95–105 samples per second. Performance then converges to hub-equivalent levels (115–117 samples per second) for the remainder of the run. Figure 3 shows this cross-Region layout. The HyperPod cluster and a CNQ spoke run in one Region, and the CNQ hub that holds the dataset runs in a second Region, connected by CDF over VPC peering.\n\n### Spoke configuration (warm cache)\n\nSubsequent runs benefit from a fully warmed NeuralCache. With a 94–96 percent cache hit rate, data is served from local NVMe at sub-5 ms latency, delivering immediate hub-equivalent performance with GPU utilizations above 99 percent.\n\n## Walkthrough\n\nThis walkthrough provides a step-by-step process for configuring and running these same tests.\n\n### Prerequisites\n\nTo set up a representative environment, you need the following prerequisites:\n\n- An AWS account with Amazon SageMaker HyperPod enabled.\n- Deploy Qumulo Cloud Native in your primary Region (us-east-2, Ohio).\n- Sufficient quota for ml.p5.48xlarge instances (two per cluster).\n- Configure VPC peering between hub and spoke Regions.\n- Basic familiarity with PyTorch and distributed training concepts.\n\n### Step 1: Deploy Qumulo Cloud Native in your hub Region\n\nDeploy Qumulo Cloud Native in us-east-2 from [AWS Marketplace](https://aws.amazon.com/marketplace/). In the [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3/) bucket that hosts your deployment files, find the [AWS CloudFormation](https://aws.amazon.com/cloudformation/) template at templates/cnq-standard.`template.yaml` and copy its object URL. Create the stack from that URL, then choose r5.8xlarge instances and a 4-node cluster in your existing VPC.\n\n### Step 2: Configure VPC peering between Regions\n\nCreate a VPC peering connection between your hub (us-east-2) and spoke (us-west-2) VPCs. Then accept the connection in the peer Region and update the route tables in both Regions so that each VPC’s CIDR range routes to the peering connection. This gives Qumulo instances in the spoke Region a path to the Qumulo instances in the hub Region. Now, Amazon SageMaker HyperPod instances in the spoke Region (us-west-2) have access to data in the hub Region (us-east-2) through their local NFS mount. This is because both Qumulo clusters are connected with a private, low-latency path.\n\nVPC peering traffic between AWS Regions is encrypted in transit. Configure security groups on both the Qumulo and HyperPod instances to allow only the required NFS ports (TCP 2049) between the two VPCs.\n\n**Note:** Replace placeholder values (shown in angle brackets) with your own resource identifiers.\n\nAccept the peering connection from the spoke Region. Then update the route tables in both Regions so that each VPC’s CIDR block routes through the peering connection:\n\n### Step 3: Upload your training dataset to Qumulo\n\nMount Qumulo over NFS and sync your pre-tokenized dataset. We used the [C4 dataset](https://huggingface.co/datasets/allenai/c4), tokenized for LLaMA (48.4 million sequences packed to 4,096-token windows). Qumulo presents a unified POSIX namespace, so hub and spoke instances use the same mount path.\n\n### Step 4: Create your HyperPod clusters\n\nCreate Amazon SageMaker HyperPod clusters in both Regions with ml.p5.48xlarge instances. HyperPod manages the underlying infrastructure (health monitoring, automatic node replacement, and checkpoint-based recovery), so your training run survives hardware failures without manual intervention. Configure a lifecycle script that mounts Qumulo at startup. The spoke cluster uses the same lifecycle script and mounts the local Qumulo spoke instance over NFS. The Qumulo spoke retrieves data from the hub transparently over VPC peering.\n\nEach cluster is defined by a CreateCluster request file. The following `hub-cluster-config.json` creates a two-node ml.p5.48xlarge instance group attached to your Amazon EKS control plane, with automatic node recovery turned on and a lifecycle script that mounts Qumulo at startup:\n\nCreate `spoke-cluster-config.json` from the same structure, changing ClusterName, the Region in the Amazon EKS cluster ARN, and the VpcConfig values to the spoke Region resources. Each HyperPod cluster must sit in the same VPC as its Amazon EKS control plane, and the subnets must be private.\n\n### Step 5: Configure the Container Storage Interface (CSI) driver to mount Qumulo\n\nInstall the Qumulo CSI driver using your preferred installation method. This example uses Helm:\n\nVerify the CSI driver pods are running before proceeding to mount:\n\nCreate a Static Persistent Volume like this YAML: The accessModes value of ReadWriteMany lets every training node mount the same volume at once, which is what allows the hub and spoke clusters to share one dataset. The server attribute points at the local Qumulo instance in each Region. The same manifest works in both hub and spoke clusters by changing only that value to the local Qumulo address.\n\nFinally, create the Persistent Volume Claim:\n\n### Step 6: Run training on your Amazon SageMaker HyperPod clusters\n\nDeploy a Test Pod using kubectl with the following YAML:\n\nLaunch distributed training using PyTorch torchrun, pointing dataset and checkpoint paths to the Qumulo mount. No code changes are needed between hub and spoke runs. The following `job.yaml` defines the run as a Kubeflow PyTorchJob with one worker per ml.p5.48xlarge instance. It mounts the Qumulo PersistentVolumeClaim at /csi and passes the dataset and checkpoint paths on that mount to the training script:\n\nApply the manifest to the cluster:\n\nConfigure 64 data loader workers with prefetch_factor=4, keeping 256 batches in flight. This masks latency during cache warmup while Fully Sharded Data Parallel (FSDP) overlaps gradient synchronization with I/O.\n\n## Performance results\n\nThe performance results for relevant tests are shown in the following sections, illustrating I/O operations per second (IOPS), throughput, and latency per test run. All tests use PyTorch 2.1 FSDP, batch size 24 per GPU, and 4,096-token sequence length.\n\n### Hub configuration\n\nThe hub delivers consistent performance throughout training: 1.0–1.3 GBps sustained read throughput, 2–3 ms read latency, and 98–100 percent GPU utilization. These numbers are the baseline that the spoke configurations are measured against, because the hub has no wide-area network in the data path. The Qumulo dashboard (Figure 4) displays a stable IOPS profile across a full training run, with the periodic spikes corresponding to checkpoint writes rather than read stalls. The dashboard stacks three time series over the same window: IOPS, throughput, and latency. Throughput holds a flat band just above 1.0 GBps for the length of the run, and the latency trace stays flat alongside it.\n\n### Spoke configuration with cold start (worst case)\n\nNeuralCache begins learning from batch zero. As it observes the data loader, it identifies the sequential access pattern that FSDP training produces and starts prefetching the next blocks before the job requests them, which minimizes data operation latency. The cache hit rate climbs steadily. It stabilizes after the prefetcher is reliably staying ahead of the data loader. Convergence lands around batch 100–150 rather than at a fixed point, because it depends on how quickly the access pattern becomes predictable. These two phases illustrate this transition.\n\n- Phase 1 (batches 0–100): cross-Region latency dominates at over 60 ms, GPU utilization dips to 80–90 percent, and throughput runs at 500–800 MBps.\n- Phase 2 (batches 100–150): cache hit rate stabilizes, latency drops to 5–10 ms and training speed climbs to approximately 115 samples per second and 98–100 percent GPU utilization.\n\nFigure 5 shows the same transition on the Qumulo dashboard. IOPS and throughput ramp upward through the warmup window while read latency falls, and all three traces flatten once NeuralCache converges.\n\nAt production scale, this one-time penalty diminishes rapidly for geo-distributed training:\n\n| **Training Scale** | **Total Batches** | **Wall Clock Time** | **Cold Start Impact** | \n| Development run | 10,000 | ~3 hours | 0.81% | \n| Production training* | 100,000 | ~31 hours | 0.08% | \n| Large-scale pretraining* | 1,000,000 | ~13 days | 0.008% | \n\n** “Production training” and “Large-scale pre-training” numbers are extrapolated from observed cold-start duration. Actual results will vary by dataset size, access pattern, and network conditions.*\n\n## Clean up resources\n\nThis walkthrough provisions infrastructure in two Regions that continues to incur charges until you remove it. When you finish, tear down the resources in roughly the reverse order you created them so that dependencies are released cleanly.\n\n- Delete the PersistentVolumeClaim and PersistentVolume, then uninstall the Qumulo CSI driver (helm uninstall qumulo-csi -n kube-system).\n- Delete both Amazon SageMaker HyperPod clusters (aws sagemaker delete-cluster) in us-east-2 and us-west-2. For the full procedure, see [Deleting a SageMaker HyperPod cluster](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-cli-command-delete-cluster.html) .\n- Delete the VPC peering connection and remove the routes you added to the route tables in both Regions. For the full procedure, see [Delete a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/delete-vpc-peering-connection.html) .\n- Delete the Qumulo Cloud Native deployment by removing its AWS CloudFormation stack, which also releases the underlying instances and storage. Before deleting the stack, verify any data you want to keep has been copied off the cluster. The CNQ storage volumes are deleted with the stack and are not recoverable. To delete, open the AWS CloudFormation console, select the stack you created for Qumulo in us-east-2, and choose **Delete** , as described in[Delete a stack from the CloudFormation console](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-delete-stack.html) . Alternatively, run`aws cloudformation delete-stack --stack-name <qumulo-stack-name> --region us-east-2` .\n\n## Conclusion\n\nBy pairing [CNQ](https://qumulo.com/product/cloud-native-qumulo) and [CDF](https://qumulo.com/product/cloud-data-fabric) with [Amazon SageMaker HyperPod](https://aws.amazon.com/sagemaker/ai/hyperpod/), you can run geo-distributed training from a single dataset without data replication. Our validation confirmed that a single copy of your data, with no replication overhead, delivers near-local latency and hub-equivalent throughput after a brief warmup. To avoid the cold start entirely, pre-warm the spoke before training begins by issuing a pre-warm request through the Qumulo API on the dataset inside your training job orchestration script with [Ray](https://www.ray.io/), [Amazon EKS](https://aws.amazon.com/eks/), [Slurm](https://slurm.schedmd.com/overview.html), or other tooling.\n\nTo extend this architecture, consider multi-spoke deployments accessing a single dataset from three or more Regions, hybrid configurations using [AWS Direct Connect](https://aws.amazon.com/directconnect/) for on-premises data, or lifecycle policies to archive checkpoints to an [Amazon S3 Glacier](https://aws.amazon.com/s3/storage-classes/glacier/) storage class.\n\nTo get started, see [Orchestrating SageMaker HyperPod clusters with Amazon EKS](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-eks.html) in the Amazon SageMaker AI Developer Guide. For the AWS CloudFormation templates, Terraform modules, and lifecycle scripts that provision clusters like the ones in this post, see the [SageMaker HyperPod EKS reference architecture](https://github.com/awslabs/awsome-distributed-ai/tree/main/architectures/sagemaker-hyperpod-eks) in the awsome-distributed-ai GitHub repository.\n\nFor more reading on Cloud Native Qumulo, read [Building a Cloud Native File System with Cloud Native Qumulo and Amazon S3](https://aws.amazon.com/blogs/apn/building-a-cloud-native-file-system-with-cloud-native-qumulo-and-amazon-s3/).\n\nFor more on the components used here, see [Introducing Amazon EKS support in Amazon SageMaker HyperPod](https://aws.amazon.com/blogs/machine-learning/introducing-amazon-eks-support-in-amazon-sagemaker-hyperpod/) and [Accelerate pre-training of Mistral’s Mathstral model with highly resilient clusters on Amazon SageMaker HyperPod](https://aws.amazon.com/blogs/machine-learning/accelerate-pre-training-of-mistrals-mathstral-model-with-highly-resilient-clusters-on-amazon-sagemaker-hyperpod/) on the [AWS Artificial Intelligence Blog](https://aws.amazon.com/blogs/machine-learning/). Leave a comment to tell us how you use multi-Region training.", "url": "https://wpnews.pro/news/multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo/", "published_at": "2026-09-25 15:49:44+00:00", "updated_at": "2026-09-25 15:59:15.719888+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "large-language-models", "artificial-intelligence"], "entities": ["Amazon Web Services", "Amazon SageMaker HyperPod", "Qumulo", "Cloud Native Qumulo", "Cloud Data Fabric", "NeuralCache", "Amazon Elastic Kubernetes Service", "Amazon EKS"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo", "markdown": "https://wpnews.pro/news/multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo.md", "text": "https://wpnews.pro/news/multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo.txt", "jsonld": "https://wpnews.pro/news/multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo.jsonld"}}