ParallelClusterMaker – CLI toolkit to manage AWS ParallelCluster stacks

ParallelClusterMaker, an open-source CLI toolkit co-written with Anthropic's Claude Code, automates the creation and destruction of AWS ParallelCluster v3 stacks, enabling researchers and engineers to deploy HPC clusters on AWS without deep infrastructure expertise. The toolkit supports multi-node distributed training on GPU instances (g4dn, g4ad, g5, g5g, g6, p3, p3dn, p4d, p4de, p5) with EFA and EFA-GDR, traditional HPC workloads, and cost-sensitive batch jobs using spot capacity by default. It includes a benchmark suite (STREAM, OSU MPI, IOR, HPCG) for cluster validation and accepts AI-assisted contributions under a public policy requiring disclosure and human responsibility.

This Open Source CLI toolkit automates creation and destruction of AWS ParallelCluster v3 https://github.com/aws/aws-parallelcluster stacks. It lets researchers and engineers stand up a working HPC cluster on AWS without deep infrastructure expertise. This codebase was co-written with Claude Code https://claude.ai/code Anthropic . AI-assisted contributions are accepted under a public policy AI POLICY.md /rmarable/ParallelClusterMaker/blob/main/AI POLICY.md : the tool or model must be disclosed, and a human remains responsible for every line submitted. Training ML models at scale — multi-node distributed training on GPU queues g4dn , g4ad , g5 , g5g , g6 , p3 , p3dn , p4d , p4de , p5 , with EFA and EFA-GDR for inter-node collective communication and FSx for Lustre for high-throughput data loading. Traditional HPC and scientific computing — CFD, molecular dynamics, genomics, and other tightly-coupled MPI workloads, using EFA and dynamic placement groups for a low-latency interconnect. Cluster and instance-type validation — the built-in STREAM, OSU MPI, IOR, and HPCG benchmark suite measures memory bandwidth, MPI latency/bandwidth, filesystem I/O, and floating-point performance before committing a production workload to a given cluster shape. Cost-sensitive batch and embarrassingly parallel work — parameter sweeps, Monte Carlo simulation, rendering — using spot capacity, the default, where jobs can tolerate interruption. On-demand research computing — teams without dedicated infrastructure or a standing DevOps function can stand up a cluster for a research sprint and tear it down when finished, rather than carrying always-on cost. See INSTALL.md /rmarable/ParallelClusterMaker/blob/main/INSTALL.md for prerequisites, AWS account setup VPC tagging, IAM permissions , and installation steps. Scheduling and compute — see Networking and Compute networking-and-compute - Slurm job scheduling - Separate CPU and GPU queues — the GPU queue exists only when --gpu instance type is set - Multi-instance-type queues via --compute instance type and --gpu instance type , each accepting a comma-separated list - Separate instance types and EBS configurations for the head node, CPU queue, and GPU queue - Spot capacity by default --cluster type=spot , with current market rates printed at build time - EFA on supported instance types in both queues --enable efa , with EFA-GDR enabled automatically on p4d/p4de/p5 - Dynamic EFA instance type lookup at launch time, with a static fallback list - Dynamic EC2 placement groups --placement group , applied to compute queues only - Selective HyperThreading disable --hyperthreading=false - Custom AMI support --custom ami - Eight base operating systems across x86 64 and Graviton --base os : ubuntu2204 , ubuntu2404 , ubuntu2204arm , ubuntu2404arm , rhel9 , rhel9arm , alinux2023 , alinux2023arm - Dev / test / stage / prod operating levels --prod level Shared storage — see Storage storage - Shared EBS at /shared , created unconditionally on every cluster - EFS at /efs --enable efs , with optional encryption --efs encryption - FSx for Lustre at /fsx --enable fsx , with optional S3 hydration/dehydration --enable fsx hydration - External NFS automount from a site filer --enable external nfs — Vast, NetApp, WekaIO, Qumulo, etc. - Every filesystem, its mount point, and the resulting Spack install path named in the build summary and the SNS report Software environment — see Software Environment software-environment - Spack + Lmod for HPC software module management - Optional benchmark suite --enable hpc benchmarks : STREAM, OSU MPI, IOR, and HPCG — STREAM recompiles per microarchitecture, so a GPU-partition job measures the GPU node - Optional Grafana/Prometheus monitoring stack --enable monitoring via aws-parallelcluster-monitoring — Grafana dashboards, Prometheus, Slurm exporter, CloudWatch exporter Operations tooling list pcluster.py — table of every cluster tracked by this repo, with optional live CloudFormation status; see Listing Clusters listing-clusters stop pcluster.py / start pcluster.py — stop or start the compute fleet without touching the head node; see Stopping and Starting the Compute Fleet stopping-and-starting-the-compute-fleet check pcluster.py — pass/fail health check; exits 0 only when every check passes; see Checking Cluster Health checking-cluster-health diagnose pcluster.py — raw diagnostic dump: CloudWatch bootstrap logs, node states, job failures, log tails; see Diagnosing a Cluster diagnosing-a-cluster cost pcluster.py — actual spend per cluster from AWS Cost Explorer; see Cost Reporting cost-reporting grafana tunnel.py — open or close the Grafana SSH tunnel for a monitoring-enabled cluster; see Monitoring monitoring rotate cluster key.py — rotate the cluster SSH keypair without a rebuild; see SSH Key Management ssh-key-management manage pcluster queue.py — add, remove, or list Slurm queues on a live cluster; see Managing Queues on a Running Cluster managing-queues-on-a-running-cluster Security and lifecycle - SSH private key stored in AWS Secrets Manager at cluster creation, recoverable via retrieve ssh key.<cluster .sh ; see SSH Key Management ssh-key-management - Resource tagging by owner, department, project, and operating level; see Tagging tagging - SNS notifications on stack create and destroy - Turbot environment support --turbot account - Hourly cost estimate in the build summary — on-demand and spot price ranges per queue from the AWS Pricing API, degrading to unavailable with a reason rather than crashing All optional parameters have hardcoded defaults and can also be persisted in a YAML defaults file. pcluster defaults.yml is the template — copy the file before use ; the tracked version is shared and may be overwritten by updates, so never pass that one directly. The most commonly referenced defaults: | Parameter | Default | |---|---| base os | ubuntu2404 hardcoded ; pcluster defaults.yml ships ubuntu2404arm to match the Graviton head node. Valid: ubuntu2204 , ubuntu2404 , ubuntu2204arm , ubuntu2404arm , rhel9 , rhel9arm , alinux2023 , alinux2023arm | scheduler | slurm | headnode instance type | c8g.xlarge required — no default fallback | compute instance type | c8g.2xlarge, c7g.2xlarge, c6g.2xlarge comma list; empty = no CPU queue | gpu instance type | empty — set to create a GPU queue e.g. p3.2xlarge,g5.2xlarge | gpu root volume size | 250 GB gp3 | cluster type | spot | initial cpu queue size | 2 | max cpu queue size | 8 | initial gpu queue size | 2 | max gpu queue size | 8 | maintain cpu initial size | false | maintain gpu initial size | false | scaledown idletime | 5 min global — applies to both queues | headnode root volume size | 100 GB gp3 | compute root volume size | 250 GB gp3 | ebs shared volume size | 250 GB gp3 | fsx size | 1200 GB | placement group | NONE | hyperthreading | true | enable monitoring | false | enable hpc benchmarks | false | enable efa | false | enable gpu | derived from gpu instance type — not user-settable | head node bootstrap timeout | 2100 s, raised automatically when enable efs or enable fsx is true | Use --use defaults=FILE to load values from your own defaults file; CLI arguments always take precedence. Copy the template and customize it for your cluster cp pcluster defaults.yml my-cluster defaults.yml Pass it at runtime ./make pcluster.py -N my-cluster -O rmarable -E rmarable@example.com -A us-east-1a \ --use defaults=my-cluster defaults.yml Name the file <cluster name defaults.yml to keep cluster namespaces scoped. make pcluster.py detects a matching file that was not loaded and prints a WARNING suggesting the flag. Loading pcluster defaults.yml directly is allowed but also warns. ./make pcluster.py -N CLUSTER NAME -O OWNER -E EMAIL -A AZ options Required arguments: | Flag | Description | |---|---| -N | Cluster name must start with a lowercase letter; lowercase letters, digits, hyphens only; no consecutive or trailing hyphens; max 27 characters | -O | Owner username | -E | Owner email | -A | Availability zone e.g. us-east-1a — pass an AZ, not a region | After a cluster finishes building, the summary includes an estimated hourly cost for the head node and each queue at maximum fleet size: Estimated hourly cost max fleet, on-demand unless noted : Head node c8g.2xlarge × 1 : $0.319/hr CPU queue c8g.2xlarge × 8 : $2.552/hr ~$1.093/hr spot GPU queue p3.2xlarge × 4 : $12.240/hr Note: spot prices are current ask; actual cost may differ. For multi-instance-type queues, a price range is shown cheapest to most expensive type, all nodes at max count . If the AWS Pricing API is unreachable or the operator policy lacks pricing:GetProducts , the affected lines report unavailable — the head node line carries the reason — instead of crashing the build. Spot prices come from ec2:DescribeSpotPriceHistory using the most recent ask and appear only when --cluster type=spot . The summary also names every shared filesystem, where it is mounted, and where Spack and the shared package tree install. Only the filesystems the cluster actually has appear — this example has all of them: Shared storage: /shared EBS gp3, 250 GB /efs EFS bursting throughput /fsx FSx for Lustre 1200 GB S3 import: s3://myImportBucket/data/in S3 export: s3://myExportBucket/data/out Hydrate: /usr/local/bin/import-s3-to-lustre.sh Export: /usr/local/bin/export-lustre-to-s3.sh Progress: /usr/local/bin/check-lustre-export-progress.sh /nfs external NFS storage.domain.com Spack and shared packages install under /fsx/pkg The pkg dir on the last line follows the storage precedence fsx efs external NFS shared EBS , so it moves when a faster filesystem is present — see Spack + Lmod spack--lmod . The same block appears in the emailed SNS build report, which is the copy that outlives the terminal scrollback. Basic cluster in us-east-1a using all defaults: ./make pcluster.py -N pcluster-test-01 -O rmarable -E rodney.marable@gmail.com -A us-east-1a EFS with encryption: ./make pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N morpheus \ --enable efs=true --efs encryption=true Fixed-size compute pool tagged to the compbio department: ./make pcluster.py -A eu-central-1a -O rmarable -E rodney.marable@gmail.com -N koolkeith \ --initial cpu queue size=4 --max cpu queue size=125 --maintain cpu initial size=true \ --scaledown idletime=30 --cluster owner department=compbio --project id=polaroid \ --compute instance type=c5.2xlarge EFA-enabled single-node cluster with performance tests: ./make pcluster.py -A us-east-1a -N rimshot -O rmarable -E rmarable@amazon.com \ --compute instance type=c5n.18xlarge --initial cpu queue size=1 \ --maintain cpu initial size=true --enable efa=true --enable hpc benchmarks=true FSx for Lustre with S3 hydration 7.2 TB, 5 GB chunk size . Import and export must name the same bucket; the default import and export prefixes keep the two sides apart: ./make pcluster.py -A us-west-2b -O rmarable -E rodney.marable@gmail.com -N louievega \ --enable fsx=true --fsx size=7200 --enable fsx hydration=true \ --fsx s3 import bucket=LouieVegaData --fsx s3 export bucket=LouieVegaData \ --fsx chunk size=5000 Large GPU cluster with 3.6 PB Lustre, tagged for production: ./make pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N gilgamesh \ --base os=ubuntu2204 --headnode instance type=r5.xlarge \ --gpu instance type=p3.16xlarge --enable fsx=true --fsx size=3600000 \ --enable fsx hydration=true --fsx s3 import bucket=GilgameshData \ --fsx s3 import path=src/ --fsx s3 export bucket=GilgameshData \ --fsx s3 export path=output/ --prod level=prod --max gpu queue size=256 Mixed CPU + GPU cluster separate queues : ./make pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N bifrost \ --base os=ubuntu2404 --headnode instance type=c5.xlarge \ --compute instance type=c5.2xlarge,c5.4xlarge \ --gpu instance type=g5.2xlarge,g5.4xlarge RHEL 9 cluster login user is ec2-user , not ubuntu : ./make pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N enkidu \ --base os=rhel9 --headnode instance type=c5.xlarge \ --compute instance type=c5.2xlarge Amazon Linux 2023 cluster on Graviton login user is ec2-user : ./make pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N ninlil \ --base os=alinux2023arm --headnode instance type=c8g.xlarge \ --compute instance type=c8g.2xlarge Building from a custom AMI must match base os : ./make pcluster.py -N starscream -O rmarable -E rodney.marable@gmail.com -A us-west-2a \ --enable fsx=true --custom ami=ami-123456789abc --base os=ubuntu2204 A new stack typically takes approximately 30–35 minutes to build. Two measured us-east-1 builds of the same cluster shape c5.xlarge head node, two queues, ubuntu2404 : 34m 24s with a 1200 GB FSx for Lustre filesystem, and 21m 14s with EFS instead. Actual time depends on region, instance type availability, and which shared filesystems are enabled — FSx dominates when it is on, because its provisioning sits on the head node's critical path see Troubleshooting . ./access cluster.py -N CLUSTER NAME Example: ./access cluster.py -N pcluster-test-01 Connecting to head node of pcluster-test-01... ./kill pcluster.py -N CLUSTER NAME -O OWNER -A AZ Teardown takes 5–10 minutes. The cluster's EFS and FSx filesystems are deleted by CloudFormation, which ParallelCluster configures with a Delete deletion policy at creation time — teardown cannot preserve them. The cluster's own S3 bucket parallelclustermaker-<serial is the one resource teardown controls, and it holds the rendered cluster config and the bootstrap scripts. Benchmark results do not live there — they are synced to the long-lived parallelclustermaker-results-<account-id -<region bucket, which teardown never touches — so keeping the per-cluster bucket is only useful for inspecting what a build actually deployed: ./kill pcluster.py -N pcluster-test-01 -O rmarable -A us-east-1a \ --delete s3 bucketname=false Teardown is always manual and at your discretion — nothing in this toolkit schedules a cluster's destruction. Idle compute cost is bounded instead by scaledown idletime : ParallelCluster terminates compute nodes that sit idle longer than that, scaling the fleet to zero on its own. A head node left running still bills, so run kill pcluster.py when you are finished with a cluster. Teardown deletes ten kinds of resource after the CloudFormation stack is gone: the S3 bucket, the FSx hydration policy, the Grafana SSM parameter, the Secrets Manager secret, four managed IAM policies, the monitoring policy, the IAM role and its instance profile, the external NFS security group, and the SNS topic. Each step tolerates its own failure so that one AWS error cannot abandon the other nine — but every ignored failure is collected and reported, and teardown then exits non-zero: ================================================================= Initiated shutdown: 2026-08-03 @ 01:33:12 Completed shutdown: 2026-08-03 @ 01:41:48 Cluster osiris has been deleted, but 2 cleanup step s FAILED. The following resources are still in the account and must be removed by hand -- re-running kill pcluster.py will not retry them once osiris.serial has been deleted: - IAM managed policies pclustermaker-policy-osiris-00000000000000-{HeadNode-Compute,HeadNode-Storage,HeadNode-IAM,ComputeNode-Base} - IAM role and instance profile pclustermaker-role-osiris-00000000000000 Serial number: osiris-00000000000000 ================================================================= The same list is included in the SNS destruction report. Act on it before the run's serial file is gone: kill pcluster.py reads the cluster's serial number to build these resource names, so once active clusters/<cluster / is removed there is nothing left to retry with and the leftovers have to be found by hand. The usual cause is a missing operator IAM permission — see Operator IAM permissions /rmarable/ParallelClusterMaker/blob/main/INSTALL.md operator-iam-permissions . stop pcluster.py and start pcluster.py stop or start the compute fleet while leaving the head node running. Use them to pause a cluster between job batches without paying for idle compute nodes. ./stop pcluster.py -N CLUSTER NAME --wait ./start pcluster.py -N CLUSTER NAME --wait | Flag | Short | Description | |---|---|---| --cluster name NAME | -N | Cluster name required | --region REGION | -R | AWS region default: from cluster record | --wait | -W | Poll until the fleet reaches the target state before exiting | Without --wait the request is submitted and the script exits immediately. With --wait , timestamped status lines are printed every 30 seconds until the fleet is STOPPED or RUNNING up to 45 minutes . Note: stopping the fleet terminates all compute nodes immediately — in-flight Slurm jobs will be killed. Drain the queue first if needed. list pcluster.py enumerates all clusters tracked by this repo anything under active clusters/ and prints a summary table from the local vars file — no AWS credentials required by default. ./list pcluster.py options | Flag | Short | Description | |---|---|---| --live | -L | Call pcluster describe-cluster for real-time status one API call per cluster | --region REGION | -R | Filter output to a single region | --owner OWNER | -O | Filter output to a single owner | --wide | -W | Disable column truncation | --json | -J | Emit a JSON array instead of a table | Example output: Cluster Owner Region Head Node CPU Types GPU Types Min/Max CPU Min/Max GPU Type Age Status ------- -------- --------- ----------- ------------------ --------- ----------- ----------- -------- --- ------ osiris rmarable us-east-1 c8g.2xlarge c8g.2xlarge, c7g.… - 0/8 -/- ondemand 4d LOCAL With --live , the Status column shows clusterStatus / cloudFormationStackStatus e.g. CREATE COMPLETE / CREATE COMPLETE . The two values diverge when a cluster update partially fails. check pcluster.py runs a sequence of health checks against a named cluster and exits 0 only if every check passes. ./check pcluster.py -N CLUSTER NAME --timeout SECONDS | Flag | Short | Description | |---|---|---| --cluster name NAME | -N | Cluster name required | --timeout SECONDS | -T | SSH timeout in seconds default: 15, clamped to 1–300 . The S3 check uses the boto3 default timeout and is unaffected. | Checks performed in order: Vars file — cluster record exists in src/vars files/<name .yml CloudFormation status — pcluster describe-cluster returns clusterStatus=CREATE COMPLETE Head node IP — public or private IP present in the describe-cluster response SSH reachability — ssh … echo OK succeeds Slurm — sinfo -s exits 0 Postinstall complete — custom action marker file /opt/parallelcluster/shared/custom action done is present on the head node Grafana health — curl -sk https://localhost:443/grafana/api/health returns "database":"ok" checked only when enable monitoring=true S3 bucket — s3.head bucket succeeds always run, independent of SSH SSH-dependent checks 4–7 are SKIP ped rather than FAIL ed when SSH is unreachable, so a single SSH failure does not obscure the S3 result. Example output all passing : Checking cluster: my-cluster PASS vars file PASS CloudFormation status: CREATE COMPLETE PASS head node IP: 54.1.2.3 PASS SSH reachability PASS Slurm sinfo -s PASS postinstall complete PASS S3 bucket: my-cluster-parallelcluster-bucket All checks passed — my-cluster is healthy. diagnose pcluster.py goes deeper than check pcluster.py — it collects raw diagnostic data rather than pass/fail checks, and always exits 0 so output is never suppressed by an early failure. ./diagnose pcluster.py -N CLUSTER NAME options | Flag | Default | Description | |---|---|---| --cluster name NAME / -N | required | Cluster name | --region REGION / -R | from vars file | Override AWS region | --timeout SECONDS / -T | 20 | SSH timeout | --cw lines N | 50 | CloudWatch log lines per stream max 500 | --log lines N | 30 | Local log file tail lines max 200 | --hours N | 24 | sacct lookback window in hours | --no cw | off | Skip CloudWatch section omit the flag to include CW output | Sections produced: CloudWatch: head node bootstrap logs — last N lines from cfn-init , cloud-init-output , and cinc client streams. PCluster appends the stack's creation timestamp to the log group name /aws/parallelcluster/<cluster name -<YYYYmmddHHMM , so the group is discovered by prefix rather than constructed; the selected group name is printed above the streams. Rebuilds of the same cluster name leave older groups behind — PCluster retains them by design — and the newest is used. Requires logs:DescribeLogGroups , logs:DescribeLogStreams , logs:FilterLogEvents , and logs:GetLogEvents on the operator identity all included in the operator policy . Pass --no cw to skip this section if permissions are unavailable. Slurm node states — sinfo -N -l output; nodes not in idle / mix / alloc are annotated with <-- not idle . Recent Slurm job failures — sacct filtered to FAILED , CANCELLED , TIMEOUT , NODE FAIL states. Prints a note if no results Slurm accounting is not enabled by default in PCluster v3 . Local log tails — last N lines of /var/log/parallelcluster/slurm resume.log , slurm suspend.log , /var/log/cinc/client.log , /var/log/cloud-init-output.log . Postinstall marker — confirms /opt/parallelcluster/shared/custom action done is present; prints the cluster serial number for cross-referencing S3 benchmark results. Example output: Diagnosing cluster: my-cluster us-east-1 serial: 20260804-abc123 === CloudWatch: head node bootstrap logs === log group: /aws/parallelcluster/my-cluster-202608041130 --- cfn-init --- 2026-08-04 17:24:45 ConfigSet: default 2026-08-04 17:24:49 Install packages: ok ... === Slurm node states sinfo -N -l === NODELIST NODES PARTITION STATE CPUS MEMORY REASON compute-1 1 cpu idle 4 8000 none compute-2 1 cpu drain 4 8000 maintenance <-- not idle === Recent Slurm job failures last 24h === No failed jobs in the last 24h If this is unexpected, Slurm accounting may not be enabled. === Local log tails last 30 lines each === --- /var/log/parallelcluster/slurm resume.log --- ... === Postinstall marker === PASS /opt/parallelcluster/shared/custom action done serial: 20260803-abc123 cost pcluster.py queries AWS Cost Explorer by the ClusterID resource tag to show actual spend per cluster. Results reflect billing data with a 24-hour lag. ./cost pcluster.py options | Flag | Short | Description | |---|---|---| --cluster name NAME | -N | Single cluster default: all in active clusters/ | --owner OWNER | -O | Filter to clusters owned by this user | --days N | -D | Lookback window in days default: 30, max: 365 | --json | -J | Emit JSON array instead of a table | Prerequisites: the operator's IAM user/role needs ce:GetCostAndUsage and ce:ListCostAllocationTags . The ClusterID tag key must also be activated as a cost allocation tag in the AWS Billing console Console → Billing → Cost allocation tags → User-defined tags . If the tag is not activated, all results show $0.00 — the script detects this and prints a warning before running queries. Example output: AWS Cost Explorer — last 30 days 24-hour data lag applies Period: 2026-06-24 – 2026-07-24 Cluster Owner Region Cost $ ------- ----- ------ -------- osiris rmarable us-east-1 $47.82 bifrost rmarable us-west-2 $12.10 With no storage flags at all, a cluster gets one shared filesystem — EBS at /shared — plus a node-local scratch directory on every instance. Everything else is opt-in and additive; enabling EFS or FSx does not replace shared EBS. | Mount point | Type | Scope | Default state | |---|---|---|---| /shared | EBS volume | Shared across head node and all compute nodes | Always created | /local scratch | Root EBS volume, or NVMe instance store | Local to one instance; not shared | Always created on every instance | /efs | EFS | Shared | Opt-in --enable efs=true | /fsx | FSx for Lustre | Shared | Opt-in --enable fsx=true | /nfs/<export | External NFS | Shared, filer-provided | Opt-in --enable external nfs=true | EFS and FSx filesystems are destroyed with the cluster and cannot be preserved by teardown — see Deleting a Cluster deleting-a-cluster . Move anything durable to S3 or to an external NFS filer before tearing down. Whichever of these a cluster ends up with, the build summary names each one and its mount point — see Building a Cluster building-a-cluster for the block. Not optional. The SharedStorage: block in templates/config.pcluster.j2 emits its Ebs entry unconditionally, so every cluster gets one shared EBS volume mounted on the head node and every compute node. There is no enable flag. | Parameter | Default | |---|---| --ebs shared dir | /shared | --ebs shared volume size | 250 GB; max 16,384 | --ebs shared volume type | gp3 gp2 , gp3 , io1 , io2 , st1 | --ebs shared volume iops | 3000 emitted for gp3 , io1 , io2 only | --ebs shared volume throughput | 125 MB/s; emitted for gp3 only | --ebs encryption | false | --ebs encryption also governs the head node, CPU queue, and GPU queue root volumes. Local to a single instance, not shared. templates/postinstall.j2 creates /local scratch as a sticky-bit directory on the root EBS volume and symlinks /scratch to it. When enable gpu is true and NVMe instance store devices are present, /local scratch is backed by those instead — one device is formatted XFS, several are assembled into a RAID0 array. See GPU gpu for the device detection logic. Postinstall is registered as an OnNodeConfigured custom action on the head node and on every compute queue templates/config.pcluster.j2 , so /local scratch is created on every instance. This matters for the instance-store path: NVMe instance store exists only on compute instances, so a head-node-only registration leaves the RAID0 block unreachable in practice. See Node Bootstrap Scripts node-bootstrap-scripts for how the script gets to the node. Data in /local scratch does not survive instance termination, and compute nodes terminate on scale-down. Enable with --enable efs=true . Mounted at /efs on all instances. Costs almost nothing in build time: measured on a generalPurpose / bursting filesystem with one mount target, the filesystem completed in 4 seconds and the mount target in 1m 33s, both finishing well before the Route53 zone and the compute-fleet nested stack that actually gate the head node's launch. A multi-AZ cluster creates one mount target per subnet and has not been timed. Configure with --efs encryption , --efs performance mode generalPurpose or maxIO , and --efs throughput mode bursting , provisioned , or elastic . Enable with --enable fsx=true . Mounted at /fsx . --fsx size must be a positive multiple of 1200 GB; the default and minimum is 1200. --fsx chunk size the S3 imported-file chunk size, default 1024 MB must fall between 1,024 MB 1 GB and 512,000 MB 500 GB . Requires --enable fsx hydration=true . Setting any --fsx s3 value without that flag is an error, as is setting the flag without --enable fsx=true . One bucket, two prefixes. FSx for Lustre requires the export bucket to be the same bucket as the import bucket — from AWS's own API model for CreateFileSystem : "The Amazon S3 export bucket must be the same as the import bucket specified by ImportPath." Only the prefixes may differ. --fsx s3 import bucket and --fsx s3 export bucket exist as separate parameters because ImportPath and ExportPath are separate FSx concepts — ImportPath is where the filesystem is seeded from at creation, ExportPath is where lfs hsm archive writes changed files back — but both must name the same bucket. A mismatch is rejected before the build starts rather than twenty minutes in, at FSx creation: ERROR: Lustre hydration: fsx s3 export bucket out-bucket must name the same bucket as fsx s3 import bucket in-bucket ... The supported shape: --enable fsx hydration=true --fsx s3 import bucket=ResearchData \ --fsx s3 import path=input/ --fsx s3 export bucket=ResearchData \ --fsx s3 export path=output/ The three accepted cases: | Import | Export | Result | |---|---|---| --fsx s3 import bucket=data --fsx s3 import path=in/ | --fsx s3 export bucket=data --fsx s3 export path=out/ | The supported shape. No warning. | --fsx s3 import bucket=data --fsx s3 import path=in/ | unset | Warns; the export side falls back to the import bucket and path, so hydration and dehydration both use s3://data/in/ | ...=data --fsx s3 import path=shared/ | ...=data --fsx s3 export path=shared/ | Warns: dehydrated files overwrite the hydration source | An unset --fsx s3 import bucket with --enable fsx hydration=true is an error — there is nothing to hydrate from. Use --enable fsx=true on its own for an empty Lustre filesystem. Both sides are validated before the build starts: the bucket must exist head bucket , with a distinct error for a 403 so an access-denied bucket policy is not reported as a missing bucket , and the prefix must contain at least one object. An empty or misspelled path fails immediately rather than at FSx creation time. Note that the export fallback overwrites --fsx s3 export path with the import path. To keep the two prefixes separate, name the bucket in both parameters rather than relying on the fallback. Helper scripts on the head node. When hydration is enabled, postinstall writes three one-line wrappers around Lustre's HSM commands into /usr/local/bin mode 755, root-owned . The build summary prints their paths. They are not run automatically — hydration itself happens at FSx creation from ImportPath ; these are for pulling and pushing on demand afterward: | Script | What it does | |---|---| import-s3-to-lustre.sh | lfs hsm restore over every file under /fsx — fetches file contents for entries FSx has listed but not yet copied down | export-lustre-to-s3.sh | lfs hsm archive over every file under /fsx — writes changed files back to the export prefix | check-lustre-export-progress.sh | Counts outstanding ARCHIVE actions, so an export in flight can be polled | All three walk the whole filesystem with find , so on a large tree they take a while and are best run under nohup the import and export scripts already background themselves . Verified on a live rhel9arm head node with a 1200 GB Lustre filesystem and lfs 2.15.6 . Enable with --enable external nfs=true --external nfs server=storage.domain.com . Exports are mounted under /nfs/ and the export list lives in templates/external nfs mount list.j2 — uncomment the lines matching the paths your filer serves. A security group permitting NFS traffic is attached to the head node and compute queues automatically. Important:The toolkit auto-discovers VPCs and subnets by convention when explicit values are not provided. Auto-discovery picks the AWS default VPC and thefirstsubnet returned by EC2 in each AZ. EC2 does not guarantee subnet ordering, so results are non-deterministic in accounts with multiple subnets per AZ.Do not rely on auto-discovery for production clusters.Always specify networking resources explicitly. | Parameter | Description | |---|---| --vpc name | VPC Name tag to use default: vpc default — the account's default VPC | --headnode subnet id | Explicit subnet ID for the head node; overrides auto-discovery | --compute subnet ids | Comma-separated subnet IDs for the CPU compute fleet; overrides auto-discovery | --compute az | Comma-separated AZs for the CPU compute fleet default: same as --az | --use private compute subnet | Only auto-discover private subnets for CPU nodes true / false , default: false | --gpu subnet ids | Comma-separated subnet IDs for the GPU queue; falls back to compute subnet ids if empty | --gpu az | Comma-separated AZs for the GPU queue; falls back to compute az then --az if empty | --use private gpu subnet | Only auto-discover private subnets for GPU nodes true / false , default: false | Subnets and security groups are generated as part of the CloudFormation stack — the toolkit does not manage them independently outside of the stack lifecycle. Single-AZ cluster explicit subnets — recommended : ./make pcluster.py -N prod01 -O rmarable -E rmarable@example.com -A us-east-1a \ --vpc name=my-hpc-vpc \ --headnode subnet id=subnet-0abc123 \ --compute subnet ids=subnet-0abc123 Multi-AZ compute fleet spanning three AZs: ./make pcluster.py -N bigcluster -O rmarable -E rmarable@example.com -A us-east-1a \ --vpc name=my-hpc-vpc \ --headnode subnet id=subnet-0abc123 \ --compute az=us-east-1a,us-east-1b,us-east-1c \ --compute subnet ids=subnet-0abc123,subnet-0def456,subnet-0ghi789 Private compute subnet head node public, compute private : ./make pcluster.py -N private01 -O rmarable -E rmarable@example.com -A us-east-1a \ --vpc name=my-hpc-vpc \ --headnode subnet id=subnet-0abc123 \ --compute subnet ids=subnet-0private1 \ --use private compute subnet=true Set --gpu instance type to create a dedicated GPU queue alongside or instead of the CPU queue. enable gpu is derived — it is true whenever gpu instance type is non-empty, and cannot be set by the user. GPU families g4dn , g4ad , g5 , g5g , g6 , p3 , p3dn , p4d , p4de , p5 are rejected from compute instance type , and non-GPU types are rejected from gpu instance type . Both compute instance type and gpu instance type accept comma-separated lists for multi-instance-type queues: GPU-only cluster ./make pcluster.py ... --gpu instance type=p3.2xlarge,g5.2xlarge CPU + GPU queues ./make pcluster.py ... --compute instance type=c5.2xlarge --gpu instance type=g5.2xlarge The head node's instance type is independent of the queues. A CPU head node fronting both a CPU queue and a GPU queue — c5.xlarge head, c5 compute partition, g5 gpu partition — is the common layout and is fully supported, including for the benchmark suite. enable cpu queue and enable gpu queue are each derived from whether the corresponding instance-type flag is non-empty, and the two SlurmQueues entries are emitted independently; nothing about the head node's own family gates either one. There is no need for a GPU head node to run GPU jobs, and paying for an idle GPU on the head node buys nothing. The one real constraint is architecture, not CPU-vs-GPU. The head node and every queue instance type must share one CPU architecture, because a cluster runs one architecture-specific base OS image. make pcluster.py checks this at creation time against ec2:DescribeInstanceTypes — which is authoritative, and covers families the hardcoded ARM prefix list does not yet name — and refuses the build with the offending types named. So an x86 64 head node with a Graviton g5g GPU queue is rejected up front rather than failing twenty minutes into the build. Mixing microarchitectures within an architecture Intel head node, AMD GPU nodes is fine and is handled automatically by the benchmark suite. What the GPU postinstall block does: templates/postinstall.j2 gates a GPU block on enable gpu == 'true' . Postinstall runs as an OnNodeConfigured custom action on the head node and on every compute queue, so this block runs on the GPU compute nodes — which are the only instances that have NVMe instance store to configure. - NVMe instance store detection — scans /sys/block/nvme and accepts a device only if it passes three independent filters: the model string matches AmazonEC2NVMeInstanceStorage whitespace stripped , holders/ is empty, and blkid finds no filesystem signature. The model check keeps EBS volumes, which also appear as /dev/nvme , from being formatted. The other two keep the toolkit off devices ParallelCluster already claimed, and neither subsumes the other — a device inside an LVM volume has holders but no signature of its own, while a formatted-but-unmounted device has a signature and no holders ParallelCluster usually claims these devices first. The aws-parallelcluster-environment::ephemeral drives cookbook runs before OnNodeConfigured and, on any instance type with instance store, puts every such device into an LVM physical volume, formatted ext4 and mounted on /scratch . On that common case the toolkit's block correctly does nothing and /local scratch is a symlink to PCluster's /scratch — verified on a live g4dn.xlarge . Without the holders/ blkid filters, mkfs.xfs on a claimed device fails with Device or resource busy , which fails the node- Single unclaimed device: formatted XFS, mounted at /local scratch with noatime,nodiratime,nofail - Multiple unclaimed devices p4d.24xlarge has 8×1000 GB, p5.48xlarge has 8×3800 GB, per aws ec2 describe-instance-types : RAID0 via mdadm , mounted at /local scratch - No instance store present e.g. p3.2xlarge : /local scratch remains a sticky-bit directory on the root EBS volume htop installed by the GPU block itself, since the main package block is head-node-only and does not run on compute nodes; nvtop is head-node-only because it lives outside the default repositories multiverse on Ubuntu, EPEL on RHEL 9 and a compute node's package index is whatever the AMI shipped. A compute node refreshes its index first apt-get update / dnf makecache because OnNodeStart — and therefore preinstall's refresh — never runs there. Both installs are non-fatal || echo "WARNING: ..." , the only ones in the file: they are diagnostics nothing in the job path imports, and one transient mirror outage would otherwise count toward the 10-failure protected-mode threshold and cost the entire stack EFA GPUDirect RDMA GDR : When --enable efa=true and any GPU queue instance type is p4d.24xlarge , p4de.24xlarge , or p5.48xlarge , GdrSupport: true is added to the GPU queue EFA config automatically. GPU volume settings: The GPU queue uses its own root volume parameters --gpu root volume size , --gpu root volume type , --gpu root volume iops , --gpu root volume throughput independent of the CPU queue. CUDA / drivers: PCluster's official deep learning AMIs include NVIDIA drivers. Pass --custom ami=<ami-id to use a pre-built DLAMI or a custom AMI with pinned driver versions. Enable with --enable efa=true . Supported on every base os value the toolkit accepts. Requires a supported instance type c5n.18xlarge, hpc6a.48xlarge, hpc7a.96xlarge, hpc7g.16xlarge, etc. . A dynamic placement group is created automatically. EFA Elastic Fabric Adapter is an OS-bypass network interface that gives tightly-coupled, latency-sensitive MPI jobs a much faster interconnect than standard TCP/IP networking — closer to InfiniBand than to a regular NIC. EFA matters most for multi-node jobs with heavy collective communication large all-reduce, halo exchanges, etc. ; a single-node job, or one bound by disk or ordinary network I/O, sees little benefit. On p4d, p4de, and p5 instances, EFA-GDR GPUDirect RDMA is enabled automatically, letting the network adapter read and write GPU memory directly instead of staging through host memory — which matters for multi-node GPU jobs. Not yet verified on hardware. EFA-enabled builds have not been run against real instances; the config generation and instance-type gating are unit-tested, but the actual interconnect behavior is unconfirmed. Enable with --placement group=DYNAMIC . PCluster creates one managed cluster placement group per queue, applied to the CPU and GPU compute queues only. The head node is never placed in a placement group. --enable efa=true sets this to DYNAMIC automatically when the setting is still NONE . Disable with --hyperthreading=false . HyperThreading Intel / SMT AMD exposes each physical CPU core as two logical vCPUs sharing the same execution resources. Many HPC workloads — anything compute- or memory-bandwidth-bound rather than I/O-bound — run faster with HyperThreading disabled, since two threads contending for one core's resources often costs more than the extra thread gains. Disabling HyperThreading also changes the per-node rank count Slurm submits with: cpu ranks per node divides vCPUs by DefaultThreadsPerCore instead of always halving see Job Submission job-submission . manage pcluster queue.py edits a live cluster's config file to add, remove, or list Slurm queue stanzas without rebuilding the cluster. Changes take effect after a pcluster update-cluster ; the compute fleet must be stopped before updating. Pass -W to have the script handle the full stop/update/restart cycle automatically. ./manage pcluster queue.py -N <cluster name -A <action -T <queue type options Required arguments: | Flag | Description | |---|---| -N | Cluster name | -A | Action: add , remove , or list | -T | Queue type: compute CPU or gpu . Required for add ; ignored by remove and list . | ./manage pcluster queue.py -N <cluster name -A list Add a spot CPU queue ./manage pcluster queue.py -N osiris -A add -T compute \ -E c5.2xlarge,c5.4xlarge -C spot \ -Q compute-spot-overflow Add an on-demand GPU queue ./manage pcluster queue.py -N osiris -A add -T gpu \ -E p3.2xlarge -C ondemand \ -Q gpu-ondemand Add a GPU queue with custom scaling ./manage pcluster queue.py -N osiris -A add -T gpu \ -E g5.2xlarge,g5.4xlarge \ --initial size 1 --max size 4 --maintain initial size true ./manage pcluster queue.py -N osiris -A remove -Q compute-spot-overflow Automated -W/--wait : the script stops the fleet, applies the config, and restarts the fleet, printing status every 30 seconds. It warns that the operation can take up to 30 minutes; each individual poll loop times out at 45 minutes. Run inside screen or tmux to avoid losing the session mid-update. ./manage pcluster queue.py -N osiris -A add -T compute -E c5.xlarge --wait Manual default : after add or remove , the script prints the exact commands needed: - Stop the compute fleet - Wait for STOPPED status - Run pcluster update-cluster with the updated config - Wait for UPDATE COMPLETE - Restart the fleet - Compute queues reject GPU instance types and vice versa - Mixed x86/Graviton instance types in a single queue are rejected - A new queue whose architecture differs from the running cluster's is rejected — a cluster runs one architecture-specific base OS image - The last remaining queue on a cluster cannot be removed p4d / p4de / p5 GPU instances support EFA-GDR; the script prints a reminder to enable it manually if needed --base os accepts eight values, in two package-manager families: base os | Architecture | Package manager | Login user | |---|---|---|---| ubuntu2204 | x86 64 | apt | ubuntu | ubuntu2404 | x86 64 | apt | ubuntu | ubuntu2204arm | Graviton aarch64 | apt | ubuntu | ubuntu2404arm | Graviton aarch64 | apt | ubuntu | rhel9 | x86 64 | dnf | ec2-user | rhel9arm | Graviton aarch64 | dnf | ec2-user | alinux2023 | x86 64 | dnf | ec2-user | alinux2023arm | Graviton aarch64 | dnf | ec2-user | The arm suffix is the toolkit's, not ParallelCluster's — it drives instance-architecture validation and is stripped before the value reaches PCluster's Os: field, so ubuntu2404arm , rhel9arm , and alinux2023arm become ubuntu2404 , rhel9 , and alinux2023 there. Mixing an ARM base os with an x86 64 instance type or vice versa is rejected before anything is created; do not mix architectures across node types either. preinstall.j2 and postinstall.j2 branch on the OS family and install the equivalent package sets; the dnf side branches again where Amazon Linux 2023 and RHEL 9 differ. A few differences are worth knowing about if you write a --post install script hook: - On RHEL 9, EPEL and CodeReady Builder are enabled during postinstall because lua-devel , lua-posix , lua-filesystem , and tcllib are in neither baseos nor appstream. EPEL is installed from its release RPM URL, since epel-release is not packaged in RHEL itself. - On Amazon Linux 2023, neither EPEL nor CRB is used, because epel-release is not packaged for al2023 at all and everything the toolkit needs is already in the core repo. Four packages the RHEL arm installs are absent from al2023 on both architectures, so that arm differs accordingly: luarocks the three Lua rocks come from the core repo as RPMs instead, and the luarocks build step is a no-op , tcllib unused by the toolkit — Lmod uses tcl itself , and nvtop so GPU clusters get htop only, on both node types . Do not add any of them back for symmetry with RHEL; each one would fail the node. pip3 is called with --break-system-packages on Ubuntu and without it on RHEL 9 and Amazon Linux 2023, both of which ship a pip that predates PEP 668 and rejects the flag.- Every pip3 install on a node carries --ignore-installed , on both families. pip cannot uninstall a distribution whose dist-info has no RECORD file, and distro-packaged Python modules routinely ship exactly that — Ubuntu's python3-pip and RHEL 9's python3-requests are both confirmed cases. Any install that resolves to replacing one dies at Attempting uninstall: , which under set -euo pipefail fails the node's bootstrap. --ignore-installed skips the uninstall and installs over the top; nothing in the toolkit needs the distro package removed. This is a different problem from --break-system-packages , which only permits writing into the system tree. nvtop GPU clusters is installed on the head node only, on Ubuntu and RHEL 9 — the package sits outside the default repositories in each, and the operator logs into the head node rather than a compute node. Amazon Linux 2023 skips nvtop entirely, since the package isn't available there.- On RHEL 9, bc is installed explicitly because Lmod's ./configure hard-quits without bc You must have bc in your path. Quitting rather than degrading. bc is on the Ubuntu and Amazon Linux 2023 package lines too, deliberately: those AMIs happen to ship bc incidentally, and depending on what a base image carries by accident is how the RHEL gap stayed hidden. Upstream's own monitoring installer claims bc is absent from the default al2023 repos; the repo metadata says otherwise on both architectures. - The dnf kernel exclusions cover both Lustre spellings — kmod-lustre and lustre-client — because the two distros name the client differently. Amazon Linux 2023 has no kmod-lustre package at all, so the RHEL glob alone would silently protect nothing there. Both Amazon Linux 2023 arms are validated on live cluster builds. alinux2023 reached CREATE COMPLETE on a c5.xlarge head node with EFS, a c5.2xlarge CPU queue, two g4dn.xlarge GPU nodes, benchmarks, and monitoring; alinux2023arm reached CREATE COMPLETE on a c8g.xlarge head node with EFS, a 1200 GB FSx for Lustre filesystem, a c8g.2xlarge CPU queue, benchmarks, and monitoring. Every package claim above was confirmed against the head node's and every compute node's own bootstrap logs on both architectures, not merely against the stack's exit status: luarocks , epel , tcllib , and nvtop appear nowhere; the three Lua rocks install as core-repo RPMs; bc is already present and Lmod's ./configure finds it; and dnf update upgrades a single package with no kernel, dracut, or initramfs activity. The Docker Compose CLI plugin is staged and checksum-verified per architecture — docker-compose-linux-x86 64-v2.29.7 and docker-compose-linux-aarch64-v2.29.7 respectively — and upstream's github.com download is removed from the extracted tree on both. Both RHEL 9 arms are validated on live cluster builds. rhel9 reached CREATE COMPLETE on a c5.xlarge head node with EFS, a CPU queue, a g4dn.xlarge GPU queue, benchmarks, and monitoring; rhel9arm reached CREATE COMPLETE on a c8g.xlarge head node with EFS, a 1200 GB FSx for Lustre filesystem, benchmarks, and monitoring. The whole RPM bootstrap path is confirmed on both architectures: EPEL by release-RPM URL, the CodeReady Builder repository id, all three luarocks rocks compiling against lua-devel with no separate header package, and dnf update upgrading dracut itself while installing zero kernel packages and regenerating no initramfs. All eight pip pins resolved from manylinux 2 17 aarch64 wheels on Graviton. Two stages run on every node, in this order: The toolkit's own scripts — templates/preinstall.j2 and templates/postinstall.j2 , rendered per cluster with that cluster's variables OS, storage layout, pkg dir , GPU flags and uploaded to the cluster's S3 bucket as preinstall.<cluster .sh and postinstall.<cluster .sh . These are the toolkit's own work: base packages, Spack, Lmod, /local scratch , the benchmark suite, and the GPU block. Your hook — the script named by --pre install script / --post install script , copied verbatim and uploaded under its own basename. Defaults are scripts/pre-deployment.sh and scripts/post-deployment.sh , both no-op placeholders. Put site-specific customization here; do not edit the toolkit templates to add it. The stages are wired as a PCluster Sequence , so stage 2 runs only if stage 1 succeeded. OnNodeStart preinstall runs on the head node only — repeating the Python/pip/AWS CLI install on every scale-up event would add boot latency to every compute node for no benefit. OnNodeConfigured postinstall runs on the head node and on every compute queue, since that is where node-local work like /local scratch belongs. When --enable monitoring=true the monitoring installer is appended as a third stage. Paths passed to --pre install script / --post install script are relative to the repository root. preinstall.j2 upgrades the AMI's packages — apt-get dist-upgrade on Ubuntu, dnf update on RHEL 9 — but holds back the running kernel along with the out-of-tree modules built against it. Ubuntu does this with apt-mark hold on the installed linux- packages; RHEL 9 with --exclude='kernel ' --exclude='kmod-lustre ' --exclude='efa ' . Two independent reasons: A kernel replacement triggers an initramfs rebuild whose runtime is unbounded , and it runs inside the window CloudFormation gives the head node to signal success. Real builds failed this way on both families: on the PCluster AMI of the day a full upgrade crossed a kernel boundary and was still rebuilding when the wait condition expired, and on the RHEL 9 AMI a full update crossed 5.14.0-611.55.1.el9 7 → 5.14.0-687.30.1.el9 8 with dracut still running when CloudFormation gave up. PCluster's AMI ships EFA and Lustre kernel modules built against the kernel it boots. Replacing that kernel without rebuilding them risks losing the interconnect or the Lustre client on the next boot. The two mechanisms are equivalent in effect but not in shape: apt-mark pins packages by name, so the Ubuntu path first enumerates the installed linux- packages and filters them to the ones dpkg reports as actually installed. --exclude takes a glob resolved when dnf builds the transaction, so the RHEL 9 path needs no enumeration and holds on every AMI revision regardless of which updates happen to be pending — every RHEL 9 kernel subpackage kernel , -core , -modules , -modules-core , -modules-extra , -headers , -devel , -tools matches kernel . The upgrade itself is deliberately kept, because preinstall.j2 installs the Python development headers and numpy / scipy / pandas / matplotlib compile from source wherever pip finds no wheel — which every arm value of base os can hit. If you add a package that needs a newer kernel, pin it in your own --pre install script hook and reboot deliberately outside the bootstrap window; do not remove the exclusions. Every stack includes Spack https://spack.io/ and Lmod https://github.com/TACC/Lmod for HPC software module management. Spack is cloned into <shared /pkg/spack , where <shared is the first available of /fsx , /efs , /nfs , or the shared EBS mount. Both are built on the head node only. Compute nodes mount the same shared storage and inherit the installation, so having every scaling node repeat the clone and chown -R would be wasted boot time on a tree that is already populated. MODULEPATH comes from Spack, not from Lmod's compiled-in root. Lmod is configured with --with-module-root-path=<shared /pkg/modulefiles , but that setting is only ever read by Lmod's init/profile script, and postinstall installs init/sh instead — which defines the module , ml , and clearMT shell functions and sets MODULESHOME , with no reference to MODULEPATH at all. What populates MODULEPATH on a login shell is /etc/profile.d/lmod spack.sh sourcing Spack's setup-env.sh , which appends Spack's own module roots. Consequently <shared /pkg/modulefiles is not created by the install and will not exist on a fresh cluster — nothing reads that path, so this is expected rather than a broken installation. If you want to hand-place modulefiles outside Spack, create that directory yourself and add its path to MODULEPATH or to LMOD SITE MODULEPATH from your own --post install script hook.A default Slurm submission script scripts/sbatch default submission script.sh is rendered from the toolkit's scripts/ directory into the login user's home directory during cluster creation ubuntu or ec2-user , depending on base os . Copy it to shared storage and customize: cp ~/sbatch default submission script.sh /fsx/scratch/my project/ sbatch /fsx/scratch/my project/sbatch default submission script.sh --partition and --ntasks are derived from this cluster's own shape, not hardcoded. A cluster with a CPU queue gets --partition=compute and --ntasks=<cpu ranks per node ; a GPU-only cluster gets --partition=gpu and --ntasks=<gpu vcpus per node . Both matter: A GPU-only cluster has no , and compute partition sbatch rejects an invalid partition outright before the job ever runs. enable cpu queue is derived from --compute instance type , so the script follows whichever queues the cluster actually has. The rank count is vCPUs, divided by — never halved unconditionally. Graviton reports one thread per core, so halving there would request half the cores every ARM node has. Where a queue holds several instance types, the count is the smallest one's, since that is the only value every node in the queue can satisfy. DefaultThreadsPerCore when --hyperthreading=false Note this is a core count, which is not the same as the GPU benchmark's --ntasks-per-node : job hpc-benchmark.sh matches its rank count to the number of NVIDIA devices per node, while this script asks for cores. A p3.2xlarge has 1 GPU and 8 vCPUs, and a general-purpose job wants the 8. Enable with --enable hpc benchmarks=true . Cluster creation deploys the benchmark suite to a personalized working directory on the head node — ~/hpc-benchmark/<cluster name /<cluster owner /slurm/ — holding the driver, a job hpc-benchmark.sh rendered for this cluster's queue layout, and a README-PERFORMANCE.md naming this cluster's own paths. Postinstall also installs the Python plotting dependencies matplotlib , numpy , pandas , scipy , seaborn and drops a second copy of the driver at ~/hpc-benchmark/hpc-benchmark.sh ; that copy is restored from S3 on every head node boot, so a replaced EBS root never loses the driver — the only file staged there. Work out of the personalized directory, since that is the one with the rendered job script. These commands run on the cluster head node SSH in via ./access cluster.py first : cd ~/hpc-benchmark/<cluster name /<cluster owner /slurm module load openmpi ./hpc-benchmark.sh install build STREAM, OSU, IOR, HPCG ~5 min ./hpc-benchmark.sh run --tests stream,osu,ior,hpcg ./hpc-benchmark.sh report Results are preserved on teardown. kill pcluster.py syncs benchmark results from the head node to s3://parallelclustermaker-results-<account-id -<region /hpc-benchmark-results/<cluster name /<cluster serial number / before deleting the cluster. That bucket is not the per-cluster bucket — keyed on your account and region, created on the first build that enables benchmarks, and never deleted by this toolkit — so results from multiple builds of the same cluster name land in separate serial-number subdirectories and accumulate rather than overwriting each other. This is the one bucket you are expected to prune by hand. STREAM is rebuilt per node. STREAM is compiled -march=native , which binds the binary to the microarchitecture rather than the architecture — a c5.xlarge head node is Intel Skylake and a g5.xlarge GPU node is AMD Zen 3, and uname -m calls both x86 64 . install caches the source and run compiles bin/stream-<march locally, so a job on a GPU partition measures that node's real bandwidth with no manual step. OSU, IOR, and HPCG are built by configure / make without -march=native and are portable across microarchitectures, so those stay in bin/ guarded by an architecture stamp. OSU builds itself on the GPU node when the head node cannot. install enables CUDA only when the node running install has both an NVIDIA device and a CUDA toolkit, because OSU's configure aborts outright on a missing -lcuda , -lcudart , or cuda.h rather than degrading — deriving that from a cluster-level flag would fail the whole install and take STREAM, IOR, and HPCG down with OSU. A CPU head node therefore produces a host-to-host OSU, and the first GPU-partition job builds a CUDA-enabled tree under bin/osu-cuda on the GPU node itself and writes osu/latency cuda.txt and osu/bandwidth cuda.txt alongside the host-to-host results. bin/ is shared storage, so later GPU jobs reuse that build. This step can never fail the run: a node that can't build CUDA support still writes the host-to-host results, with the reason printed. Both partitions are benchmarkable from a CPU head node. The shipped job hpc-benchmark.sh is submittable as-is: its SBATCH --partition= and --ntasks-per-node= directives are rendered from this cluster's queue layout. A GPU-only cluster has no compute partition, so a hardcoded partition would be rejected by sbatch before anything ran; on such a cluster the rank count is the NVIDIA GPU count reported for the queue's instance types, so one rank lands per GPU. On a cluster with both queues the script targets compute , and the GPU run is the same script with two directives overridden: sbatch job hpc-benchmark.sh compute partition sbatch --partition=gpu --ntasks-per-node=<gpu count job hpc-benchmark.sh The exact second command, with this cluster's GPU count already substituted, is in the commented GPU section at the bottom of the rendered script. See hpc-benchmark/README-PERFORMANCE.md for full documentation. The copy in the personalized working directory on the head node is the same document with this cluster's name, owner, and paths substituted in; the copy in this repo spells them <cluster name and <cluster owner and is not deployed. Enable with --enable monitoring=true default: false . Deploys the aws-parallelcluster-monitoring https://github.com/aws-samples/aws-parallelcluster-monitoring Grafana/Prometheus stack. The same install script runs on every node; it branches on node type internally, so the head node gets the full stack and compute nodes get only a metrics exporter. On the head node: - Grafana port 443, self-signed TLS - Prometheus, pushgateway, cloudwatch-exporter, nginx, node exporter Docker Compose - prometheus-slurm-exporter systemd, scrapes Slurm metrics every 30 s On each compute node: - node exporter Docker Compose - NVIDIA DCGM exporter on GPU instances, from compose/compute.gpu.yml upstream — requires the nvidia-container-toolkit , which the upstream installer provides Access Grafana: grafana tunnel.py opens or closes the tunnel without locating the per-cluster script manually: Open tunnel background SSH, prints URL and password command ./grafana tunnel.py -N CLUSTER NAME Use a different local port ./grafana tunnel.py -N CLUSTER NAME -P 9443 Close the tunnel ./grafana tunnel.py -N CLUSTER NAME --stop The script verifies that monitoring is enabled for the cluster and exits with a message if it is not. The generated per-cluster script also works directly: Open ./active clusters/<cluster name /grafana tunnel.<cluster name .sh Close ./active clusters/<cluster name /grafana tunnel.<cluster name .sh 8443 stop Then open https://localhost:8443/grafana/ in your browser and accept the self-signed certificate warning. Pass a different local port if 8443 is in use. If the head node has a public IP you can also open https://<head-node-public-ip /grafana/ directly requires port 443 open in the security group . Retrieve the Grafana admin password: aws ssm get-parameter \ --name "/parallelcluster/<cluster name /grafana/admin-password" \ --with-decryption \ --query "Parameter.Value" --output text Note: PCluster head nodes ship a web server running on port 80 — apache2 on Ubuntu, httpd on RHEL 9. The monitoring installer stops and disables whichever is present so the nginx container can bind ports 80 and 443. Monitoring is verified on RHEL 9, on both architectures. Upstream's detect platform resolves PLATFORM ID=platform:el9 on x86 64 and Graviton alike, and the container stack reached Started on both — so the v2.6 installer is arch-agnostic on el9, not x86-only. IAM: Monitoring permissions are granted via a separate managed policy <ec2 iam policy -HeadNode-Monitoring 8 statements, ~1,550 bytes minified . It is created and attached during make pcluster.py and deleted during kill pcluster.py . Supply chain: The aws-parallelcluster-monitoring tarball is downloaded from GitHub at cluster-build time, checksum-verified, and staged in the cluster's S3 bucket. Head nodes pull from S3, not GitHub, so private-subnet nodes and air-gapped environments work without internet access. On Amazon Linux 2023 that applies to the Docker Compose CLI plugin as well. AL2023 does not package docker-compose-plugin , so upstream's installer downloads the binary from github.com on every node at boot, with no integrity check — which fails outright on a private subnet. The toolkit instead downloads the binary once at build time, verifies the checksum against --docker compose checksum x86 64 / --docker compose checksum aarch64 both defaulted in pcluster defaults.yml and matching Docker's own published sums for v2.29.7 , stages the verified binary to S3, and installs from there on every node before the monitoring installer runs. The wrapper also deletes upstream's download from the extracted tree so that download can never overwrite the verified copy, and fails the build by name if that edit stops matching in a future monitoring release. Because the download is removed rather than reused, --docker compose version is the toolkit's own pin and need not match upstream's. The Ubuntu and RHEL 9 arms install the plugin from a signed distro repository and use none of this. Version: Pin a specific release tag with --monitoring version=v2.6 default . Custom AMI: The Docker Compose installation adds several minutes to head node boot time. For production clusters or fast iteration, build a custom AMI with the monitoring stack pre-installed. At cluster creation, the SSH private key is stored in AWS Secrets Manager at: parallelcluster/<cluster name /<cluster serial number /ssh-private-key The secret is deleted automatically when kill pcluster.py runs, along with the EC2 keypair and the local .pem file. If the CloudFormation stack deletion itself reaches DELETE FAILED e.g. a dangling ENI, security group, or EFA interface , all three are deliberately preserved instead — this keeps the head node reachable for manual troubleshooting until the stack is fully torn down. Re-run kill pcluster.py after resolving the CloudFormation dependency to complete cleanup. If your local .pem file is lost , retrieve it from Secrets Manager: active clusters/<cluster name /retrieve ssh key.<cluster name .sh optionally specify a destination: active clusters/<cluster name /retrieve ssh key.<cluster name .sh --out /tmp/mykey.pem access cluster.py calls the retrieve script automatically if the local key is missing. Rotating the SSH keypair without rebuilding the cluster: ./rotate cluster key.py -N <cluster name -A <az preview what will change: ./rotate cluster key.py -N <cluster name -A <az --dry run Rotation: generates a new ED25519 keypair locally, appends the public key to ~/.ssh/authorized keys on the head node, imports it as the new EC2 keypair, updates the Secrets Manager secret, overwrites the local .pem , and deletes the old EC2 keypair. IAM requirements operator's user/role — not the cluster head node role : | Permission | Purpose | |---|---| secretsmanager:CreateSecret | Store key at cluster creation | secretsmanager:PutSecretValue | Update key on rotation | secretsmanager:GetSecretValue | Retrieve key via retrieve script | secretsmanager:DeleteSecret | Remove key on teardown | ec2:ImportKeyPair | Register new public key during rotation | ec2:DeleteKeyPair | Remove old keypair after rotation | All resources are tagged automatically: | Tag | Source | |---|---| ClusterID | --cluster name | ClusterOwner | --cluster owner | ClusterOwnerEmail | --cluster owner email | ClusterOwnerDepartment | --cluster owner department | ClusterStackType | ParallelCluster | ClusterOSType | --base os | ClusterScheduler | --scheduler | ClusterSerialNumber | generated | ProdLevel | --prod level | ProjectID | --project id if set | DEPLOYMENT DATE | generated | Supported departments: analytics , clinical , commercial , compbio , compchem , datasci , design , development , hpc , imaging , manufacturing , medical , modeling , operations , proteomics , qa , research , robotics , scicomp . ParallelClusterMaker does not create or modify VPCs, subnets, gateways, routes, or Transit Gateways. It creates IAM roles, policies, and instance profiles scoped to each individual cluster stack. Templates are in templates/ and can be customized. If you hit permissions errors, the IAM policy template is the right starting point for working with your security team. IAM permissions: Check templates/HeadNode-Compute.json src , HeadNode-Storage.json src , HeadNode-IAM.json src , ComputeNode-Base.json src , and when enable monitoring=true HeadNode-Monitoring.json src . The instance policy is split by role into five managed policies to stay under the IAM managed policy size limit. IAM role and instance-profile resources use both flat-name ARNs parallelcluster-<CLUSTER NAME - and path-based ARNs parallelcluster/<CLUSTER NAME / — PCluster v3 uses the latter for compute fleet roles. Most build failures trace back to missing IAM permissions. Spot capacity: Compute nodes that fail to launch surface as a ComputeFleet - CREATE FAILED CloudFormation error. Retry the build or switch to --cluster type=ondemand . Build fails with HeadNodeWaitCondition timing out CREATE FAILED, 0 of 1 signals : The head node did not finish bootstrapping inside the window CloudFormation allows. Note that this clock starts when CloudFormation begins creating the wait condition — before the head node instance exists — and shared filesystem provisioning sits on the head node's critical path. A 1200 GB FSx for Lustre filesystem measured 17m 22s, over half of PCluster's stock 2100 s budget, before the instance had even launched. The toolkit raises head node bootstrap timeout automatically for this: +1800 s when enable fsx is true, +600 s when enable efs is true, whichever is larger the two provision concurrently, so the head node waits on the slower one, not the sum . A INFO line names the filesystem that drove the increase. Both allowances are measured against live builds rather than estimated. FSx: a second 1200 GB filesystem took 19m 20s before its instance existed, and the build completed in 34m 24s of the 3900 s granted. EFS generalPurpose / bursting , one mount target : the filesystem itself completed in 4 s and its mount target in 1m 33s, with the instance appearing 4m 24s in and the wait condition satisfied in 20m 52s of the 2700 s granted — so the EFS allowance carries roughly 2.3x headroom. Two caveats worth knowing before you rely on them: a multi-AZ EFS cluster creates one mount target per subnet and has not been timed, and on the EFS build the mount target was not in fact what gated the instance the head node launch template holds no reference to it , so 600 s covers the observed pre-instance window rather than a proven dependency. If it still times out, set head node bootstrap timeout explicitly in your defaults file — any value other than 2100 is used verbatim and disables the automatic increase, so set the total you want, not an increment: head node bootstrap timeout: 5400 The ceiling is 43200 12 hours , CloudFormation's own limit; larger values are clamped with a warning. The value cannot be changed on a running cluster — PCluster marks that setting UpdatePolicy.UNSUPPORTED , so changing the timeout means a full rebuild. To find where the time actually went, compare the CREATE IN PROGRESS / CREATE COMPLETE timestamps per resource: aws cloudformation describe-stack-events --stack-name <cluster name \ --query 'StackEvents . Timestamp,LogicalResourceId,ResourceStatus ' --output text | sort The preinstall / postinstall scripts exclude the kernel from their package upgrades precisely because a kernel bump added an unbounded initramfs rebuild to this window — see The Kernel Is Never Upgraded the-kernel-is-never-upgraded . Compute nodes fail to bootstrap after editing postinstall.j2: Postinstall runs on the head node and on every compute node, and it runs under set -euo pipefail — a non-zero exit fails the node's bootstrap.A compute-node failure does not stop the way a head-node failure does, which costs more, not less. clustermgtd marks the node DOWN , relaunches the node, and repeats until the queue's bootstrap-failure count reaches 10 , at which point clusterstatusmgtd puts the cluster in PROTECTED state and the stack fails — after ten instance launches and, in one measured case, 82 minutes. So a two-line mistake in a block that runs on compute nodes costs the whole build. Check the compute node's log stream, not just the head node's: the head node can finish cleanly while the rest of the fleet fails. Anything added there must declare where it belongs: - Work on shared storage /shared , /efs , /fsx , $HOME , /opt/parallelcluster/shared belongs inside a "$NODE TYPE" == "HeadNode" guard. These paths are NFS-exported from the head node, so N compute nodes writing to one file is a concurrent read-modify-write, and a write that root-squash denies aborts the node's own bootstrap. - Work on node-local state instance store, local packages, sysctls belongs outside the guard. - Packages needed on compute nodes must be installed outside the guard. The main apt-get block is head-node-only. NODE TYPE is read from cfn node type in /etc/parallelcluster/cfnconfig , which ParallelCluster writes before any custom action runs. There is no PARALLELCLUSTER NODE TYPE environment variable — reading one silently makes every compute node take the head-node path. HeadNode is the default only when the cfnconfig file is absent, which means the script is being re-run by hand off-cluster; a cfnconfig with no cfn node type , or an unrecognized value, exits 1 rather than skipping every guard. tests/test templates.py::TestPostinstallNodeTypeGating executes the rendered script for each node type with all external commands stubbed, so a block placed on the wrong side of a guard fails the suite. A bootstrap failure whose log ends on a cheerful-looking line: cfn-init captures stdout only — a failing command's stderr is written nowhere. So the last line of cfn-init-cmd.log is routinely the successful-looking start of the step that failed: Attempting uninstall: requests for a pip failure, a list of successful set on hold lines for an apt-mark exit 100, the luarocks download banner for a compiler error. Read the last line as where execution stopped, never as why . To get the reason, re-run the same command by hand on the node aws ssm start-session --target <instance-id , then execute the rendered /opt/parallelcluster/scripts/... step or the individual command and read its stderr directly. A related consequence: any block in the toolkit's own scripts that runs without set -x — the monitoring wrapper, for instance — leaves no trace in the log at all, so absence of a command from cfn-init-cmd.log is not evidence it did not run. Postinstall appears to do nothing: Check that postinstall.<cluster .sh in the cluster's S3 bucket is rendered shell and not raw Jinja2. The toolkit's templates are rendered by a template: task in src/create pcluster.yml ; only your own --post install script hook is copied verbatim. If the two are conflated, nodes run the hook and skip everything the toolkit's script does — Spack, Lmod, the package installs, /local scratch , and the GPU block. See Node Bootstrap Scripts node-bootstrap-scripts . EBS root volume tagging: May fail on macOS due to IAM tag permission restrictions. Build from an EC2 instance to avoid this. Interrupted build recovery: If make pcluster.py is interrupted mid-run, re-run the same command with the same flags. The tool detects the existing serial file under active clusters/<cluster name / and resumes from that identity — all AWS resource names S3 bucket, IAM role, IAM policy are re-derived from the same serial number, so no orphaned resources are left behind. make test pytest — template rendering + unit tests make lint ansible-lint on src/create pcluster.yml and src/delete pcluster.yml make shellcheck shellcheck on hpc-benchmark/hpc-benchmark.sh make test invokes .venv/bin/python -m pytest directly, so no manual venv activation is needed. The venv must exist python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt before running it. The top-level scripts fire a venv guard at import time; pytest must run inside .venv/ or collection fails with INTERNALERROR: SystemExit . macOS needs additional tools on PATH to run this suite a modern bash , GNU coreutils , shellcheck — see Development environment macOS /rmarable/ParallelClusterMaker/blob/main/INSTALL.md development-environment-macos in INSTALL.md. Without them, dozens of tests fail with messages like nproc: command not found or a silently wrong result from a bash-version mismatch — none indicate an actual defect. CI runs all three automatically on every push and pull request. The test job creates .venv/ explicitly and invokes .venv/bin/python -m pytest directly for the same reason. Template tests render with the same Jinja settings ansible.builtin.template uses — trim blocks=True , lstrip blocks=False — so what a test asserts on is the file the node actually receives. tests/test templates.py::TestTheTestEnvironmentMatchesAnsible reads both defaults back out of the installed Ansible instead of restating them, and fails if a playbook task overrides either. Every OS arm is rendered and executed. tests/conftest.py ships Ubuntu, RHEL 9, and Amazon Linux 2023 fixtures plus GPU variants of each , and TestPackageManagersMatchTheRenderedOs runs the rendered preinstall.j2 and postinstall.j2 under real bash with the package managers stubbed, asserting on the resulting command trace: no arm may execute the other family's package manager, and each arm must actually install its own sentinel packages rather than rendering to nothing. Adding a base os value means adding a fixture — an unexercised Jinja2 arm passes every text assertion written against it. A live end-to-end smoke test is available at tests/integration/run integration test.sh . It provisions a real cluster using your own defaults file, submits a Slurm job, verifies the output, and tears everything down. Integration tests are NOT run by make test, pytest, or CI. They build real infrastructure and must be invoked manually: source .venv/bin/activate ./tests/integration/run integration test.sh \ --az us-east-1a \ --owner test \ --email test@example.com \ --defaults tests/integration/itest defaults.yml \ --profile my-aws-profile \ --keep The first four flags are all required; the script has no --help , and any unrecognized argument exits 1. Start from tests/integration/itest defaults.yml.example — the copy at tests/integration/itest defaults.yml is gitignored, so account-specific values are never committed. A run costs roughly $0.21-$0.34 in EC2 charges. See tests/integration/README.md /rmarable/ParallelClusterMaker/blob/main/tests/integration/README.md for the flag reference, prerequisites, cost derivation, log paths, and exit codes. make lint exits 0 but emits a small number of warnings that are intentional and safe to ignore: | Warning | Reason | |---|---| yaml line-length — ssh/chown/cp commands | One-liners that are 162 chars 2 over limit ; splitting would harm readability | no-changed-when | pcluster CLI commands are inherently stateful; changed when on every poll would be misleading | ignore-errors | Intentional on cleanup tasks S3 bucket, SNS topic, IAM role that may not exist at delete time | no-handler | Deliberate pattern; notify/handler would require restructuring without benefit | These are all tracked in .ansible-lint under warn list with the same rationale. Potential future improvements, roughly ordered by impact: EasyBuild easyconfig workflow — accept a user-supplied list of EasyBuild module specs, download matching easyconfigs from the EasyBuild repository, build and install the modules on the head node, and run a smoke-test job for each one via Slurm. Useful for validating that a new OS or instance type can successfully build a site's standard software stack. Terraform / CDK parity — the toolkit is Ansible-native. A Terraform or AWS CDK implementation of the same lifecycle make / kill / access would fit more naturally into infrastructure-as-code pipelines that already use those tools. This software is licensed under the Apache License, Version 2.0 with the Commons Clause restriction. You may use, modify, and distribute it freely, but you may not sell it or offer it as a commercial product or service without the explicit written consent of Rodney Marable. See LICENSE for full terms. By using this software: - You accept all potential risks involved with your use of this Open Source software. - You agree that the author shall have no responsibility or liability for any losses or damages incurred in conjunction with your use of this Open Source software. - You acknowledge that bugs may still be present, unexpected behavior might be observed, and some features may not be completely documented. This software is authored by Rodney Marable in his individual capacity and is neither endorsed nor supported by Amazon Web Services. You cannot create cases with AWS Technical Support or engage AWS support engineers in public forums if you have any questions, problems, or issues using this software. "Play at your own risk " — Planet Patrol https://github.com/rmarable/ParallelClusterMaker/issues https://github.com/rmarable/ParallelClusterMaker/issues Pull requests welcome: https://github.com/rmarable/ParallelClusterMaker/pulls https://github.com/rmarable/ParallelClusterMaker/pulls AI-assisted contributions are welcome — see AI POLICY.md /rmarable/ParallelClusterMaker/blob/main/AI POLICY.md for submission guidance.