{"slug": "parallelclustermaker-cli-toolkit-to-manage-aws-parallelcluster-stacks", "title": "ParallelClusterMaker – CLI toolkit to manage AWS ParallelCluster stacks", "summary": "ParallelClusterMaker, an open-source CLI toolkit co-written with Anthropic's Claude Code, automates the creation and destruction of AWS ParallelCluster v3 stacks, enabling researchers and engineers to deploy HPC clusters on AWS without deep infrastructure expertise. The toolkit supports multi-node distributed training on GPU instances (g4dn, g4ad, g5, g5g, g6, p3, p3dn, p4d, p4de, p5) with EFA and EFA-GDR, traditional HPC workloads, and cost-sensitive batch jobs using spot capacity by default. It includes a benchmark suite (STREAM, OSU MPI, IOR, HPCG) for cluster validation and accepts AI-assisted contributions under a public policy requiring disclosure and human responsibility.", "body_md": "This Open Source CLI toolkit automates creation and destruction of [AWS ParallelCluster v3](https://github.com/aws/aws-parallelcluster) stacks. It lets researchers and engineers stand up a working HPC cluster on AWS without deep infrastructure expertise.\n\nThis codebase was co-written with [Claude Code](https://claude.ai/code) (Anthropic). AI-assisted contributions are accepted under a public policy ([AI_POLICY.md](/rmarable/ParallelClusterMaker/blob/main/AI_POLICY.md)): the tool or model must be disclosed, and a human remains responsible for every line submitted.\n\n**Training ML models at scale**— multi-node distributed training on GPU queues (`g4dn`\n\n,`g4ad`\n\n,`g5`\n\n,`g5g`\n\n,`g6`\n\n,`p3`\n\n,`p3dn`\n\n,`p4d`\n\n,`p4de`\n\n,`p5`\n\n), with EFA and EFA-GDR for inter-node collective communication and FSx for Lustre for high-throughput data loading.**Traditional HPC and scientific computing**— CFD, molecular dynamics, genomics, and other tightly-coupled MPI workloads, using EFA and dynamic placement groups for a low-latency interconnect.**Cluster and instance-type validation**— the built-in STREAM, OSU MPI, IOR, and HPCG benchmark suite measures memory bandwidth, MPI latency/bandwidth, filesystem I/O, and floating-point performance before committing a production workload to a given cluster shape.**Cost-sensitive batch and embarrassingly parallel work**— parameter sweeps, Monte Carlo simulation, rendering — using spot capacity, the default, where jobs can tolerate interruption.**On-demand research computing**— teams without dedicated infrastructure or a standing DevOps function can stand up a cluster for a research sprint and tear it down when finished, rather than carrying always-on cost.\n\nSee [INSTALL.md](/rmarable/ParallelClusterMaker/blob/main/INSTALL.md) for prerequisites, AWS account setup (VPC tagging, IAM\npermissions), and installation steps.\n\n**Scheduling and compute** — see [Networking and Compute](#networking-and-compute)\n\n- Slurm job scheduling\n- Separate CPU and GPU queues — the GPU queue exists only when\n`--gpu_instance_type`\n\nis set - Multi-instance-type queues via\n`--compute_instance_type`\n\nand`--gpu_instance_type`\n\n, each accepting a comma-separated list - Separate instance types and EBS configurations for the head node, CPU queue, and GPU queue\n- Spot capacity by default (\n`--cluster_type=spot`\n\n), with current market rates printed at build time - EFA on supported instance types in both queues (\n`--enable_efa`\n\n), with EFA-GDR enabled automatically on p4d/p4de/p5 - Dynamic EFA instance type lookup at launch time, with a static fallback list\n- Dynamic EC2 placement groups (\n`--placement_group`\n\n), applied to compute queues only - Selective HyperThreading disable (\n`--hyperthreading=false`\n\n) - Custom AMI support (\n`--custom_ami`\n\n) - Eight base operating systems across x86_64 and Graviton (\n`--base_os`\n\n):`ubuntu2204`\n\n,`ubuntu2404`\n\n,`ubuntu2204arm`\n\n,`ubuntu2404arm`\n\n,`rhel9`\n\n,`rhel9arm`\n\n,`alinux2023`\n\n,`alinux2023arm`\n\n- Dev / test / stage / prod operating levels (\n`--prod_level`\n\n)\n\n**Shared storage** — see [Storage](#storage)\n\n- Shared EBS at\n`/shared`\n\n, created unconditionally on every cluster - EFS at\n`/efs`\n\n(`--enable_efs`\n\n), with optional encryption (`--efs_encryption`\n\n) - FSx for Lustre at\n`/fsx`\n\n(`--enable_fsx`\n\n), with optional S3 hydration/dehydration (`--enable_fsx_hydration`\n\n) - External NFS automount from a site filer (\n`--enable_external_nfs`\n\n) — Vast, NetApp, WekaIO, Qumulo, etc. - Every filesystem, its mount point, and the resulting Spack install path named in the build summary and the SNS report\n\n**Software environment** — see [Software Environment](#software-environment)\n\n- Spack + Lmod for HPC software module management\n- Optional benchmark suite (\n`--enable_hpc_benchmarks`\n\n): STREAM, OSU MPI, IOR, and HPCG — STREAM recompiles per microarchitecture, so a GPU-partition job measures the GPU node - Optional Grafana/Prometheus monitoring stack (\n`--enable_monitoring`\n\n) via`aws-parallelcluster-monitoring`\n\n— Grafana dashboards, Prometheus, Slurm exporter, CloudWatch exporter\n\n**Operations tooling**\n\n`list_pcluster.py`\n\n— table of every cluster tracked by this repo, with optional live CloudFormation status; see[Listing Clusters](#listing-clusters)`stop_pcluster.py`\n\n/`start_pcluster.py`\n\n— stop or start the compute fleet without touching the head node; see[Stopping and Starting the Compute Fleet](#stopping-and-starting-the-compute-fleet)`check_pcluster.py`\n\n— pass/fail health check; exits 0 only when every check passes; see[Checking Cluster Health](#checking-cluster-health)`diagnose_pcluster.py`\n\n— raw diagnostic dump: CloudWatch bootstrap logs, node states, job failures, log tails; see[Diagnosing a Cluster](#diagnosing-a-cluster)`cost_pcluster.py`\n\n— actual spend per cluster from AWS Cost Explorer; see[Cost Reporting](#cost-reporting)`grafana_tunnel.py`\n\n— open or close the Grafana SSH tunnel for a monitoring-enabled cluster; see[Monitoring](#monitoring)`rotate_cluster_key.py`\n\n— rotate the cluster SSH keypair without a rebuild; see[SSH Key Management](#ssh-key-management)`manage_pcluster_queue.py`\n\n— add, remove, or list Slurm queues on a live cluster; see[Managing Queues on a Running Cluster](#managing-queues-on-a-running-cluster)\n\n**Security and lifecycle**\n\n- SSH private key stored in AWS Secrets Manager at cluster creation, recoverable via\n`retrieve_ssh_key..sh`\n\n; see[SSH Key Management](#ssh-key-management) - Resource tagging by owner, department, project, and operating level; see\n[Tagging](#tagging) - SNS notifications on stack create and destroy\n- Turbot environment support (\n`--turbot_account`\n\n) - Hourly cost estimate in the build summary — on-demand and spot price ranges per queue from the AWS Pricing API, degrading to\n`unavailable`\n\nwith a reason rather than crashing\n\nAll optional parameters have hardcoded defaults and can also be persisted in a YAML defaults file. `pcluster_defaults.yml`\n\nis the template — **copy the file before use**; the tracked version is shared and may be overwritten by updates, so never pass that one directly. The most commonly referenced defaults:\n\n| Parameter | Default |\n|---|---|\n`base_os` |\nubuntu2404 (hardcoded); `pcluster_defaults.yml` ships ubuntu2404arm to match the Graviton head node. Valid: `ubuntu2204` , `ubuntu2404` , `ubuntu2204arm` , `ubuntu2404arm` , `rhel9` , `rhel9arm` , `alinux2023` , `alinux2023arm` |\n`scheduler` |\nslurm |\n`headnode_instance_type` |\nc8g.xlarge (required — no default fallback) |\n`compute_instance_type` |\nc8g.2xlarge, c7g.2xlarge, c6g.2xlarge (comma list; empty = no CPU queue) |\n`gpu_instance_type` |\n(empty) — set to create a GPU queue (e.g. `p3.2xlarge,g5.2xlarge` ) |\n`gpu_root_volume_size` |\n250 GB (gp3) |\n`cluster_type` |\nspot |\n`initial_cpu_queue_size` |\n2 |\n`max_cpu_queue_size` |\n8 |\n`initial_gpu_queue_size` |\n2 |\n`max_gpu_queue_size` |\n8 |\n`maintain_cpu_initial_size` |\nfalse |\n`maintain_gpu_initial_size` |\nfalse |\n`scaledown_idletime` |\n5 min (global — applies to both queues) |\n`headnode_root_volume_size` |\n100 GB (gp3) |\n`compute_root_volume_size` |\n250 GB (gp3) |\n`ebs_shared_volume_size` |\n250 GB (gp3) |\n`fsx_size` |\n1200 GB |\n`placement_group` |\nNONE |\n`hyperthreading` |\ntrue |\n`enable_monitoring` |\nfalse |\n`enable_hpc_benchmarks` |\nfalse |\n`enable_efa` |\nfalse |\n`enable_gpu` |\nderived from `gpu_instance_type` — not user-settable |\n`head_node_bootstrap_timeout` |\n2100 s, raised automatically when `enable_efs` or `enable_fsx` is true |\n\nUse `--use_defaults=FILE`\n\nto load values from your own defaults file; CLI arguments always take precedence.\n\n```\n# Copy the template and customize it for your cluster\ncp pcluster_defaults.yml my-cluster_defaults.yml\n# Pass it at runtime\n./make_pcluster.py -N my-cluster -O rmarable -E rmarable@example.com -A us-east-1a \\\n --use_defaults=my-cluster_defaults.yml\n```\n\nName the file `_defaults.yml`\n\nto keep cluster namespaces scoped. `make_pcluster.py`\n\ndetects a matching file that was not loaded and prints a `*** WARNING ***`\n\nsuggesting the flag. Loading `pcluster_defaults.yml`\n\ndirectly is allowed but also warns.\n\n```\n./make_pcluster.py -N CLUSTER_NAME -O OWNER -E EMAIL -A AZ [options]\n```\n\nRequired arguments:\n\n| Flag | Description |\n|---|---|\n`-N` |\nCluster name (must start with a lowercase letter; lowercase letters, digits, hyphens only; no consecutive or trailing hyphens; max 27 characters) |\n`-O` |\nOwner username |\n`-E` |\nOwner email |\n`-A` |\nAvailability zone (e.g. `us-east-1a` ) — pass an AZ, not a region |\n\nAfter a cluster finishes building, the summary includes an estimated hourly cost for the head node and each queue at maximum fleet size:\n\n```\n Estimated hourly cost (max fleet, on-demand unless noted):\n Head node (c8g.2xlarge × 1): $0.319/hr\n CPU queue (c8g.2xlarge × 8): $2.552/hr [~$1.093/hr spot]\n GPU queue (p3.2xlarge × 4): $12.240/hr\n Note: spot prices are current ask; actual cost may differ.\n```\n\nFor multi-instance-type queues, a price range is shown (cheapest to most expensive type, all nodes at max count). If the AWS Pricing API is unreachable or the operator policy lacks `pricing:GetProducts`\n\n, the affected lines report `unavailable`\n\n— the head node line carries the reason — instead of crashing the build. Spot prices come from `ec2:DescribeSpotPriceHistory`\n\nusing the most recent ask and appear only when `--cluster_type=spot`\n\n.\n\nThe summary also names every shared filesystem, where it is mounted, and where Spack and the shared package tree install. Only the filesystems the cluster actually has appear — this example has all of them:\n\n```\n Shared storage:\n /shared EBS (gp3, 250 GB)\n /efs EFS (bursting throughput)\n /fsx FSx for Lustre (1200 GB)\n S3 import: s3://myImportBucket/data/in\n S3 export: s3://myExportBucket/data/out\n Hydrate: /usr/local/bin/import-s3-to-lustre.sh\n Export: /usr/local/bin/export-lustre-to-s3.sh\n Progress: /usr/local/bin/check-lustre-export-progress.sh\n /nfs external NFS (storage.domain.com)\n Spack and shared packages install under /fsx/pkg\n```\n\nThe `pkg_dir`\n\non the last line follows the storage precedence `fsx > efs > external NFS > shared EBS`\n\n, so it moves when a faster filesystem is present — see [Spack + Lmod](#spack--lmod). The same block appears in the emailed SNS build report, which is the copy that outlives the terminal scrollback.\n\nBasic cluster in us-east-1a using all defaults:\n\n```\n./make_pcluster.py -N pcluster-test-01 -O rmarable -E rodney.marable@gmail.com -A us-east-1a\n```\n\nEFS with encryption:\n\n```\n./make_pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N morpheus \\\n --enable_efs=true --efs_encryption=true\n```\n\nFixed-size compute pool tagged to the compbio department:\n\n```\n./make_pcluster.py -A eu-central-1a -O rmarable -E rodney.marable@gmail.com -N koolkeith \\\n --initial_cpu_queue_size=4 --max_cpu_queue_size=125 --maintain_cpu_initial_size=true \\\n --scaledown_idletime=30 --cluster_owner_department=compbio --project_id=polaroid \\\n --compute_instance_type=c5.2xlarge\n```\n\nEFA-enabled single-node cluster with performance tests:\n\n```\n./make_pcluster.py -A us-east-1a -N rimshot -O rmarable -E rmarable@amazon.com \\\n --compute_instance_type=c5n.18xlarge --initial_cpu_queue_size=1 \\\n --maintain_cpu_initial_size=true --enable_efa=true --enable_hpc_benchmarks=true\n```\n\nFSx for Lustre with S3 hydration (7.2 TB, 5 GB chunk size). Import and export must name the **same** bucket; the default `import`\n\nand `export`\n\nprefixes keep the two sides apart:\n\n```\n./make_pcluster.py -A us-west-2b -O rmarable -E rodney.marable@gmail.com -N louievega \\\n --enable_fsx=true --fsx_size=7200 --enable_fsx_hydration=true \\\n --fsx_s3_import_bucket=LouieVegaData --fsx_s3_export_bucket=LouieVegaData \\\n --fsx_chunk_size=5000\n```\n\nLarge GPU cluster with 3.6 PB Lustre, tagged for production:\n\n```\n./make_pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N gilgamesh \\\n --base_os=ubuntu2204 --headnode_instance_type=r5.xlarge \\\n --gpu_instance_type=p3.16xlarge --enable_fsx=true --fsx_size=3600000 \\\n --enable_fsx_hydration=true --fsx_s3_import_bucket=GilgameshData \\\n --fsx_s3_import_path=src/ --fsx_s3_export_bucket=GilgameshData \\\n --fsx_s3_export_path=output/ --prod_level=prod --max_gpu_queue_size=256\n```\n\nMixed CPU + GPU cluster (separate queues):\n\n```\n./make_pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N bifrost \\\n --base_os=ubuntu2404 --headnode_instance_type=c5.xlarge \\\n --compute_instance_type=c5.2xlarge,c5.4xlarge \\\n --gpu_instance_type=g5.2xlarge,g5.4xlarge\n```\n\nRHEL 9 cluster (login user is `ec2-user`\n\n, not `ubuntu`\n\n):\n\n```\n./make_pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N enkidu \\\n --base_os=rhel9 --headnode_instance_type=c5.xlarge \\\n --compute_instance_type=c5.2xlarge\n```\n\nAmazon Linux 2023 cluster on Graviton (login user is `ec2-user`\n\n):\n\n```\n./make_pcluster.py -A us-east-1a -O rmarable -E rodney.marable@gmail.com -N ninlil \\\n --base_os=alinux2023arm --headnode_instance_type=c8g.xlarge \\\n --compute_instance_type=c8g.2xlarge\n```\n\nBuilding from a custom AMI (must match base_os):\n\n```\n./make_pcluster.py -N starscream -O rmarable -E rodney.marable@gmail.com -A us-west-2a \\\n --enable_fsx=true --custom_ami=ami-123456789abc --base_os=ubuntu2204\n```\n\nA new stack typically takes approximately 30–35 minutes to build. Two measured `us-east-1`\n\nbuilds of the same cluster shape (`c5.xlarge`\n\nhead node, two queues, `ubuntu2404`\n\n): **34m 24s** with a 1200 GB FSx for Lustre filesystem, and **21m 14s** with EFS instead. Actual time depends on region, instance type availability, and which shared filesystems are enabled — FSx dominates when it is on, because its provisioning sits on the head node's critical path (see Troubleshooting).\n\n```\n./access_cluster.py -N CLUSTER_NAME\n```\n\nExample:\n\n```\n./access_cluster.py -N pcluster-test-01\nConnecting to head node of pcluster-test-01...\n./kill_pcluster.py -N CLUSTER_NAME -O OWNER -A AZ\n```\n\nTeardown takes 5–10 minutes. The cluster's EFS and FSx filesystems are deleted by CloudFormation, which ParallelCluster configures with a `Delete`\n\ndeletion policy at *creation* time — teardown cannot preserve them. The cluster's own S3 bucket (`parallelclustermaker-`\n\n) is the one resource teardown controls, and it holds the rendered cluster config and the bootstrap scripts. Benchmark results do **not** live there — they are synced to the long-lived `parallelclustermaker-results--`\n\nbucket, which teardown never touches — so keeping the per-cluster bucket is only useful for inspecting what a build actually deployed:\n\n```\n./kill_pcluster.py -N pcluster-test-01 -O rmarable -A us-east-1a \\\n --delete_s3_bucketname=false\n```\n\nTeardown is always manual and at your discretion — nothing in this toolkit schedules a cluster's destruction. Idle compute cost is bounded instead by `scaledown_idletime`\n\n: ParallelCluster terminates compute nodes that sit idle longer than that, scaling the fleet to zero on its own. A head node left running still bills, so run `kill_pcluster.py`\n\nwhen you are finished with a cluster.\n\nTeardown deletes ten kinds of resource after the CloudFormation stack is gone: the S3 bucket, the FSx hydration policy, the Grafana SSM parameter, the Secrets Manager secret, four managed IAM policies, the monitoring policy, the IAM role and its instance profile, the external NFS security group, and the SNS topic. Each step tolerates its own failure so that one AWS error cannot abandon the other nine — but every ignored failure is collected and reported, and teardown then exits non-zero:\n\n```\n=================================================================\n\nInitiated shutdown: 2026-08-03 @ 01:33:12\nCompleted shutdown: 2026-08-03 @ 01:41:48\n\nCluster osiris has been deleted, but 2 cleanup step(s) FAILED.\nThe following resources are still in the account and must be\nremoved by hand -- re-running kill_pcluster.py will not retry them\nonce osiris.serial has been deleted:\n\n - IAM managed policies pclustermaker-policy-osiris-00000000000000-{HeadNode-Compute,HeadNode-Storage,HeadNode-IAM,ComputeNode-Base}\n - IAM role and instance profile pclustermaker-role-osiris-00000000000000\n\nSerial number: osiris-00000000000000\n=================================================================\n```\n\nThe same list is included in the SNS destruction report. Act on it before the run's serial file is gone: `kill_pcluster.py`\n\nreads the cluster's serial number to build these resource names, so once `active_clusters//`\n\nis removed there is nothing left to retry with and the leftovers have to be found by hand. The usual cause is a missing operator IAM permission — see [Operator IAM permissions](/rmarable/ParallelClusterMaker/blob/main/INSTALL.md#operator-iam-permissions).\n\n`stop_pcluster.py`\n\nand `start_pcluster.py`\n\nstop or start the compute fleet while leaving the head node running. Use them to pause a cluster between job batches without paying for idle compute nodes.\n\n```\n./stop_pcluster.py -N CLUSTER_NAME [--wait]\n./start_pcluster.py -N CLUSTER_NAME [--wait]\n```\n\n| Flag | Short | Description |\n|---|---|---|\n`--cluster_name NAME` |\n`-N` |\nCluster name (required) |\n`--region REGION` |\n`-R` |\nAWS region (default: from cluster record) |\n`--wait` |\n`-W` |\nPoll until the fleet reaches the target state before exiting |\n\nWithout `--wait`\n\nthe request is submitted and the script exits immediately. With `--wait`\n\n, timestamped status lines are printed every 30 seconds until the fleet is `STOPPED`\n\nor `RUNNING`\n\n(up to 45 minutes).\n\n**Note:** stopping the fleet terminates all compute nodes immediately — in-flight Slurm jobs will be killed. Drain the queue first if needed.\n\n`list_pcluster.py`\n\nenumerates all clusters tracked by this repo (anything under `active_clusters/`\n\n) and prints a summary table from the local vars file — no AWS credentials required by default.\n\n```\n./list_pcluster.py [options]\n```\n\n| Flag | Short | Description |\n|---|---|---|\n`--live` |\n`-L` |\nCall `pcluster describe-cluster` for real-time status (one API call per cluster) |\n`--region REGION` |\n`-R` |\nFilter output to a single region |\n`--owner OWNER` |\n`-O` |\nFilter output to a single owner |\n`--wide` |\n`-W` |\nDisable column truncation |\n`--json` |\n`-J` |\nEmit a JSON array instead of a table |\n\nExample output:\n\n```\nCluster Owner Region Head Node CPU Types GPU Types Min/Max CPU Min/Max GPU Type Age Status\n------- -------- --------- ----------- ------------------ --------- ----------- ----------- -------- --- ------\nosiris rmarable us-east-1 c8g.2xlarge c8g.2xlarge, c7g.… - 0/8 -/- ondemand 4d LOCAL\n```\n\nWith `--live`\n\n, the `Status`\n\ncolumn shows `clusterStatus / cloudFormationStackStatus`\n\n(e.g. `CREATE_COMPLETE / CREATE_COMPLETE`\n\n). The two values diverge when a cluster update partially fails.\n\n`check_pcluster.py`\n\nruns a sequence of health checks against a named cluster and exits 0 only if every check passes.\n\n```\n./check_pcluster.py -N CLUSTER_NAME [--timeout SECONDS]\n```\n\n| Flag | Short | Description |\n|---|---|---|\n`--cluster_name NAME` |\n`-N` |\nCluster name (required) |\n`--timeout SECONDS` |\n`-T` |\nSSH timeout in seconds (default: 15, clamped to 1–300). The S3 check uses the boto3 default timeout and is unaffected. |\n\nChecks performed in order:\n\n**Vars file**— cluster record exists in`src/vars_files/.yml`\n\n**CloudFormation status**—`pcluster describe-cluster`\n\nreturns`clusterStatus=CREATE_COMPLETE`\n\n**Head node IP**— public or private IP present in the describe-cluster response** SSH reachability**—`ssh … echo OK`\n\nsucceeds**Slurm**—`sinfo -s`\n\nexits 0**Postinstall complete**— custom action marker file`/opt/parallelcluster/shared/custom_action_done`\n\nis present on the head node**Grafana health**—`curl -sk https://localhost:443/grafana/api/health`\n\nreturns`\"database\":\"ok\"`\n\n(checked only when`enable_monitoring=true`\n\n)**S3 bucket**—`s3.head_bucket`\n\nsucceeds (always run, independent of SSH)\n\nSSH-dependent checks (4–7) are `[SKIP]`\n\nped rather than `[FAIL]`\n\ned when SSH is unreachable, so a single SSH failure does not obscure the S3 result.\n\nExample output (all passing):\n\n```\nChecking cluster: my-cluster\n [PASS] vars file\n [PASS] CloudFormation status: CREATE_COMPLETE\n [PASS] head node IP: 54.1.2.3\n [PASS] SSH reachability\n [PASS] Slurm (sinfo -s)\n [PASS] postinstall complete\n [PASS] S3 bucket: my-cluster-parallelcluster-bucket\n\nAll checks passed — my-cluster is healthy.\n```\n\n`diagnose_pcluster.py`\n\ngoes deeper than `check_pcluster.py`\n\n— it collects raw diagnostic data rather than pass/fail checks, and always exits 0 so output is never suppressed by an early failure.\n\n```\n./diagnose_pcluster.py -N CLUSTER_NAME [options]\n```\n\n| Flag | Default | Description |\n|---|---|---|\n`--cluster_name NAME` / `-N` |\nrequired | Cluster name |\n`--region REGION` / `-R` |\nfrom vars file | Override AWS region |\n`--timeout SECONDS` / `-T` |\n20 | SSH timeout |\n`--cw_lines N` |\n50 | CloudWatch log lines per stream (max 500) |\n`--log_lines N` |\n30 | Local log file tail lines (max 200) |\n`--hours N` |\n24 | `sacct` lookback window in hours |\n`--no_cw` |\noff | Skip CloudWatch section (omit the flag to include CW output) |\n\nSections produced:\n\n**CloudWatch: head node bootstrap logs**— last N lines from`cfn-init`\n\n,`cloud-init-output`\n\n, and`cinc_client`\n\nstreams. PCluster appends the stack's creation timestamp to the log group name (`/aws/parallelcluster/-`\n\n), so the group is discovered by prefix rather than constructed; the selected group name is printed above the streams. Rebuilds of the same cluster name leave older groups behind — PCluster retains them by design — and the newest is used. Requires`logs:DescribeLogGroups`\n\n,`logs:DescribeLogStreams`\n\n,`logs:FilterLogEvents`\n\n, and`logs:GetLogEvents`\n\non the operator identity (all included in the operator policy). Pass`--no_cw`\n\nto skip this section if permissions are unavailable.**Slurm node states**—`sinfo -N -l`\n\noutput; nodes not in`idle`\n\n/`mix`\n\n/`alloc`\n\nare annotated with`<-- not idle`\n\n.**Recent Slurm job failures**—`sacct`\n\nfiltered to`FAILED`\n\n,`CANCELLED`\n\n,`TIMEOUT`\n\n,`NODE_FAIL`\n\nstates. Prints a note if no results (Slurm accounting is not enabled by default in PCluster v3).**Local log tails**— last N lines of`/var/log/parallelcluster/slurm_resume.log`\n\n,`slurm_suspend.log`\n\n,`/var/log/cinc/client.log`\n\n,`/var/log/cloud-init-output.log`\n\n.**Postinstall marker**— confirms`/opt/parallelcluster/shared/custom_action_done`\n\nis present; prints the cluster serial number for cross-referencing S3 benchmark results.\n\nExample output:\n\n```\nDiagnosing cluster: my-cluster (us-east-1)\n serial: 20260804-abc123\n\n=== CloudWatch: head node bootstrap logs ===\n\n log group: /aws/parallelcluster/my-cluster-202608041130\n\n --- cfn-init ---\n 2026-08-04 17:24:45 ConfigSet: default\n 2026-08-04 17:24:49 Install packages: ok\n ...\n\n=== Slurm node states (sinfo -N -l) ===\n\n NODELIST NODES PARTITION STATE CPUS MEMORY REASON\n compute-1 1 cpu idle 4 8000 none\n compute-2 1 cpu drain 4 8000 maintenance <-- not idle\n\n=== Recent Slurm job failures (last 24h) ===\n\n No failed jobs in the last 24h\n (If this is unexpected, Slurm accounting may not be enabled.)\n\n=== Local log tails (last 30 lines each) ===\n\n --- /var/log/parallelcluster/slurm_resume.log ---\n ...\n\n=== Postinstall marker ===\n\n [PASS] /opt/parallelcluster/shared/custom_action_done (serial: 20260803-abc123)\n```\n\n`cost_pcluster.py`\n\nqueries AWS Cost Explorer by the `ClusterID`\n\nresource tag to show actual spend per cluster. Results reflect billing data with a 24-hour lag.\n\n```\n./cost_pcluster.py [options]\n```\n\n| Flag | Short | Description |\n|---|---|---|\n`--cluster_name NAME` |\n`-N` |\nSingle cluster (default: all in `active_clusters/` ) |\n`--owner OWNER` |\n`-O` |\nFilter to clusters owned by this user |\n`--days N` |\n`-D` |\nLookback window in days (default: 30, max: 365) |\n`--json` |\n`-J` |\nEmit JSON array instead of a table |\n\n**Prerequisites:** the operator's IAM user/role needs `ce:GetCostAndUsage`\n\nand `ce:ListCostAllocationTags`\n\n. The `ClusterID`\n\ntag key must also be activated as a cost allocation tag in the AWS Billing console (Console → Billing → Cost allocation tags → User-defined tags). If the tag is not activated, all results show `$0.00`\n\n— the script detects this and prints a warning before running queries.\n\nExample output:\n\n```\nAWS Cost Explorer — last 30 days (24-hour data lag applies)\n\nPeriod: 2026-06-24 – 2026-07-24\n\nCluster Owner Region Cost ($)\n------- ----- ------ --------\nosiris rmarable us-east-1 $47.82\nbifrost rmarable us-west-2 $12.10\n```\n\nWith no storage flags at all, a cluster gets one shared filesystem — EBS at `/shared`\n\n— plus a node-local scratch directory on every instance. Everything else is opt-in and additive; enabling EFS or FSx does not replace shared EBS.\n\n| Mount point | Type | Scope | Default state |\n|---|---|---|---|\n`/shared` |\nEBS volume | Shared across head node and all compute nodes | Always created |\n`/local_scratch` |\nRoot EBS volume, or NVMe instance store | Local to one instance; not shared | Always created on every instance |\n`/efs` |\nEFS | Shared | Opt-in (`--enable_efs=true` ) |\n`/fsx` |\nFSx for Lustre | Shared | Opt-in (`--enable_fsx=true` ) |\n`/nfs/` |\nExternal NFS | Shared, filer-provided | Opt-in (`--enable_external_nfs=true` ) |\n\nEFS and FSx filesystems are destroyed with the cluster and cannot be preserved by teardown — see [Deleting a Cluster](#deleting-a-cluster). Move anything durable to S3 or to an external NFS filer before tearing down.\n\nWhichever of these a cluster ends up with, the build summary names each one and its mount point — see [Building a Cluster](#building-a-cluster) for the block.\n\nNot optional. The `SharedStorage:`\n\nblock in `templates/config.pcluster.j2`\n\nemits its `Ebs`\n\nentry unconditionally, so every cluster gets one shared EBS volume mounted on the head node and every compute node. There is no enable flag.\n\n| Parameter | Default |\n|---|---|\n`--ebs_shared_dir` |\n`/shared` |\n`--ebs_shared_volume_size` |\n250 (GB; max 16,384) |\n`--ebs_shared_volume_type` |\n`gp3` (`gp2` , `gp3` , `io1` , `io2` , `st1` ) |\n`--ebs_shared_volume_iops` |\n3000 (emitted for `gp3` , `io1` , `io2` only) |\n`--ebs_shared_volume_throughput` |\n125 (MB/s; emitted for `gp3` only) |\n`--ebs_encryption` |\n`false` |\n\n`--ebs_encryption`\n\nalso governs the head node, CPU queue, and GPU queue root volumes.\n\nLocal to a single instance, not shared. `templates/postinstall.j2`\n\ncreates `/local_scratch`\n\nas a sticky-bit directory on the root EBS volume and symlinks `/scratch`\n\nto it. When `enable_gpu`\n\nis `true`\n\nand NVMe instance store devices are present, `/local_scratch`\n\nis backed by those instead — one device is formatted XFS, several are assembled into a RAID0 array. See [GPU](#gpu) for the device detection logic.\n\nPostinstall is registered as an `OnNodeConfigured`\n\ncustom action on the head node and on every compute queue (`templates/config.pcluster.j2`\n\n), so `/local_scratch`\n\nis created on every instance. This matters for the instance-store path: NVMe instance store exists only on compute instances, so a head-node-only registration leaves the RAID0 block unreachable in practice. See [Node Bootstrap Scripts](#node-bootstrap-scripts) for how the script gets to the node.\n\nData in `/local_scratch`\n\ndoes not survive instance termination, and compute nodes terminate on scale-down.\n\nEnable with `--enable_efs=true`\n\n. Mounted at `/efs`\n\non all instances. Costs almost nothing in build time: measured on a `generalPurpose`\n\n/`bursting`\n\nfilesystem with one mount target, the filesystem completed in 4 seconds and the mount target in 1m 33s, both finishing well before the Route53 zone and the compute-fleet nested stack that actually gate the head node's launch. A multi-AZ cluster creates one mount target per subnet and has not been timed. Configure with `--efs_encryption`\n\n, `--efs_performance_mode`\n\n(`generalPurpose`\n\nor `maxIO`\n\n), and `--efs_throughput_mode`\n\n(`bursting`\n\n, `provisioned`\n\n, or `elastic`\n\n).\n\nEnable with `--enable_fsx=true`\n\n. Mounted at `/fsx`\n\n. `--fsx_size`\n\nmust be a positive multiple of 1200 GB; the default and minimum is 1200. `--fsx_chunk_size`\n\n(the S3 imported-file chunk size, default 1024 MB) must fall between 1,024 MB (1 GB) and 512,000 MB (500 GB).\n\nRequires `--enable_fsx_hydration=true`\n\n. Setting any `--fsx_s3_*`\n\nvalue without that flag is an error, as is setting the flag without `--enable_fsx=true`\n\n.\n\n**One bucket, two prefixes.** FSx for Lustre requires the export bucket to be the same bucket as the import bucket — from AWS's own API model for `CreateFileSystem`\n\n: *\"The Amazon S3 export bucket must be the same as the import bucket specified by ImportPath.\"* Only the prefixes may differ.\n\n`--fsx_s3_import_bucket`\n\nand `--fsx_s3_export_bucket`\n\nexist as separate parameters because `ImportPath`\n\nand `ExportPath`\n\nare separate FSx concepts — `ImportPath`\n\nis where the filesystem is seeded from at creation, `ExportPath`\n\nis where `lfs hsm_archive`\n\nwrites changed files back — but both must name the same bucket. A mismatch is rejected before the build starts rather than twenty minutes in, at FSx creation:\n\n```\nERROR: Lustre hydration: fsx_s3_export_bucket (out-bucket) must name the same\nbucket as fsx_s3_import_bucket (in-bucket) ...\n```\n\nThe supported shape:\n\n```\n--enable_fsx_hydration=true --fsx_s3_import_bucket=ResearchData \\\n--fsx_s3_import_path=input/ --fsx_s3_export_bucket=ResearchData \\\n--fsx_s3_export_path=output/\n```\n\nThe three accepted cases:\n\n| Import | Export | Result |\n|---|---|---|\n`--fsx_s3_import_bucket=data --fsx_s3_import_path=in/` |\n`--fsx_s3_export_bucket=data --fsx_s3_export_path=out/` |\nThe supported shape. No warning. |\n`--fsx_s3_import_bucket=data --fsx_s3_import_path=in/` |\nunset | Warns; the export side falls back to the import bucket and path, so hydration and dehydration both use `s3://data/in/` |\n`...=data --fsx_s3_import_path=shared/` |\n`...=data --fsx_s3_export_path=shared/` |\nWarns: dehydrated files overwrite the hydration source |\n\nAn unset `--fsx_s3_import_bucket`\n\nwith `--enable_fsx_hydration=true`\n\nis an error — there is nothing to hydrate from. Use `--enable_fsx=true`\n\non its own for an empty Lustre filesystem.\n\nBoth sides are validated before the build starts: the bucket must exist (`head_bucket`\n\n, with a distinct error for a 403 so an access-denied bucket policy is not reported as a missing bucket), and the prefix must contain at least one object. An empty or misspelled path fails immediately rather than at FSx creation time.\n\nNote that the export fallback overwrites `--fsx_s3_export_path`\n\nwith the import path. To keep the two prefixes separate, name the bucket in both parameters rather than relying on the fallback.\n\n**Helper scripts on the head node.** When hydration is enabled, postinstall writes three one-line wrappers around Lustre's HSM commands into `/usr/local/bin`\n\n(mode 755, root-owned). The build summary prints their paths. They are not run automatically — hydration itself happens at FSx creation from `ImportPath`\n\n; these are for pulling and pushing on demand afterward:\n\n| Script | What it does |\n|---|---|\n`import-s3-to-lustre.sh` |\n`lfs hsm_restore` over every file under `/fsx` — fetches file contents for entries FSx has listed but not yet copied down |\n`export-lustre-to-s3.sh` |\n`lfs hsm_archive` over every file under `/fsx` — writes changed files back to the export prefix |\n`check-lustre-export-progress.sh` |\nCounts outstanding `ARCHIVE` actions, so an export in flight can be polled |\n\nAll three walk the whole filesystem with `find`\n\n, so on a large tree they take a while and are best run under `nohup`\n\n(the import and export scripts already background themselves). Verified on a live `rhel9arm`\n\nhead node with a 1200 GB Lustre filesystem and `lfs 2.15.6`\n\n.\n\nEnable with `--enable_external_nfs=true --external_nfs_server=storage.domain.com`\n\n. Exports are mounted under `/nfs/`\n\nand the export list lives in `templates/external_nfs_mount_list.j2`\n\n— uncomment the lines matching the paths your filer serves. A security group permitting NFS traffic is attached to the head node and compute queues automatically.\n\nImportant:The toolkit auto-discovers VPCs and subnets by convention when explicit values are not provided. Auto-discovery picks the AWS default VPC and thefirstsubnet returned by EC2 in each AZ. EC2 does not guarantee subnet ordering, so results are non-deterministic in accounts with multiple subnets per AZ.Do not rely on auto-discovery for production clusters.Always specify networking resources explicitly.\n\n| Parameter | Description |\n|---|---|\n`--vpc_name` |\nVPC `Name` tag to use (default: `vpc_default` — the account's default VPC) |\n`--headnode_subnet_id` |\nExplicit subnet ID for the head node; overrides auto-discovery |\n`--compute_subnet_ids` |\nComma-separated subnet IDs for the CPU compute fleet; overrides auto-discovery |\n`--compute_az` |\nComma-separated AZs for the CPU compute fleet (default: same as `--az` ) |\n`--use_private_compute_subnet` |\nOnly auto-discover private subnets for CPU nodes (`true` /`false` , default: `false` ) |\n`--gpu_subnet_ids` |\nComma-separated subnet IDs for the GPU queue; falls back to `compute_subnet_ids` if empty |\n`--gpu_az` |\nComma-separated AZs for the GPU queue; falls back to `compute_az` then `--az` if empty |\n`--use_private_gpu_subnet` |\nOnly auto-discover private subnets for GPU nodes (`true` /`false` , default: `false` ) |\n\nSubnets and security groups are generated as part of the CloudFormation stack — the toolkit does not manage them independently outside of the stack lifecycle.\n\n**Single-AZ cluster (explicit subnets — recommended):**\n\n```\n./make_pcluster.py -N prod01 -O rmarable -E rmarable@example.com -A us-east-1a \\\n --vpc_name=my-hpc-vpc \\\n --headnode_subnet_id=subnet-0abc123 \\\n --compute_subnet_ids=subnet-0abc123\n```\n\n**Multi-AZ compute fleet spanning three AZs:**\n\n```\n./make_pcluster.py -N bigcluster -O rmarable -E rmarable@example.com -A us-east-1a \\\n --vpc_name=my-hpc-vpc \\\n --headnode_subnet_id=subnet-0abc123 \\\n --compute_az=us-east-1a,us-east-1b,us-east-1c \\\n --compute_subnet_ids=subnet-0abc123,subnet-0def456,subnet-0ghi789\n```\n\n**Private compute subnet (head node public, compute private):**\n\n```\n./make_pcluster.py -N private01 -O rmarable -E rmarable@example.com -A us-east-1a \\\n --vpc_name=my-hpc-vpc \\\n --headnode_subnet_id=subnet-0abc123 \\\n --compute_subnet_ids=subnet-0private1 \\\n --use_private_compute_subnet=true\n```\n\nSet `--gpu_instance_type`\n\nto create a dedicated GPU queue alongside (or instead of) the CPU queue. `enable_gpu`\n\nis derived — it is `true`\n\nwhenever `gpu_instance_type`\n\nis non-empty, and cannot be set by the user. GPU families (`g4dn`\n\n, `g4ad`\n\n, `g5`\n\n, `g5g`\n\n, `g6`\n\n, `p3`\n\n, `p3dn`\n\n, `p4d`\n\n, `p4de`\n\n, `p5`\n\n) are rejected from `compute_instance_type`\n\n, and non-GPU types are rejected from `gpu_instance_type`\n\n.\n\nBoth `compute_instance_type`\n\nand `gpu_instance_type`\n\naccept comma-separated lists for multi-instance-type queues:\n\n```\n# GPU-only cluster\n./make_pcluster.py ... --gpu_instance_type=p3.2xlarge,g5.2xlarge\n\n# CPU + GPU queues\n./make_pcluster.py ... --compute_instance_type=c5.2xlarge --gpu_instance_type=g5.2xlarge\n```\n\n**The head node's instance type is independent of the queues.** A CPU head node fronting both a CPU queue and a GPU queue — `c5.xlarge`\n\nhead, `c5`\n\n`compute`\n\npartition, `g5`\n\n`gpu`\n\npartition — is the common layout and is fully supported, including for the benchmark suite. `enable_cpu_queue`\n\nand `enable_gpu_queue`\n\nare each derived from whether the corresponding instance-type flag is non-empty, and the two `SlurmQueues`\n\nentries are emitted independently; nothing about the head node's own family gates either one. There is no need for a GPU head node to run GPU jobs, and paying for an idle GPU on the head node buys nothing.\n\n**The one real constraint is architecture, not CPU-vs-GPU.** The head node and every queue instance type must share one CPU architecture, because a cluster runs one architecture-specific base OS image. `make_pcluster.py`\n\nchecks this at creation time against `ec2:DescribeInstanceTypes`\n\n— which is authoritative, and covers families the hardcoded ARM prefix list does not yet name — and refuses the build with the offending types named. So an x86_64 head node with a Graviton `g5g`\n\nGPU queue is rejected up front rather than failing twenty minutes into the build. Mixing *microarchitectures* within an architecture (Intel head node, AMD GPU nodes) is fine and is handled automatically by the benchmark suite.\n\n**What the GPU postinstall block does:** `templates/postinstall.j2`\n\ngates a GPU block on `enable_gpu == 'true'`\n\n. Postinstall runs as an `OnNodeConfigured`\n\ncustom action on the head node and on every compute queue, so this block runs on the GPU compute nodes — which are the only instances that have NVMe instance store to configure.\n\n- NVMe instance store detection — scans\n`/sys/block/nvme*`\n\nand accepts a device only if it passes three independent filters: the model string matches`AmazonEC2NVMeInstanceStorage`\n\n(whitespace stripped),`holders/`\n\nis empty, and`blkid`\n\nfinds no filesystem signature. The model check keeps EBS volumes, which also appear as`/dev/nvme*`\n\n, from being formatted. The other two keep the toolkit off devices ParallelCluster already claimed, and neither subsumes the other — a device inside an LVM volume has holders but no signature of its own, while a formatted-but-unmounted device has a signature and no holders **ParallelCluster usually claims these devices first.** The`aws-parallelcluster-environment::ephemeral_drives`\n\ncookbook runs*before*`OnNodeConfigured`\n\nand, on any instance type with instance store, puts every such device into an LVM physical volume, formatted`ext4`\n\nand mounted on`/scratch`\n\n. On that common case the toolkit's block correctly does nothing and`/local_scratch`\n\nis a symlink to PCluster's`/scratch`\n\n— verified on a live`g4dn.xlarge`\n\n. Without the holders/`blkid`\n\nfilters,`mkfs.xfs`\n\non a claimed device fails with`Device or resource busy`\n\n, which fails the node- Single unclaimed device: formatted XFS, mounted at\n`/local_scratch`\n\nwith`noatime,nodiratime,nofail`\n\n- Multiple unclaimed devices (\n`p4d.24xlarge`\n\nhas 8×1000 GB,`p5.48xlarge`\n\nhas 8×3800 GB, per`aws ec2 describe-instance-types`\n\n): RAID0 via`mdadm`\n\n, mounted at`/local_scratch`\n\n- No instance store present (e.g.\n`p3.2xlarge`\n\n):`/local_scratch`\n\nremains a sticky-bit directory on the root EBS volume `htop`\n\ninstalled by the GPU block itself, since the main package block is head-node-only and does not run on compute nodes;`nvtop`\n\nis head-node-only because it lives outside the default repositories (`multiverse`\n\non Ubuntu, EPEL on RHEL 9) and a compute node's package index is whatever the AMI shipped. A compute node refreshes its index first (`apt-get update`\n\n/`dnf makecache`\n\n) because`OnNodeStart`\n\n— and therefore preinstall's refresh — never runs there. Both installs are non-fatal (`|| echo \"WARNING: ...\"`\n\n), the only ones in the file: they are diagnostics nothing in the job path imports, and one transient mirror outage would otherwise count toward the 10-failure protected-mode threshold and cost the entire stack\n\n**EFA GPUDirect RDMA (GDR):** When `--enable_efa=true`\n\nand any GPU queue instance type is `p4d.24xlarge`\n\n, `p4de.24xlarge`\n\n, or `p5.48xlarge`\n\n, `GdrSupport: true`\n\nis added to the GPU queue EFA config automatically.\n\n**GPU volume settings:** The GPU queue uses its own root volume parameters (`--gpu_root_volume_size`\n\n, `--gpu_root_volume_type`\n\n, `--gpu_root_volume_iops`\n\n, `--gpu_root_volume_throughput`\n\n) independent of the CPU queue.\n\n**CUDA / drivers:** PCluster's official deep learning AMIs include NVIDIA drivers. Pass `--custom_ami=`\n\nto use a pre-built DLAMI or a custom AMI with pinned driver versions.\n\nEnable with `--enable_efa=true`\n\n. Supported on every `base_os`\n\nvalue the toolkit accepts. Requires a supported instance type (c5n.18xlarge, hpc6a.48xlarge, hpc7a.96xlarge, hpc7g.16xlarge, etc.). A dynamic placement group is created automatically.\n\nEFA (Elastic Fabric Adapter) is an OS-bypass network interface that gives tightly-coupled, latency-sensitive MPI jobs a much faster interconnect than standard TCP/IP networking — closer to InfiniBand than to a regular NIC. EFA matters most for multi-node jobs with heavy collective communication (large all-reduce, halo exchanges, etc.); a single-node job, or one bound by disk or ordinary network I/O, sees little benefit. On p4d, p4de, and p5 instances, EFA-GDR (GPUDirect RDMA) is enabled automatically, letting the network adapter read and write GPU memory directly instead of staging through host memory — which matters for multi-node GPU jobs.\n\n**Not yet verified on hardware.** EFA-enabled builds have not been run against real instances; the config generation and instance-type gating are unit-tested, but the actual interconnect behavior is unconfirmed.\n\nEnable with `--placement_group=DYNAMIC`\n\n. PCluster creates one managed cluster placement group per queue, applied to the CPU and GPU compute queues only. The head node is never placed in a placement group. `--enable_efa=true`\n\nsets this to `DYNAMIC`\n\nautomatically when the setting is still `NONE`\n\n.\n\nDisable with `--hyperthreading=false`\n\n.\n\nHyperThreading (Intel) / SMT (AMD) exposes each physical CPU core as two logical vCPUs sharing the same execution resources. Many HPC workloads — anything compute- or memory-bandwidth-bound rather than I/O-bound — run faster with HyperThreading disabled, since two threads contending for one core's resources often costs more than the extra thread gains. Disabling HyperThreading also changes the per-node rank count Slurm submits with: `cpu_ranks_per_node`\n\ndivides vCPUs by `DefaultThreadsPerCore`\n\ninstead of always halving (see [Job Submission](#job-submission)).\n\n`manage_pcluster_queue.py`\n\nedits a live cluster's config file to add, remove, or list Slurm queue stanzas without rebuilding the cluster. Changes take effect after a `pcluster update-cluster`\n\n; the compute fleet must be stopped before updating. Pass `-W`\n\nto have the script handle the full stop/update/restart cycle automatically.\n\n```\n./manage_pcluster_queue.py -N -A -T [options]\n```\n\nRequired arguments:\n\n| Flag | Description |\n|---|---|\n`-N` |\nCluster name |\n`-A` |\nAction: `add` , `remove` , or `list` |\n`-T` |\nQueue type: `compute` (CPU) or `gpu` . Required for `add` ; ignored by `remove` and `list` . |\n\n```\n./manage_pcluster_queue.py -N -A list\n# Add a spot CPU queue\n./manage_pcluster_queue.py -N osiris -A add -T compute \\\n -E c5.2xlarge,c5.4xlarge -C spot \\\n -Q compute-spot-overflow\n\n# Add an on-demand GPU queue\n./manage_pcluster_queue.py -N osiris -A add -T gpu \\\n -E p3.2xlarge -C ondemand \\\n -Q gpu-ondemand\n\n# Add a GPU queue with custom scaling\n./manage_pcluster_queue.py -N osiris -A add -T gpu \\\n -E g5.2xlarge,g5.4xlarge \\\n --initial_size 1 --max_size 4 --maintain_initial_size true\n./manage_pcluster_queue.py -N osiris -A remove -Q compute-spot-overflow\n```\n\n**Automated ( -W/--wait):** the script stops the fleet, applies the config, and restarts the fleet, printing status every 30 seconds. It warns that the operation can take up to 30 minutes; each individual poll loop times out at 45 minutes. Run inside\n\n`screen`\n\nor `tmux`\n\nto avoid losing the session mid-update.\n\n```\n./manage_pcluster_queue.py -N osiris -A add -T compute -E c5.xlarge --wait\n```\n\n**Manual (default):** after `add`\n\nor `remove`\n\n, the script prints the exact commands needed:\n\n- Stop the compute fleet\n- Wait for STOPPED status\n- Run\n`pcluster update-cluster`\n\nwith the updated config - Wait for UPDATE_COMPLETE\n- Restart the fleet\n\n- Compute queues reject GPU instance types and vice versa\n- Mixed x86/Graviton instance types in a single queue are rejected\n- A new queue whose architecture differs from the running cluster's is rejected — a cluster runs one architecture-specific base OS image\n- The last remaining queue on a cluster cannot be removed\n`p4d`\n\n/`p4de`\n\n/`p5`\n\nGPU instances support EFA-GDR; the script prints a reminder to enable it manually if needed\n\n`--base_os`\n\naccepts eight values, in two package-manager families:\n\n`base_os` |\nArchitecture | Package manager | Login user |\n|---|---|---|---|\n`ubuntu2204` |\nx86_64 | apt | `ubuntu` |\n`ubuntu2404` |\nx86_64 | apt | `ubuntu` |\n`ubuntu2204arm` |\nGraviton (aarch64) | apt | `ubuntu` |\n`ubuntu2404arm` |\nGraviton (aarch64) | apt | `ubuntu` |\n`rhel9` |\nx86_64 | dnf | `ec2-user` |\n`rhel9arm` |\nGraviton (aarch64) | dnf | `ec2-user` |\n`alinux2023` |\nx86_64 | dnf | `ec2-user` |\n`alinux2023arm` |\nGraviton (aarch64) | dnf | `ec2-user` |\n\nThe `arm`\n\nsuffix is the toolkit's, not ParallelCluster's — it drives instance-architecture validation and is stripped before the value reaches PCluster's `Os:`\n\nfield, so `ubuntu2404arm`\n\n, `rhel9arm`\n\n, and `alinux2023arm`\n\nbecome `ubuntu2404`\n\n, `rhel9`\n\n, and `alinux2023`\n\nthere. Mixing an ARM `base_os`\n\nwith an x86_64 instance type (or vice versa) is rejected before anything is created; do not mix architectures across node types either.\n\n`preinstall.j2`\n\nand `postinstall.j2`\n\nbranch on the OS family and install the equivalent package sets; the dnf side branches again where Amazon Linux 2023 and RHEL 9 differ. A few differences are worth knowing about if you write a `--post_install_script`\n\nhook:\n\n- On RHEL 9, EPEL and CodeReady Builder are enabled during postinstall because\n`lua-devel`\n\n,`lua-posix`\n\n,`lua-filesystem`\n\n, and`tcllib`\n\nare in neither baseos nor appstream. EPEL is installed from its release RPM URL, since`epel-release`\n\nis not packaged in RHEL itself. - On Amazon Linux 2023, neither EPEL nor CRB is used, because\n`epel-release`\n\nis not packaged for al2023 at all and everything the toolkit needs is already in the core repo. Four packages the RHEL arm installs are absent from al2023 on both architectures, so that arm differs accordingly:`luarocks`\n\n(the three Lua rocks come from the core repo as RPMs instead, and the luarocks build step is a no-op),`tcllib`\n\n(unused by the toolkit — Lmod uses`tcl`\n\nitself), and`nvtop`\n\n(so GPU clusters get`htop`\n\nonly, on both node types). Do not add any of them back for symmetry with RHEL; each one would fail the node. `pip3`\n\nis called with`--break-system-packages`\n\non Ubuntu and without it on RHEL 9 and Amazon Linux 2023, both of which ship a pip that predates PEP 668 and rejects the flag.- Every\n`pip3 install`\n\non a node carries`--ignore-installed`\n\n, on both families. pip cannot uninstall a distribution whose`dist-info`\n\nhas no`RECORD`\n\nfile, and distro-packaged Python modules routinely ship exactly that — Ubuntu's`python3-pip`\n\nand RHEL 9's`python3-requests`\n\nare both confirmed cases. Any install that resolves to replacing one dies at`Attempting uninstall:`\n\n, which under`set -euo pipefail`\n\nfails the node's bootstrap.`--ignore-installed`\n\nskips the uninstall and installs over the top; nothing in the toolkit needs the distro package removed. This is a different problem from`--break-system-packages`\n\n, which only permits writing into the system tree. `nvtop`\n\n(GPU clusters) is installed on the head node only, on Ubuntu and RHEL 9 — the package sits outside the default repositories in each, and the operator logs into the head node rather than a compute node. Amazon Linux 2023 skips`nvtop`\n\nentirely, since the package isn't available there.- On RHEL 9,\n`bc`\n\nis installed explicitly because Lmod's`./configure`\n\nhard-quits without`bc`\n\n(`You must have bc in your path. Quitting!`\n\n) rather than degrading.`bc`\n\nis on the Ubuntu and Amazon Linux 2023 package lines too, deliberately: those AMIs happen to ship`bc`\n\nincidentally, and depending on what a base image carries by accident is how the RHEL gap stayed hidden. (Upstream's own monitoring installer claims`bc`\n\nis absent from the default al2023 repos; the repo metadata says otherwise on both architectures.) - The dnf kernel exclusions cover both Lustre spellings —\n`kmod-lustre*`\n\nand`lustre-client*`\n\n— because the two distros name the client differently. Amazon Linux 2023 has no`kmod-lustre*`\n\npackage at all, so the RHEL glob alone would silently protect nothing there.\n\n**Both Amazon Linux 2023 arms are validated on live cluster builds.** `alinux2023`\n\nreached `CREATE_COMPLETE`\n\non a `c5.xlarge`\n\nhead node with EFS, a `c5.2xlarge`\n\nCPU queue, two `g4dn.xlarge`\n\nGPU nodes, benchmarks, and monitoring; `alinux2023arm`\n\nreached `CREATE_COMPLETE`\n\non a `c8g.xlarge`\n\nhead node with EFS, a 1200 GB FSx for Lustre filesystem, a `c8g.2xlarge`\n\nCPU queue, benchmarks, and monitoring. Every package claim above was confirmed against the head node's and every compute node's own bootstrap logs on both architectures, not merely against the stack's exit status: `luarocks`\n\n, `epel`\n\n, `tcllib`\n\n, and `nvtop`\n\nappear nowhere; the three Lua rocks install as core-repo RPMs; `bc`\n\nis already present and Lmod's `./configure`\n\nfinds it; and `dnf update`\n\nupgrades a single package with no kernel, dracut, or initramfs activity. The Docker Compose CLI plugin is staged and checksum-verified per architecture — `docker-compose-linux-x86_64-v2.29.7`\n\nand `docker-compose-linux-aarch64-v2.29.7`\n\nrespectively — and upstream's `github.com`\n\ndownload is removed from the extracted tree on both.\n\n**Both RHEL 9 arms are validated on live cluster builds.** `rhel9`\n\nreached `CREATE_COMPLETE`\n\non a `c5.xlarge`\n\nhead node with EFS, a CPU queue, a `g4dn.xlarge`\n\nGPU queue, benchmarks, and monitoring; `rhel9arm`\n\nreached `CREATE_COMPLETE`\n\non a `c8g.xlarge`\n\nhead node with EFS, a 1200 GB FSx for Lustre filesystem, benchmarks, and monitoring. The whole RPM bootstrap path is confirmed on both architectures: EPEL by release-RPM URL, the CodeReady Builder repository id, all three luarocks rocks compiling against `lua-devel`\n\nwith no separate header package, and `dnf update`\n\nupgrading `dracut`\n\nitself while installing zero `kernel*`\n\npackages and regenerating no initramfs. All eight pip pins resolved from `manylinux_2_17_aarch64`\n\nwheels on Graviton.\n\nTwo stages run on every node, in this order:\n\n**The toolkit's own scripts**—`templates/preinstall.j2`\n\nand`templates/postinstall.j2`\n\n, rendered per cluster with that cluster's variables (OS, storage layout,`pkg_dir`\n\n, GPU flags) and uploaded to the cluster's S3 bucket as`preinstall..sh`\n\nand`postinstall..sh`\n\n. These are the toolkit's own work: base packages, Spack, Lmod,`/local_scratch`\n\n, the benchmark suite, and the GPU block.**Your hook**— the script named by`--pre_install_script`\n\n/`--post_install_script`\n\n, copied verbatim and uploaded under its own basename. Defaults are`scripts/pre-deployment.sh`\n\nand`scripts/post-deployment.sh`\n\n, both no-op placeholders. Put site-specific customization here; do not edit the toolkit templates to add it.\n\nThe stages are wired as a PCluster `Sequence`\n\n, so stage 2 runs only if stage 1 succeeded. `OnNodeStart`\n\n(preinstall) runs on the head node only — repeating the Python/pip/AWS CLI install on every scale-up event would add boot latency to every compute node for no benefit. `OnNodeConfigured`\n\n(postinstall) runs on the head node and on every compute queue, since that is where node-local work like `/local_scratch`\n\nbelongs. When `--enable_monitoring=true`\n\nthe monitoring installer is appended as a third stage.\n\nPaths passed to `--pre_install_script`\n\n/ `--post_install_script`\n\nare relative to the repository root.\n\n`preinstall.j2`\n\nupgrades the AMI's packages — `apt-get dist-upgrade`\n\non Ubuntu, `dnf update`\n\non RHEL 9 — but holds back the running kernel along with the out-of-tree modules built against it. Ubuntu does this with `apt-mark hold`\n\non the installed `linux-*`\n\npackages; RHEL 9 with `--exclude='kernel*' --exclude='kmod-lustre*' --exclude='efa*'`\n\n. Two independent reasons:\n\n**A kernel replacement triggers an initramfs rebuild whose runtime is unbounded**, and it runs inside the window CloudFormation gives the head node to signal success. Real builds failed this way on both families: on the PCluster AMI of the day a full upgrade crossed a kernel boundary and was still rebuilding when the wait condition expired, and on the RHEL 9 AMI a full update crossed`5.14.0-611.55.1.el9_7`\n\n→`5.14.0-687.30.1.el9_8`\n\nwith dracut still running when CloudFormation gave up.**PCluster's AMI ships EFA and Lustre kernel modules built against the kernel it boots.** Replacing that kernel without rebuilding them risks losing the interconnect or the Lustre client on the next boot.\n\nThe two mechanisms are equivalent in effect but not in shape: `apt-mark`\n\npins packages by name, so the Ubuntu path first enumerates the installed `linux-*`\n\npackages and filters them to the ones dpkg reports as actually installed. `--exclude`\n\ntakes a glob resolved when dnf builds the transaction, so the RHEL 9 path needs no enumeration and holds on every AMI revision regardless of which updates happen to be pending — every RHEL 9 kernel subpackage (`kernel`\n\n, `-core`\n\n, `-modules`\n\n, `-modules-core`\n\n, `-modules-extra`\n\n, `-headers`\n\n, `-devel`\n\n, `-tools`\n\n) matches `kernel*`\n\n.\n\nThe upgrade itself is deliberately kept, because `preinstall.j2`\n\ninstalls the Python development headers and `numpy`\n\n/`scipy`\n\n/`pandas`\n\n/`matplotlib`\n\ncompile from source wherever pip finds no wheel — which every `*arm`\n\nvalue of `base_os`\n\ncan hit. If you add a package that needs a newer kernel, pin it in your own `--pre_install_script`\n\nhook and reboot deliberately outside the bootstrap window; do not remove the exclusions.\n\nEvery stack includes [Spack](https://spack.io/) and [Lmod](https://github.com/TACC/Lmod) for HPC software module management. Spack is cloned into `/pkg/spack`\n\n, where ``\n\nis the first available of `/fsx`\n\n, `/efs`\n\n, `/nfs`\n\n, or the shared EBS mount.\n\nBoth are built on the head node only. Compute nodes mount the same shared storage and inherit the installation, so having every scaling node repeat the clone and `chown -R`\n\nwould be wasted boot time on a tree that is already populated.\n\n** MODULEPATH comes from Spack, not from Lmod's compiled-in root.** Lmod is configured with\n\n`--with-module-root-path=/pkg/modulefiles`\n\n, but that setting is only ever read by Lmod's `init/profile`\n\nscript, and postinstall installs `init/sh`\n\ninstead — which defines the `module`\n\n, `ml`\n\n, and `clearMT`\n\nshell functions and sets `MODULESHOME`\n\n, with no reference to `MODULEPATH`\n\nat all. What populates `MODULEPATH`\n\non a login shell is `/etc/profile.d/lmod_spack.sh`\n\nsourcing Spack's `setup-env.sh`\n\n, which appends Spack's own module roots. Consequently `/pkg/modulefiles`\n\nis not created by the install and will not exist on a fresh cluster — nothing reads that path, so this is expected rather than a broken installation. If you want to hand-place modulefiles outside Spack, create that directory yourself and add its path to `MODULEPATH`\n\n(or to `LMOD_SITE_MODULEPATH`\n\n) from your own `--post_install_script`\n\nhook.A default Slurm submission script (`scripts/sbatch_default_submission_script.sh`\n\n) is rendered from the toolkit's `scripts/`\n\ndirectory into the login user's home directory during cluster creation (`ubuntu`\n\nor `ec2-user`\n\n, depending on `base_os`\n\n). Copy it to shared storage and customize:\n\n```\ncp ~/sbatch_default_submission_script.sh /fsx/scratch/my_project/\nsbatch /fsx/scratch/my_project/sbatch_default_submission_script.sh\n```\n\n** --partition and --ntasks are derived from this cluster's own shape, not hardcoded.** A cluster with a CPU queue gets\n\n`--partition=compute`\n\nand `--ntasks=`\n\n; a GPU-only cluster gets `--partition=gpu`\n\nand `--ntasks=`\n\n. Both matter:**A GPU-only cluster has no**, and`compute`\n\npartition`sbatch`\n\nrejects an invalid partition outright before the job ever runs.`enable_cpu_queue`\n\nis derived from`--compute_instance_type`\n\n, so the script follows whichever queues the cluster actually has.**The rank count is vCPUs, divided by**— never halved unconditionally. Graviton reports one thread per core, so halving there would request half the cores every ARM node has. Where a queue holds several instance types, the count is the smallest one's, since that is the only value every node in the queue can satisfy.`DefaultThreadsPerCore`\n\nwhen`--hyperthreading=false`\n\nNote this is a *core* count, which is not the same as the GPU benchmark's `--ntasks-per-node`\n\n: `job_hpc-benchmark.sh`\n\nmatches its rank count to the number of NVIDIA devices per node, while this script asks for cores. A `p3.2xlarge`\n\nhas 1 GPU and 8 vCPUs, and a general-purpose job wants the 8.\n\nEnable with `--enable_hpc_benchmarks=true`\n\n. Cluster creation deploys the benchmark suite to a personalized working directory on the head node — `~/hpc-benchmark///slurm/`\n\n— holding the driver, a `job_hpc-benchmark.sh`\n\nrendered for this cluster's queue layout, and a `README-PERFORMANCE.md`\n\nnaming this cluster's own paths. Postinstall also installs the Python plotting dependencies (`matplotlib`\n\n, `numpy`\n\n, `pandas`\n\n, `scipy`\n\n, `seaborn`\n\n) and drops a second copy of the driver at `~/hpc-benchmark/hpc-benchmark.sh`\n\n; that copy is restored from S3 on every head node boot, so a replaced EBS root never loses the driver — the only file staged there. Work out of the personalized directory, since that is the one with the rendered job script.\n\n**These commands run on the cluster head node** (SSH in via `./access_cluster.py`\n\nfirst):\n\n```\ncd ~/hpc-benchmark///slurm\nmodule load openmpi\n./hpc-benchmark.sh install # build STREAM, OSU, IOR, HPCG (~5 min)\n./hpc-benchmark.sh run --tests stream,osu,ior,hpcg\n./hpc-benchmark.sh report\n```\n\n**Results are preserved on teardown.** `kill_pcluster.py`\n\nsyncs benchmark results from the head node to `s3://parallelclustermaker-results--/hpc-benchmark-results///`\n\nbefore deleting the cluster. That bucket is **not** the per-cluster bucket — keyed on your account and region, created on the first build that enables benchmarks, and never deleted by this toolkit — so results from multiple builds of the same cluster name land in separate serial-number subdirectories and accumulate rather than overwriting each other. This is the one bucket you are expected to prune by hand.\n\n**STREAM is rebuilt per node.** STREAM is compiled `-march=native`\n\n, which binds the binary to the *microarchitecture* rather than the architecture — a `c5.xlarge`\n\nhead node is Intel Skylake and a `g5.xlarge`\n\nGPU node is AMD Zen 3, and `uname -m`\n\ncalls both `x86_64`\n\n. `install`\n\ncaches the source and `run`\n\ncompiles `bin/stream-`\n\nlocally, so a job on a GPU partition measures that node's real bandwidth with no manual step. OSU, IOR, and HPCG are built by `configure`\n\n/`make`\n\nwithout `-march=native`\n\nand are portable across microarchitectures, so those stay in `bin/`\n\nguarded by an architecture stamp.\n\n**OSU builds itself on the GPU node when the head node cannot.** `install`\n\nenables CUDA only when the node running `install`\n\nhas both an NVIDIA device and a CUDA toolkit, because OSU's `configure`\n\naborts outright on a missing `-lcuda`\n\n, `-lcudart`\n\n, or `cuda.h`\n\nrather than degrading — deriving that from a cluster-level flag would fail the whole install and take STREAM, IOR, and HPCG down with OSU. A CPU head node therefore produces a host-to-host OSU, and the first GPU-partition job builds a CUDA-enabled tree under `bin/osu-cuda`\n\non the GPU node itself and writes `osu/latency_cuda.txt`\n\nand `osu/bandwidth_cuda.txt`\n\nalongside the host-to-host results. `bin/`\n\nis shared storage, so later GPU jobs reuse that build. This step can never fail the run: a node that can't build CUDA support still writes the host-to-host results, with the reason printed.\n\n**Both partitions are benchmarkable from a CPU head node.** The shipped `job_hpc-benchmark.sh`\n\nis submittable as-is: its `#SBATCH --partition=`\n\nand `--ntasks-per-node=`\n\ndirectives are rendered from this cluster's queue layout. A GPU-only cluster has no `compute`\n\npartition, so a hardcoded partition would be rejected by `sbatch`\n\nbefore anything ran; on such a cluster the rank count is the NVIDIA GPU count reported for the queue's instance types, so one rank lands per GPU. On a cluster with both queues the script targets `compute`\n\n, and the GPU run is the same script with two directives overridden:\n\n```\nsbatch job_hpc-benchmark.sh # compute partition\nsbatch --partition=gpu --ntasks-per-node= job_hpc-benchmark.sh\n```\n\nThe exact second command, with this cluster's GPU count already substituted, is in the commented GPU section at the bottom of the rendered script.\n\nSee `hpc-benchmark/README-PERFORMANCE.md`\n\nfor full documentation. The copy in the personalized working directory on the head node is the same document with this cluster's name, owner, and paths substituted in; the copy in this repo spells them ``\n\nand ``\n\nand is not deployed.\n\nEnable with `--enable_monitoring=true`\n\n(default: `false`\n\n). Deploys the [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) Grafana/Prometheus stack. The same install script runs on every node; it branches on node type internally, so the head node gets the full stack and compute nodes get only a metrics exporter.\n\n**On the head node:**\n\n- Grafana (port 443, self-signed TLS)\n- Prometheus, pushgateway, cloudwatch-exporter, nginx, node_exporter (Docker Compose)\n- prometheus-slurm-exporter (systemd, scrapes Slurm metrics every 30 s)\n\n**On each compute node:**\n\n- node_exporter (Docker Compose)\n- NVIDIA DCGM exporter on GPU instances, from\n`compose/compute.gpu.yml`\n\nupstream — requires the`nvidia-container-toolkit`\n\n, which the upstream installer provides\n\n**Access Grafana:**\n\n`grafana_tunnel.py`\n\nopens or closes the tunnel without locating the per-cluster script manually:\n\n```\n# Open tunnel (background SSH, prints URL and password command)\n./grafana_tunnel.py -N CLUSTER_NAME\n\n# Use a different local port\n./grafana_tunnel.py -N CLUSTER_NAME -P 9443\n\n# Close the tunnel\n./grafana_tunnel.py -N CLUSTER_NAME --stop\n```\n\nThe script verifies that monitoring is enabled for the cluster and exits with a message if it is not. The generated per-cluster script also works directly:\n\n```\n# Open\n./active_clusters//grafana_tunnel..sh\n\n# Close\n./active_clusters//grafana_tunnel..sh 8443 stop\n```\n\nThen open `https://localhost:8443/grafana/`\n\nin your browser and accept the self-signed certificate warning. Pass a different local port if 8443 is in use.\n\nIf the head node has a public IP you can also open `https:///grafana/`\n\ndirectly (requires port 443 open in the security group).\n\n**Retrieve the Grafana admin password:**\n\n```\naws ssm get-parameter \\\n --name \"/parallelcluster//grafana/admin-password\" \\\n --with-decryption \\\n --query \"Parameter.Value\" --output text\n```\n\n**Note:** PCluster head nodes ship a web server running on port 80 — `apache2`\n\non Ubuntu, `httpd`\n\non RHEL 9. The monitoring installer stops and disables whichever is present so the nginx container can bind ports 80 and 443.\n\n**Monitoring is verified on RHEL 9, on both architectures.** Upstream's `detect_platform`\n\nresolves `PLATFORM_ID=platform:el9`\n\non x86_64 and Graviton alike, and the container stack reached `Started`\n\non both — so the v2.6 installer is arch-agnostic on el9, not x86-only.\n\n**IAM:** Monitoring permissions are granted via a separate managed policy `-HeadNode-Monitoring`\n\n(8 statements, ~1,550 bytes minified). It is created and attached during `make_pcluster.py`\n\nand deleted during `kill_pcluster.py`\n\n.\n\n**Supply chain:** The `aws-parallelcluster-monitoring`\n\ntarball is downloaded from GitHub at cluster-build time, checksum-verified, and staged in the cluster's S3 bucket. Head nodes pull from S3, not GitHub, so private-subnet nodes and air-gapped environments work without internet access.\n\nOn Amazon Linux 2023 that applies to the Docker Compose CLI plugin as well. AL2023 does not package `docker-compose-plugin`\n\n, so upstream's installer downloads the binary from `github.com`\n\non every node at boot, with no integrity check — which fails outright on a private subnet. The toolkit instead downloads the binary once at build time, verifies the checksum against `--docker_compose_checksum_x86_64`\n\n/ `--docker_compose_checksum_aarch64`\n\n(both defaulted in `pcluster_defaults.yml`\n\nand matching Docker's own published sums for `v2.29.7`\n\n), stages the verified binary to S3, and installs from there on every node before the monitoring installer runs. The wrapper also deletes upstream's download from the extracted tree so that download can never overwrite the verified copy, and fails the build by name if that edit stops matching in a future monitoring release. Because the download is removed rather than reused, `--docker_compose_version`\n\nis the toolkit's own pin and need not match upstream's. The Ubuntu and RHEL 9 arms install the plugin from a signed distro repository and use none of this.\n\n**Version:** Pin a specific release tag with `--monitoring_version=v2.6`\n\n(default).\n\n**Custom AMI:** The Docker Compose installation adds several minutes to head node boot time. For production clusters or fast iteration, build a custom AMI with the monitoring stack pre-installed.\n\nAt cluster creation, the SSH private key is stored in AWS Secrets Manager at:\n\n```\nparallelcluster///ssh-private-key\n```\n\nThe secret is deleted automatically when `kill_pcluster.py`\n\nruns, along with the EC2 keypair and the local `.pem`\n\nfile. If the CloudFormation stack deletion itself reaches `DELETE_FAILED`\n\n(e.g. a dangling ENI, security group, or EFA interface), all three are deliberately preserved instead — this keeps the head node reachable for manual troubleshooting until the stack is fully torn down. Re-run `kill_pcluster.py`\n\nafter resolving the CloudFormation dependency to complete cleanup.\n\n**If your local .pem file is lost**, retrieve it from Secrets Manager:\n\n```\nactive_clusters//retrieve_ssh_key..sh\n# optionally specify a destination:\nactive_clusters//retrieve_ssh_key..sh --out /tmp/mykey.pem\n```\n\n`access_cluster.py`\n\ncalls the retrieve script automatically if the local key is missing.\n\n**Rotating the SSH keypair** without rebuilding the cluster:\n\n```\n./rotate_cluster_key.py -N -A \n# preview what will change:\n./rotate_cluster_key.py -N -A --dry_run\n```\n\nRotation: generates a new ED25519 keypair locally, appends the public key to `~/.ssh/authorized_keys`\n\non the head node, imports it as the new EC2 keypair, updates the Secrets Manager secret, overwrites the local `.pem`\n\n, and deletes the old EC2 keypair.\n\n**IAM requirements** (operator's user/role — not the cluster head node role):\n\n| Permission | Purpose |\n|---|---|\n`secretsmanager:CreateSecret` |\nStore key at cluster creation |\n`secretsmanager:PutSecretValue` |\nUpdate key on rotation |\n`secretsmanager:GetSecretValue` |\nRetrieve key via retrieve script |\n`secretsmanager:DeleteSecret` |\nRemove key on teardown |\n`ec2:ImportKeyPair` |\nRegister new public key during rotation |\n`ec2:DeleteKeyPair` |\nRemove old keypair after rotation |\n\nAll resources are tagged automatically:\n\n| Tag | Source |\n|---|---|\n`ClusterID` |\n`--cluster_name` |\n`ClusterOwner` |\n`--cluster_owner` |\n`ClusterOwnerEmail` |\n`--cluster_owner_email` |\n`ClusterOwnerDepartment` |\n`--cluster_owner_department` |\n`ClusterStackType` |\nParallelCluster |\n`ClusterOSType` |\n`--base_os` |\n`ClusterScheduler` |\n`--scheduler` |\n`ClusterSerialNumber` |\ngenerated |\n`ProdLevel` |\n`--prod_level` |\n`ProjectID` |\n`--project_id` (if set) |\n`DEPLOYMENT_DATE` |\ngenerated |\n\nSupported departments: `analytics`\n\n, `clinical`\n\n, `commercial`\n\n, `compbio`\n\n, `compchem`\n\n, `datasci`\n\n, `design`\n\n, `development`\n\n, `hpc`\n\n, `imaging`\n\n, `manufacturing`\n\n, `medical`\n\n, `modeling`\n\n, `operations`\n\n, `proteomics`\n\n, `qa`\n\n, `research`\n\n, `robotics`\n\n, `scicomp`\n\n.\n\nParallelClusterMaker does **not** create or modify VPCs, subnets, gateways, routes, or Transit Gateways. It creates IAM roles, policies, and instance profiles scoped to each individual cluster stack. Templates are in `templates/`\n\nand can be customized. If you hit permissions errors, the IAM policy template is the right starting point for working with your security team.\n\n**IAM permissions:** Check `templates/HeadNode-Compute.json_src`\n\n, `HeadNode-Storage.json_src`\n\n, `HeadNode-IAM.json_src`\n\n, `ComputeNode-Base.json_src`\n\n, and (when `enable_monitoring=true`\n\n) `HeadNode-Monitoring.json_src`\n\n. The instance policy is split by role into five managed policies to stay under the IAM managed policy size limit. IAM role and instance-profile resources use both flat-name ARNs (`parallelcluster--*`\n\n) and path-based ARNs (`parallelcluster//*`\n\n) — PCluster v3 uses the latter for compute fleet roles. Most build failures trace back to missing IAM permissions.\n\n**Spot capacity:** Compute nodes that fail to launch surface as a `ComputeFleet - CREATE_FAILED`\n\nCloudFormation error. Retry the build or switch to `--cluster_type=ondemand`\n\n.\n\n**Build fails with HeadNodeWaitCondition timing out (CREATE_FAILED, 0 of 1 signals):** The head node did not finish bootstrapping inside the window CloudFormation allows. Note that this clock starts when CloudFormation\n\n*begins creating the wait condition*— before the head node instance exists — and shared filesystem provisioning sits on the head node's critical path. A 1200 GB FSx for Lustre filesystem measured 17m 22s, over half of PCluster's stock 2100 s budget, before the instance had even launched.\n\nThe toolkit raises `head_node_bootstrap_timeout`\n\nautomatically for this: +1800 s when `enable_fsx`\n\nis true, +600 s when `enable_efs`\n\nis true, whichever is larger (the two provision concurrently, so the head node waits on the slower one, not the sum). A `*** INFO ***`\n\nline names the filesystem that drove the increase.\n\nBoth allowances are measured against live builds rather than estimated. FSx: a second 1200 GB filesystem took 19m 20s before its instance existed, and the build completed in 34m 24s of the 3900 s granted. EFS (`generalPurpose`\n\n/`bursting`\n\n, one mount target): the filesystem itself completed in 4 s and its mount target in 1m 33s, with the instance appearing 4m 24s in and the wait condition satisfied in 20m 52s of the 2700 s granted — so the EFS allowance carries roughly 2.3x headroom. Two caveats worth knowing before you rely on them: a multi-AZ EFS cluster creates one mount target per subnet and has not been timed, and on the EFS build the mount target was not in fact what gated the instance (the head node launch template holds no reference to it), so 600 s covers the observed pre-instance window rather than a proven dependency.\n\nIf it still times out, set `head_node_bootstrap_timeout`\n\nexplicitly in your defaults file — any value other than 2100 is used verbatim and disables the automatic increase, so set the *total* you want, not an increment:\n\n```\nhead_node_bootstrap_timeout: 5400\n```\n\nThe ceiling is 43200 (12 hours), CloudFormation's own limit; larger values are clamped with a warning. The value cannot be changed on a running cluster — PCluster marks that setting `UpdatePolicy.UNSUPPORTED`\n\n, so changing the timeout means a full rebuild. To find where the time actually went, compare the `CREATE_IN_PROGRESS`\n\n/`CREATE_COMPLETE`\n\ntimestamps per resource:\n\n```\naws cloudformation describe-stack-events --stack-name \\\n --query 'StackEvents[].[Timestamp,LogicalResourceId,ResourceStatus]' --output text | sort\n```\n\nThe `preinstall`\n\n/`postinstall`\n\nscripts exclude the kernel from their package upgrades precisely because a kernel bump added an unbounded initramfs rebuild to this window — see [The Kernel Is Never Upgraded](#the-kernel-is-never-upgraded).\n\n**Compute nodes fail to bootstrap after editing postinstall.j2:** Postinstall runs on the head node\n\n*and*on every compute node, and it runs under\n\n`set -euo pipefail`\n\n— a non-zero exit fails the node's bootstrap.A compute-node failure does not stop the way a head-node failure does, which costs more, not less. `clustermgtd`\n\nmarks the node `DOWN`\n\n, relaunches the node, and repeats until the queue's bootstrap-failure count reaches **10**, at which point `clusterstatusmgtd`\n\nputs the cluster in `PROTECTED`\n\nstate and the stack fails — after ten instance launches and, in one measured case, 82 minutes. So a two-line mistake in a block that runs on compute nodes costs the whole build. Check the *compute* node's log stream, not just the head node's: the head node can finish cleanly while the rest of the fleet fails.\n\nAnything added there must declare where it belongs:\n\n- Work on shared storage (\n`/shared`\n\n,`/efs`\n\n,`/fsx`\n\n,`$HOME`\n\n,`/opt/parallelcluster/shared`\n\n) belongs inside a`[ \"$NODE_TYPE\" == \"HeadNode\" ]`\n\nguard. These paths are NFS-exported from the head node, so N compute nodes writing to one file is a concurrent read-modify-write, and a write that root-squash denies aborts the node's own bootstrap. - Work on node-local state (instance store, local packages, sysctls) belongs outside the guard.\n- Packages needed on compute nodes must be installed outside the guard. The main\n`apt-get`\n\nblock is head-node-only. `NODE_TYPE`\n\nis read from`cfn_node_type`\n\nin`/etc/parallelcluster/cfnconfig`\n\n, which ParallelCluster writes before any custom action runs. There is no`PARALLELCLUSTER_NODE_TYPE`\n\nenvironment variable — reading one silently makes every compute node take the head-node path.`HeadNode`\n\nis the default only when the cfnconfig file is absent, which means the script is being re-run by hand off-cluster; a cfnconfig with no`cfn_node_type`\n\n, or an unrecognized value, exits 1 rather than skipping every guard.\n\n`tests/test_templates.py::TestPostinstallNodeTypeGating`\n\nexecutes the rendered script for each node type with all external commands stubbed, so a block placed on the wrong side of a guard fails the suite.\n\n**A bootstrap failure whose log ends on a cheerful-looking line:** `cfn-init`\n\ncaptures **stdout only** — a failing command's `stderr`\n\nis written nowhere. So the last line of `cfn-init-cmd.log`\n\nis routinely the successful-looking start of the step that failed: `Attempting uninstall: requests`\n\nfor a pip failure, a list of successful `set on hold`\n\nlines for an `apt-mark`\n\nexit 100, the `luarocks`\n\ndownload banner for a compiler error. Read the last line as *where* execution stopped, never as *why*. To get the reason, re-run the same command by hand on the node (`aws ssm start-session --target `\n\n, then execute the rendered `/opt/parallelcluster/scripts/...`\n\nstep or the individual command) and read its stderr directly. A related consequence: any block in the toolkit's own scripts that runs without `set -x`\n\n— the monitoring wrapper, for instance — leaves no trace in the log at all, so absence of a command from `cfn-init-cmd.log`\n\nis not evidence it did not run.\n\n**Postinstall appears to do nothing:** Check that `postinstall..sh`\n\nin the cluster's S3 bucket is rendered shell and not raw Jinja2. The toolkit's templates are rendered by a `template:`\n\ntask in `src/create_pcluster.yml`\n\n; only your own `--post_install_script`\n\nhook is copied verbatim. If the two are conflated, nodes run the hook and skip everything the toolkit's script does — Spack, Lmod, the package installs, `/local_scratch`\n\n, and the GPU block. See [Node Bootstrap Scripts](#node-bootstrap-scripts).\n\n**EBS root volume tagging:** May fail on macOS due to IAM tag permission restrictions. Build from an EC2 instance to avoid this.\n\n**Interrupted build recovery:** If `make_pcluster.py`\n\nis interrupted mid-run, re-run the same command with the same flags. The tool detects the existing serial file under `active_clusters//`\n\nand resumes from that identity — all AWS resource names (S3 bucket, IAM role, IAM policy) are re-derived from the same serial number, so no orphaned resources are left behind.\n\n```\nmake test # pytest — template rendering + unit tests\nmake lint # ansible-lint on src/create_pcluster.yml and src/delete_pcluster.yml\nmake shellcheck # shellcheck on hpc-benchmark/hpc-benchmark.sh\n```\n\n`make test`\n\ninvokes `.venv/bin/python -m pytest`\n\ndirectly, so no manual venv activation is\nneeded. The venv must exist (`python3.12 -m venv .venv && .venv/bin/pip install -r requirements.txt`\n\n)\nbefore running it. The top-level scripts fire a venv guard at import time; pytest must run\ninside `.venv/`\n\nor collection fails with `INTERNALERROR: SystemExit`\n\n.\n\nmacOS needs additional tools on `PATH`\n\nto run this suite (a modern `bash`\n\n, GNU `coreutils`\n\n,\n`shellcheck`\n\n) — see [Development environment (macOS)](/rmarable/ParallelClusterMaker/blob/main/INSTALL.md#development-environment-macos)\nin INSTALL.md. Without them, dozens of tests fail with messages like `nproc: command not found`\n\nor a silently wrong result from a bash-version mismatch — none indicate an actual defect.\n\nCI runs all three automatically on every push and pull request. The `test`\n\njob creates `.venv/`\n\nexplicitly and invokes `.venv/bin/python -m pytest`\n\ndirectly for the same reason.\n\nTemplate tests render with the same Jinja settings `ansible.builtin.template`\n\nuses —\n`trim_blocks=True`\n\n, `lstrip_blocks=False`\n\n— so what a test asserts on is the file the node\nactually receives. `tests/test_templates.py::TestTheTestEnvironmentMatchesAnsible`\n\nreads both\ndefaults back out of the installed Ansible instead of restating them, and fails if a playbook\ntask overrides either.\n\nEvery OS arm is rendered and executed. `tests/conftest.py`\n\nships Ubuntu, RHEL 9, and Amazon\nLinux 2023 fixtures (plus GPU variants of each), and `TestPackageManagersMatchTheRenderedOs`\n\nruns the\nrendered `preinstall.j2`\n\nand `postinstall.j2`\n\nunder real `bash`\n\nwith the package managers\nstubbed, asserting on the resulting command trace: no arm may execute the other family's\npackage manager, and each arm must actually install its own sentinel packages rather than\nrendering to nothing. Adding a `base_os`\n\nvalue means adding a fixture — an unexercised\nJinja2 arm passes every text assertion written against it.\n\nA live end-to-end smoke test is available at `tests/integration/run_integration_test.sh`\n\n.\nIt provisions a real cluster using your own defaults file, submits a Slurm job, verifies\nthe output, and tears everything down.\n\n**Integration tests are NOT run by make test, pytest, or CI.** They build real\ninfrastructure and must be invoked manually:\n\n```\nsource .venv/bin/activate\n\n./tests/integration/run_integration_test.sh \\\n --az us-east-1a \\\n --owner test \\\n --email test@example.com \\\n --defaults tests/integration/itest_defaults.yml \\\n [--profile my-aws-profile] \\\n [--keep]\n```\n\nThe first four flags are all required; the script has no `--help`\n\n, and any\nunrecognized argument exits 1. Start from\n`tests/integration/itest_defaults.yml.example`\n\n— the copy at\n`tests/integration/itest_defaults.yml`\n\nis gitignored, so account-specific values\nare never committed. A run costs roughly $0.21-$0.34 in EC2 charges.\n\nSee [ tests/integration/README.md](/rmarable/ParallelClusterMaker/blob/main/tests/integration/README.md) for the flag\nreference, prerequisites, cost derivation, log paths, and exit codes.\n\n`make lint`\n\nexits 0 but emits a small number of warnings that are intentional and safe to ignore:\n\n| Warning | Reason |\n|---|---|\n`yaml[line-length]` — ssh/chown/cp commands |\nOne-liners that are 162 chars (2 over limit); splitting would harm readability |\n`no-changed-when` |\n`pcluster` CLI commands are inherently stateful; `changed_when` on every poll would be misleading |\n`ignore-errors` |\nIntentional on cleanup tasks (S3 bucket, SNS topic, IAM role) that may not exist at delete time |\n`no-handler` |\nDeliberate pattern; notify/handler would require restructuring without benefit |\n\nThese are all tracked in `.ansible-lint`\n\nunder `warn_list`\n\nwith the same rationale.\n\nPotential future improvements, roughly ordered by impact:\n\n**EasyBuild easyconfig workflow**— accept a user-supplied list of EasyBuild module specs, download matching easyconfigs from the EasyBuild repository, build and install the modules on the head node, and run a smoke-test job for each one via Slurm. Useful for validating that a new OS or instance type can successfully build a site's standard software stack.\n\n**Terraform / CDK parity**— the toolkit is Ansible-native. A Terraform or AWS CDK implementation of the same lifecycle (`make`\n\n/`kill`\n\n/`access`\n\n) would fit more naturally into infrastructure-as-code pipelines that already use those tools.\n\nThis software is licensed under the Apache License, Version 2.0 with the Commons Clause restriction. You may use, modify, and distribute it freely, but you may not sell it or offer it as a commercial product or service without the explicit written consent of Rodney Marable. See `LICENSE`\n\nfor full terms.\n\nBy using this software:\n\n- You accept all potential risks involved with your use of this Open Source software.\n- You agree that the author shall have no responsibility or liability for any losses or damages incurred in conjunction with your use of this Open Source software.\n- You acknowledge that bugs may still be present, unexpected behavior might be observed, and some features may not be completely documented.\n\n**This software is authored by Rodney Marable in his individual capacity and is neither endorsed nor supported by Amazon Web Services.** You cannot create cases with AWS Technical Support or engage AWS support engineers in public forums if you have any questions, problems, or issues using this software.\n\n\"Play at your own risk!\" — Planet Patrol\n\n[https://github.com/rmarable/ParallelClusterMaker/issues](https://github.com/rmarable/ParallelClusterMaker/issues)\n\nPull requests welcome: [https://github.com/rmarable/ParallelClusterMaker/pulls](https://github.com/rmarable/ParallelClusterMaker/pulls)\n\nAI-assisted contributions are welcome — see [AI_POLICY.md](/rmarable/ParallelClusterMaker/blob/main/AI_POLICY.md) for submission guidance.", "url": "https://wpnews.pro/news/parallelclustermaker-cli-toolkit-to-manage-aws-parallelcluster-stacks", "canonical_source": "https://github.com/rmarable/ParallelClusterMaker", "published_at": "2026-08-09 13:52:16+00:00", "updated_at": "2026-08-09 14:05:14.453780+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "ai-infrastructure"], "entities": ["ParallelClusterMaker", "AWS ParallelCluster", "Claude Code", "Anthropic", "AWS", "Slurm", "Spack", "Lmod"], "alternates": {"html": "https://wpnews.pro/news/parallelclustermaker-cli-toolkit-to-manage-aws-parallelcluster-stacks", "markdown": "https://wpnews.pro/news/parallelclustermaker-cli-toolkit-to-manage-aws-parallelcluster-stacks.md", "text": "https://wpnews.pro/news/parallelclustermaker-cli-toolkit-to-manage-aws-parallelcluster-stacks.txt", "jsonld": "https://wpnews.pro/news/parallelclustermaker-cli-toolkit-to-manage-aws-parallelcluster-stacks.jsonld"}}