Building an HPC Environment with AWS ParallelCluster - Automated Slurm Cluster Provisioning and Scaling
Automatically provision HPC clusters with CloudFormation and manage jobs with the Slurm scheduler. Also covers cost optimization with Spot Instances.
Overview of ParallelCluster
ParallelCluster is an open-source HPC cluster management tool provided by AWS. You define cluster configuration (instance types, node count, storage, networking) in a YAML configuration file and automatically provision it as a CloudFormation stack with the pcluster create-cluster command. The Slurm job scheduler is configured by default, so you can use your existing Slurm job scripts. You can run parallel computing workloads such as computational fluid dynamics (CFD), molecular dynamics, genome analysis, weather simulation, and financial risk calculation. Version 3 introduced Multi-Queue mode, which allows defining multiple queues within a single cluster, each with different instance types and storage configurations. You can apply configuration changes to an existing cluster with pcluster update-cluster, and for some settings, such as the OS, the update is not allowed if the setting is changed.
Auto Scaling and Cost Optimization
ParallelCluster's auto scaling works in conjunction with Slurm's job queue. When jobs are submitted, compute nodes automatically launch, and when nodes remain idle for a specified period after job completion, they automatically terminate (default idle timer is 10 minutes). During periods with no jobs, the compute node count drops to zero, and no compute node charges are incurred. Using Spot Instances can lower HPC workload costs. Specifying multiple instance types with Allocation Strategy: capacity-optimized gives you a configuration prepared for Spot interruptions. For checkpoint-capable applications, you can configure automatic job requeuing when Spot interruptions occur. You can run the head node on On-Demand instances and use Spot only for compute nodes. The ScaledownIdletime parameter allows adjusting the idle timer; in environments where short jobs are frequently submitted, increasing this value reduces node launch/termination overhead.
EFA and Shared Storage
EFA (Elastic Fabric Adapter) is a high-speed network interface for HPC workloads that lowers inter-node communication latency. It is effective for MPI (Message Passing Interface) based parallel computing where inter-node data exchange becomes a bottleneck. Enabling EFA in the ParallelCluster configuration provisions EFA on compute nodes. You place nodes that use EFA in a placement group, and ParallelCluster creates a cluster placement group through the Placement section configuration. For shared storage, you can choose from FSx for Lustre, EFS, and EBS. You can use FSx for Lustre for parallel reads of large datasets, and its integration with S3 automatically imports data from S3 into the Lustre file system and exports computation results back to S3. EFS is effective for sharing data across multiple clusters or in environments with high node count variability, but has higher latency than FSx for Lustre.
Design Best Practices and Pitfalls
Items to review when running ParallelCluster in production: clusters with hundreds of nodes increase Slurm controller memory consumption, so you size the head node to match the node count. Since the entire cluster becomes unavailable if the head node stops, configure periodic backups using EBS snapshots. For network design, place compute nodes in private subnets and access S3 and CloudWatch via NAT gateways or VPC endpoints. Using an S3 VPC endpoint (gateway type) avoids NAT gateway data transfer charges. When using custom AMIs, you customize an official AMI that includes ParallelCluster-required packages (Slurm, Munge, supervisord) and build it with the pcluster build-image command. You can choose the OS from Amazon Linux 2023, Ubuntu 22.04 / 24.04, RHEL 8 / 9, and Rocky Linux 8 / 9.
Comparison with AWS Batch
AWS offers two services for HPC/batch workloads: ParallelCluster and AWS Batch, and you choose between them by workload characteristics. ParallelCluster uses Slurm as its scheduler, so organizations that use Slurm in on-premises HPC environments can use it to migrate to the cloud without changing job scripts. For tightly-coupled parallel computing that uses low-latency inter-node communication via EFA (CFD, molecular dynamics), one approach is to use ParallelCluster. In contrast, AWS Batch is a fully managed service that packages workloads with container-based job definitions. For high-throughput computing that executes large numbers of independent tasks in parallel (genome sequence alignment, image processing pipelines), you use AWS Batch. ParallelCluster lets you specify cluster configuration in detail, so you can use it for research computing that requires specific instance placement and storage configurations.
Summary
ParallelCluster is a tool that automatically provisions Slurm-based HPC clusters on AWS. Auto scaling provides resource management aligned with job demand, Spot Instances reduce costs, and EFA delivers high-speed inter-node communication. You can use it for migrating from on-premises HPC clusters or for handling temporary increases in computing demand. Multi-Queue mode consolidates different workloads into a single cluster, and FSx for Lustre with S3 integration streamlines large-scale data I/O.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
- AWS ParallelCluster official page
- Amazon EC2 official page
- Amazon FSx official page
- AWS Documentation (official)
If this page and the official documentation disagree, treat the official documentation as authoritative.